Making MTP faster on Symbian^3

Over the last two months, we have been working with developers from the kernelhwsrv, mds and musicplayer packages to improve the performance of MTP synchronisation.

speedy-mug

Media Transfer Protocol (MTP) is a standard originally developed by Microsoft that allows users to transfer music files (in our case from the PC to a Symbian device).

The music metadata such as the album name, artist, cover art etc. is stored in Symbian SQL so any performance improvements in persistentdata make a difference towards the end-user performance. I have summarised the changes relating to persistentdata below.

Other changes have been made in mds and musicplayer packages as well, e.g. tuning of database configuration parameters, but I haven’t described these.

Increased the soft heap limit to 8MB

Symbian SQL uses SQLite in shared-cache mode. This reduces the amount of memory and file I/O needed by the Page Cache because multiple connections to the same database share a single cache. In this mode, there is a single page cache for each open database.

The memory available for all page caches is limited by the soft heap limit: a limit enforced by the SQLite library on the total amount of heap memory allocated by SQLite. The limit is called a “soft” limit because SQLite does not guarantee the memory usage of the library will stay under this limit. SQLite tries to stay within the limit, but will still allocate memory if it is unable to find any memory to reuse. If SQLite has already reached the soft heap limit during a database transaction, it will remove unmodified pages from the page cache and write pages to disk to allow new pages to be read and updated.

The soft heap limit on Symbian^3 was originally set to 1024kB. During our investigation, we found that this was no longer an appropriate size considering the current usage of the database. For example, in MTP synchronisation, the thumbnail server writes batches of 60 thumbnails for the album art to its database. The total amount of data (metadata + BLOB) to be written in each batch is typically between 1.7MB and 2.5MB. This means that the whole transaction cannot be stored in memory before being flushed to disk.

When the soft heap limit is too low, the database commit will have a more random I/O pattern (it jumps back and forth in file position) because the database pages are written more than once to stay under the soft heap limit. This leads to bad performance especially on slower media like SD cards.

After some experimentation with batch sizes and soft heap limit configurations, we determined that the soft heap limit should be substantially increased. The default soft heap limit for Symbian SQL was increased from 1MB to 8MB for hardware targets in the week 21 delivery from Nokia (difference listing). This change reduced the time taken for MTP music download by 20%.

The soft heap limit for the WINSCW emulator configuration was not modified (left as 1MB). There is a limited virtual address space available for the emulator (as discussed on the forum) and increasing the soft heap limit means that our maximum EPOCHEAPSIZE would need to change too.

Increased the capacity of RFileBuf64 to hold 4 pages

All file access in Symbian SQL uses the RFileBuf64 class. The implementation has been updated to ensure that the buffer is always large enough to contain 4 pages from the database being accessed.

If 4 pages or more can fit into the default buffer size of 8kB, then the buffer size will not been modified. If this isn’t possible (e.g. because the database page size is 16kB) then the buffer is resized to contain 4 pages. The read-ahead is set as a multiple of the page size. This applies to the journal and main database file.

Reduced calls to RFile::SetSize made by RFileBuf64

The RFileBuf64 implementation was updated to defer calls to RFile::SetSize until the pages are going to be written to disk. This avoids redundant file metadata updates when pages are written out-of-order at the end of the database file.

Journal ‘resting size’ increased

Symbian SQL uses the SQLite PRAGMA journal_size_limit to limit the size of journal files after a database transaction has been committed. The ‘resting size’ for Symbian^3 journals before this change was 64kB.

One of my colleagues observed from a trace that committing a 60 thumbnail transaction took approximately 700ms, of which 200-300ms was FAT updates to allocate and deallocate clusters to the journal file. The thumbnail database journal typically grows to 150kB or more during each transaction, so a ‘resting size’ of 64kB results in cluster allocation/deallocation every time.

To avoid the repeated allocation and deallocation of FAT clusters on databases with larger page sizes, the resting size is set to 16 pages or 512kb (whichever is the smaller size). The minimum resting size remains 64kB.

Cluster allocation during file server cache write-back (kernelhwsrv)

Each bulk addition of thumbnails results in approx 2.5MB of new database pages. As the file server cache writes this to disk the FAT metadata is updated multiple times as clusters are allocated to the file. The file server cache implementation was modified to consolidate FAT cluster allocations during write-back.


Image speedy-mug by Symbian Foundation used under a Creative Commons Attribution Non Commercial Share Alike licence.

Advertisements

2 Responses to “Making MTP faster on Symbian^3”

  1. teknologskip Says:

    Fantastic as always, Chris! Stories about optimization of vertical use cases like this really shows how the stack works together. More please!

  2. Tweets that mention Making MTP faster on Symbian^3 « Chris Dudding -- Topsy.com Says:

    […] This post was mentioned on Twitter by Tyson Key, Sebastian Brannstrom. Sebastian Brannstrom said: Fantastic blog post on optimizing media transfer in Symbian^3 – http://bit.ly/cvLUu3 […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: