Archive for the ‘development’ Category

Making MTP faster on Symbian^3

July 19, 2010

Over the last two months, we have been working with developers from the kernelhwsrv, mds and musicplayer packages to improve the performance of MTP synchronisation.

speedy-mug

Media Transfer Protocol (MTP) is a standard originally developed by Microsoft that allows users to transfer music files (in our case from the PC to a Symbian device).

The music metadata such as the album name, artist, cover art etc. is stored in Symbian SQL so any performance improvements in persistentdata make a difference towards the end-user performance. I have summarised the changes relating to persistentdata below.

Other changes have been made in mds and musicplayer packages as well, e.g. tuning of database configuration parameters, but I haven’t described these.

Increased the soft heap limit to 8MB

Symbian SQL uses SQLite in shared-cache mode. This reduces the amount of memory and file I/O needed by the Page Cache because multiple connections to the same database share a single cache. In this mode, there is a single page cache for each open database.

The memory available for all page caches is limited by the soft heap limit: a limit enforced by the SQLite library on the total amount of heap memory allocated by SQLite. The limit is called a “soft” limit because SQLite does not guarantee the memory usage of the library will stay under this limit. SQLite tries to stay within the limit, but will still allocate memory if it is unable to find any memory to reuse. If SQLite has already reached the soft heap limit during a database transaction, it will remove unmodified pages from the page cache and write pages to disk to allow new pages to be read and updated.

The soft heap limit on Symbian^3 was originally set to 1024kB. During our investigation, we found that this was no longer an appropriate size considering the current usage of the database. For example, in MTP synchronisation, the thumbnail server writes batches of 60 thumbnails for the album art to its database. The total amount of data (metadata + BLOB) to be written in each batch is typically between 1.7MB and 2.5MB. This means that the whole transaction cannot be stored in memory before being flushed to disk.

When the soft heap limit is too low, the database commit will have a more random I/O pattern (it jumps back and forth in file position) because the database pages are written more than once to stay under the soft heap limit. This leads to bad performance especially on slower media like SD cards.

After some experimentation with batch sizes and soft heap limit configurations, we determined that the soft heap limit should be substantially increased. The default soft heap limit for Symbian SQL was increased from 1MB to 8MB for hardware targets in the week 21 delivery from Nokia (difference listing). This change reduced the time taken for MTP music download by 20%.

The soft heap limit for the WINSCW emulator configuration was not modified (left as 1MB). There is a limited virtual address space available for the emulator (as discussed on the forum) and increasing the soft heap limit means that our maximum EPOCHEAPSIZE would need to change too.

Increased the capacity of RFileBuf64 to hold 4 pages

All file access in Symbian SQL uses the RFileBuf64 class. The implementation has been updated to ensure that the buffer is always large enough to contain 4 pages from the database being accessed.

If 4 pages or more can fit into the default buffer size of 8kB, then the buffer size will not been modified. If this isn’t possible (e.g. because the database page size is 16kB) then the buffer is resized to contain 4 pages. The read-ahead is set as a multiple of the page size. This applies to the journal and main database file.

Reduced calls to RFile::SetSize made by RFileBuf64

The RFileBuf64 implementation was updated to defer calls to RFile::SetSize until the pages are going to be written to disk. This avoids redundant file metadata updates when pages are written out-of-order at the end of the database file.

Journal ‘resting size’ increased

Symbian SQL uses the SQLite PRAGMA journal_size_limit to limit the size of journal files after a database transaction has been committed. The ‘resting size’ for Symbian^3 journals before this change was 64kB.

One of my colleagues observed from a trace that committing a 60 thumbnail transaction took approximately 700ms, of which 200-300ms was FAT updates to allocate and deallocate clusters to the journal file. The thumbnail database journal typically grows to 150kB or more during each transaction, so a ‘resting size’ of 64kB results in cluster allocation/deallocation every time.

To avoid the repeated allocation and deallocation of FAT clusters on databases with larger page sizes, the resting size is set to 16 pages or 512kb (whichever is the smaller size). The minimum resting size remains 64kB.

Cluster allocation during file server cache write-back (kernelhwsrv)

Each bulk addition of thumbnails results in approx 2.5MB of new database pages. As the file server cache writes this to disk the FAT metadata is updated multiple times as clusters are allocated to the file. The file server cache implementation was modified to consolidate FAT cluster allocations during write-back.


Image speedy-mug by Symbian Foundation used under a Creative Commons Attribution Non Commercial Share Alike licence.

Join the Symbian Bug Squad

May 21, 2010

Symbian Bug Squad

The Symbian Bug Squad is looking for new members to find, locate and fix bugs in key areas of the Symbian Platform.

The next test day is on Monday (24th May) and it is not too late to get involved. The team will focus on the Organizer package (calendar, to-do, alarms, notes). More information is available from the forum announcement.

We intend to organize a persistentdata bug day in the near future. If you are interested to join us or have ideas what we should focus on, please let us know by adding a comment to this post or via the persistentdata-dev mailing list.

Help us with GCC-E compilation errors

February 12, 2010

As Mark Wilcox posted on the Symbian blog in It’s Open, So Now What? and mentioned in the theregister.co.uk today we can’t build all of the Symbian packages with the open source GCC-E compiler yet.

We have a couple of GCC-E bugs in persistentdata: bug 1767 and bug 1795.

If you’d like to help, have a look at the Compiler Compatibility page for more details.

Getting started

February 12, 2010

Want to get started developing on Symbian?

These steps explain what you need to do to set-up a development environment for the Symbian platform.

1. Install Perl
To build the code you need ActivePerl, a standard distribution of the Perl language. The Symbian Foundation recommend Perl 5.8.9. Further information in Kits Q&A.

2. Install Python
Symbian Build System v2 (SBSv2) needs ActiveState Python 2.5.1 or later installed. This is a standard distribution of the Python language. Symbian Foundation suggests Python 2.5.4, but I installed ActivePython 2.6.4.10 as the latest version of SBSv2 didn’t find python26.dll otherwise. Further information in Kits Q&A.

3. Install Java
Download the Java SE Development Kit from java.sun.com. I used JDK 6 Update 18

4. Install 7-zip
The Product Development Kit is distributed as a collection of zip files. You should install 7-zip from www.7-zip.org to unpack them because other zip tools have been known to have problems with the number of files in the archives.

5. Get a Product Development Kit
The Product Development Kit (PDK) contains the binaries, tools and source code you need to develop code and build products. The latest PDK is Symbian^3 Platform Release v3.0.f.

You can download the zip files using your browser and unzip them using 7-Zip, or better still use the downloadkit.py utility as described in How to Download a PDK. Unzip everything to a new drive, created with the subst command.

6. Install PDT 1.5
The Product Development Toolkit (PDT) contains a complete set of PC-side tools (such as Carbide.c++) and documentation. I installed the Product Development Toolkit (PDT) v1.5 and then the PDK overlay to install the tools to the epoc32 tree.

7. Install the GCC compiler
Install Sourcery G++ Lite 2009q3-63 to build for ARM hardware using the free GCC toolchain. When installing, do not use the default installation path. You need to install it without any spaces in the path as described in The GCCE toolchain initiative

Download the update needed to SBSv2 from the PDK download page (GCC Support – Raptor 2.12.1) and unzip it over the Raptor directory in your PDT install.

You might need to set-up some environment variables to get everything running. There’s good advice on developer.symbian.org if you do a search. I had to add:

set SBS_GCCE432BIN=C:/Apps/CodeSourcery/GCCE/bin

8. Install Mercurial
Mercurial is the SCM tool used by the Symbian Foundation. Install it from the Mercurial site and follow the guidance in Mercurial Quick Start to configure the tool.

9. Check your development environment
Follow the instructions on developer.symbian.org to test your development environment. The output will look something like this:

Z:\>perl -version

This is perl, v5.8.9 built for MSWin32-x86-multi-thread
(with 12 registered patches, see perl -V for more detail)

Z:\>python -V
Python 2.5.4

Z:\>java -version
java version “1.6.0_18”
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)

Z:\>mwccsym2.exe -version
Nokia Codewarrior C/C++ Compiler for Windows/x86.
Copyright (c) 2008, Nokia Corporation
All rights reserved.
Version 3.2.5 build 473 (Build 473)
Runtime Built: Apr 30 2008 14:33:37

Z:\>arm-none-symbianelf-gcc.exe -v
Using built-in specs.
Target: arm-none-symbianelf
Configured with: ….
Thread model: single
gcc version 4.4.1 (Sourcery G++ Lite 2009q3-63)

Start the emulator with \epoc32\release\winscw\udeb\epoc.exe

10. Get the code, and make changes
Get the latest persistentdata MCL code using Mercurial:

Z:\>hg clone http://developer.symbian.org/oss/MCL/sf/os/persistentdata/
destination directory: persistentdata
requesting all changes
adding changesets
adding manifests
adding file changes
added 5 changesets with 5738 changes to 5727 files (+1 heads)
updating to branch default
5721 files updated, 0 files merged, 0 files removed, 0 files unresolved

Everything you need to know about building the code, working with Mercurial and making your first change is available from the Symbian Platform Contributors category on developer.symbian.org. If you get stuck, use the Developer Forums. We’ll be happy to help.

Summary
This post describes how you can install and configure a Symbian development environment for development. It might seem like a lot of steps to get set-up, but most of the tools are a one-off installation and once installed can be used for every new release of the platform.

There is a nice graphical representation of the installation process on the Symbian developer site.