Nokia N8 is shipping

October 1, 2010

The first Symbian^3 device is now shipping: the Nokia N8.

Pictures of the first devices leaving the factories can be seen at conversations.nokia.com.

Advertisements

Making MTP faster on Symbian^3

July 19, 2010

Over the last two months, we have been working with developers from the kernelhwsrv, mds and musicplayer packages to improve the performance of MTP synchronisation.

speedy-mug

Media Transfer Protocol (MTP) is a standard originally developed by Microsoft that allows users to transfer music files (in our case from the PC to a Symbian device).

The music metadata such as the album name, artist, cover art etc. is stored in Symbian SQL so any performance improvements in persistentdata make a difference towards the end-user performance. I have summarised the changes relating to persistentdata below.

Other changes have been made in mds and musicplayer packages as well, e.g. tuning of database configuration parameters, but I haven’t described these.

Increased the soft heap limit to 8MB

Symbian SQL uses SQLite in shared-cache mode. This reduces the amount of memory and file I/O needed by the Page Cache because multiple connections to the same database share a single cache. In this mode, there is a single page cache for each open database.

The memory available for all page caches is limited by the soft heap limit: a limit enforced by the SQLite library on the total amount of heap memory allocated by SQLite. The limit is called a “soft” limit because SQLite does not guarantee the memory usage of the library will stay under this limit. SQLite tries to stay within the limit, but will still allocate memory if it is unable to find any memory to reuse. If SQLite has already reached the soft heap limit during a database transaction, it will remove unmodified pages from the page cache and write pages to disk to allow new pages to be read and updated.

The soft heap limit on Symbian^3 was originally set to 1024kB. During our investigation, we found that this was no longer an appropriate size considering the current usage of the database. For example, in MTP synchronisation, the thumbnail server writes batches of 60 thumbnails for the album art to its database. The total amount of data (metadata + BLOB) to be written in each batch is typically between 1.7MB and 2.5MB. This means that the whole transaction cannot be stored in memory before being flushed to disk.

When the soft heap limit is too low, the database commit will have a more random I/O pattern (it jumps back and forth in file position) because the database pages are written more than once to stay under the soft heap limit. This leads to bad performance especially on slower media like SD cards.

After some experimentation with batch sizes and soft heap limit configurations, we determined that the soft heap limit should be substantially increased. The default soft heap limit for Symbian SQL was increased from 1MB to 8MB for hardware targets in the week 21 delivery from Nokia (difference listing). This change reduced the time taken for MTP music download by 20%.

The soft heap limit for the WINSCW emulator configuration was not modified (left as 1MB). There is a limited virtual address space available for the emulator (as discussed on the forum) and increasing the soft heap limit means that our maximum EPOCHEAPSIZE would need to change too.

Increased the capacity of RFileBuf64 to hold 4 pages

All file access in Symbian SQL uses the RFileBuf64 class. The implementation has been updated to ensure that the buffer is always large enough to contain 4 pages from the database being accessed.

If 4 pages or more can fit into the default buffer size of 8kB, then the buffer size will not been modified. If this isn’t possible (e.g. because the database page size is 16kB) then the buffer is resized to contain 4 pages. The read-ahead is set as a multiple of the page size. This applies to the journal and main database file.

Reduced calls to RFile::SetSize made by RFileBuf64

The RFileBuf64 implementation was updated to defer calls to RFile::SetSize until the pages are going to be written to disk. This avoids redundant file metadata updates when pages are written out-of-order at the end of the database file.

Journal ‘resting size’ increased

Symbian SQL uses the SQLite PRAGMA journal_size_limit to limit the size of journal files after a database transaction has been committed. The ‘resting size’ for Symbian^3 journals before this change was 64kB.

One of my colleagues observed from a trace that committing a 60 thumbnail transaction took approximately 700ms, of which 200-300ms was FAT updates to allocate and deallocate clusters to the journal file. The thumbnail database journal typically grows to 150kB or more during each transaction, so a ‘resting size’ of 64kB results in cluster allocation/deallocation every time.

To avoid the repeated allocation and deallocation of FAT clusters on databases with larger page sizes, the resting size is set to 16 pages or 512kb (whichever is the smaller size). The minimum resting size remains 64kB.

Cluster allocation during file server cache write-back (kernelhwsrv)

Each bulk addition of thumbnails results in approx 2.5MB of new database pages. As the file server cache writes this to disk the FAT metadata is updated multiple times as clusters are allocated to the file. The file server cache implementation was modified to consolidate FAT cluster allocations during write-back.


Image speedy-mug by Symbian Foundation used under a Creative Commons Attribution Non Commercial Share Alike licence.

NTT DOCOMO contributes again

July 14, 2010


I integrated a second contribution from NTT DOCOMO last week to fix a bug in one of the Symbian SQL test harnesses.

The bug (bug 3170) concerned the handling of end-of-line characters in an input file for the t_sqlperformance4 test. The input file contained a sequence of SQL statements executed during a Music Player application use case. These statements are parsed and executed by the test harness to measure performance.

The bug was only reproducible when the source code was obtained from Mercurial because the input file would have UNIX line-endings (line feed) rather than the traditional Windows line-endings (carriage-return followed by line feed). We didn’t spot the problem ourselves because the clients for our internal source code management systems automatically convert to Windows line-endings.

The problem is that the test assumed that the line endings for the text file would follow Windows conventions leading to the SQL statements being truncated when only line feed character was used. The fix allows both formats to be handled.

This type of problem was anticipated by William Roberts in May 2009. He wrote an excellent post about the handling of line-endings in Mercurial repositories called “A dilemma at the end of the line?” on the Symbian blog. It’s worth a read.

Thanks to NTT DOCOMO, INC for identifying this bug and contributing the solution to the community!


Image Symbian_31 by Symbian Foundation used under a Creative Commons Attribution Non Commercial Share Alike licence.

Happy 10th birthday, SQLite

June 1, 2010

Birthday Cake

Ten years ago, on 29th May 2000, D. Richard Hipp made the first check-in of the SQLite code.

The SQLite development team, led by DRH, has enhanced SQLite’s capabilities whilst preserving the compact size of the library, improving reliability and optimising performance. They made 7114 check-ins over ten years – an average of 2.1 check-ins per day.

SQLite is a hugely successful project. It is probably the most widely deployed SQL database engine in the world and an essential part of the Symbian persistentdata package.

I’d like to thank Richard and his development team for SQLite and congratulate them on this important milestone.

Happy Birthday SQLite!


Image Birthday Cake by Theresa Thompson used under a Creative Commons Attribution licence.

Join the Symbian Bug Squad

May 21, 2010

Symbian Bug Squad

The Symbian Bug Squad is looking for new members to find, locate and fix bugs in key areas of the Symbian Platform.

The next test day is on Monday (24th May) and it is not too late to get involved. The team will focus on the Organizer package (calendar, to-do, alarms, notes). More information is available from the forum announcement.

We intend to organize a persistentdata bug day in the near future. If you are interested to join us or have ideas what we should focus on, please let us know by adding a comment to this post or via the persistentdata-dev mailing list.

Persistentdata news summary: April 2010

April 30, 2010

The work needed to allow the Symbian platform to compile with a free toolchain (Software Freedom Fighters) continues. The bugs in persistentdata relate to our usage of the offsetof macro. Lars Kurth has submitted a proposal how to resolve the issues with the macro to the Architecture council. We wait for their response.

We reviewed the report from the Unslow static analysis tool. We found that all issues except one were false positives. See bug 2227 for our analysis. A new bug has been created to track the changes we’d like to make to central repository (bug 2467).

Pete created a Getting Started page on the wiki to describe how to build the persistentdata components and run their unit tests. Let us know if you get stuck.

I participated in the Contacts package test day to find out about the Symbian Bugsquad. We’ve decided to organize our own test and bug fixing day in the next couple of months. Please join the discussion about it in the persistentdata-dev mailing list.

Nokia announced its first Symbian^3 device: Nokia N8.

I discussed how we can improve the information available about the platform and met some familiar faces at the last couple of London Stammtisch events.

I met with Victor Palau and Sebastian Brännström from Symbian to discuss their ideas about assessing the health of a package from a community point of view. The package scorecard concept proposed looks promising to me. More information can be found in Victor’s forum post.

We doubled the default size of the central repository cache to 200kB to avoid CommsDat from being continuously evicted and reloaded. Combined with changes from selected central repository clients, Browser performance on Symbian^3 has improved.

Introducing Pete Sanders: our new committer

March 12, 2010

I appointed the first committer for the package this week: Pete Sanders from Nokia.

Pete joined Symbian Software Ltd in 2000 as a Technical Consultant. In this role, he worked closely with a number of Symbian Licensees and Partners contributing to their Symbian devices. He is currently working at Nokia, as a member of the Persistent Data Services package team.

Pete will be helping me to maintain and develop the package and is my deputy for package owner responsibilities.

NTT DOCOMO contributes to persistentdata

March 3, 2010

I accepted the first external contribution to the package today.

Bug 1915 SQL server panics when using long column type strings

The contribution was a defect fix for Bug 1915. This defect would cause a USER 11 panic in the Symbian SQL server if the RSqlStatement::DeclaredColumnType() API was used to retrieve a column type definition longer than 19 characters.

The defect was found and fixed by NTT DOCOMO as part of their development work on Symbian^2. Their contribution included the code changes to fix the defect and new regression test cases.

I’d like to thank NTT DOCOMO for their contribution, Lars Kurth and Antti Enqvist from the Symbian Foundation for their support with the process, and my package team colleagues Alex and Pete for their technical review.

I hope this will be the first of many contributions from the community this year.

Open source at the V&A

March 1, 2010

I had an enjoyable afternoon attending the SAP Digital Design Festival at the Victoria and Albert Museum on Sunday. The festival was a weekend of special events organised to accompany their Decode: Digital Design Sensations exhibition.

I attended a workshop, interacted with some N900 hacks and spoke with their authors. I left the museum feeling quite inspired.

Recode Decode workshop

The Recode Decode workshop given by Karsten Schmidt explained how the digital identity for the Decode exhibition was created and how it can be remixed by the public because it is provided as open source code.

If you live or work in London, you may have seen the digital identity for the exhibition showing on the digital screens in London Underground stations now.

The identity was created using Processing: an open source programming language and environment for programming images, animations and interactive graphics. I’d not come across it before, but after experimenting with it in the workshop I think it deserves to be more famous.

If you want to try remixing it yourself, the code is hosted on Google code and can be fetched using Mercurial (hg clone https://decode.googlecode.com/hg/ decode).

There is more information about the open source identity concept on Karsten’s website.

PUSH N900

 PUSH N900 exhibiting at V&A DECODE

PUSH N900 exhibiting at V&A DECODE

The PUSH N900 exhibit showcased a number of hacks to Nokia’s N900 device.

There was a skateboard fitted with additional sensors to track skateboard tricks communicating via Bluetooth to the N900, a haptic belt that could vibrate to indicate which direction you should go based on turn-by-turn navigation and a motorised Etch-a-Sketch that could draw pictures taken with the camera. You can find out more from the Push N900 website.

However, my favourite was the KAPing with the N900 project: an N900 in a motorised jig attached to a kite that could take aerial photographs controlled by another N900.

The controls used the sensors in the device to move the camera, so you could simply hold the device in your palm and tilt it to the left to angle the camera to the left. Live images from the N900 in the kite are sent using wifi to the N900 used as a controller and the whole thing is written in Python. Brilliant!

We need these type of projects on Symbian too

Wiley publishes Inside Symbian SQL

February 26, 2010

Inside Symbian SQL: A Mobile Developer's Guide to SQLite

Inside Symbian SQL, by Ivan Litovski and Richard Maynard, was published by John Wiley & Sons today.

You can find out more on the Symbian blog and by visiting the Inside Symbian SQL wiki site. The site contains a sample chapter, links to further resources, example code and a promotional discount code from the publisher.