Thursday, February 14, 2008

Synchronization in Android

Note: this example program was written before the new m5-rc14 release of the SDK was published and therefore it refers to m3-rc37a SDK release. I intend to investigate the sync issue in the new release too.

The first thing I noted about Android content providers that they support synchronization. This fact has huge implications and shows, how well the Android mobile platform is designed. Synchronization is a key enabler in mobile applications but is often "engineered in" later into the platforms. This approach is about as realistic as "adding security later" to a non security-aware platform.

It is not surprising for an Android-hardened developer that synchronization does not work in the m3-rc37a release and the example program presented here is not able to accomplish any real synchronization. This statement would discourage lesser programmers but not the ones dealing with the Android platform! Decyphering the Android synchronization framework is an interesting exercise in itself even though that framework may change in the future. Key elements are already there and there are some interesting findings regarding synchronization as a service.

You can download the example program from here.



The naive observation that led me to deal with the Android synchronization framework is that the log clearly indicates a running synchronization service. The synchronization service (actually called android.content.ContentService) listens to data connection/disconnection events and logs these state changes. In order to actually activate the sync part of the service (other part of the service deals with the XMPP connection), the following steps should be taken.
  • The sync part of the service must be enabled by manipulating the sync_enabled property in the settings table. This is done by calling android.provider.Settings.Gservices.putString( cr,"sync_enabled","true" ). You noticed it well: the G in Gservices refers to Google. Clearly, Android sync framework programmers considered synchronization a Google service. In order for this call to succeed, the application must have android.permission.WRITE_SETTINGS permission (check the manifest of the example program).
  • Account information must be provided. Interestingly again, the service handling this logic is called GoogleLoginService. ContentService obtains account information from GoogleLoginService and if there are no accounts specified, synchronization stops. This is a bit frightening but in m3-rc37a release the account information is stored in a local SQLite table. Do the following:
    adb shell
    cd data/data/com.google.android.providers.googleapps/databases
    sqlite3 accounts.db

    INSERT INTO "accounts" VALUES(1,'user','password',1);

    .quit
  • The sync engine stores its settings, history and stats in a content provider under the content://sync/ tree. Strangely, this content provider is not implemented at all. The example program provides a naive implementation which is good enough to make the sync service work.
The sync service is ready for work now. In Android, only synchronizable data providers can be synchronized and this makes this whole exercise more valuable than just pure hacking. Content providers must jump some fences to become synchronizable. Pretty few built-in content providers of the SDK are synchronizable (you can examine these lists using the example program's "List sync providers" and "List content provider" menu items).

The steps necessary to make a content provider synchronizable are the following:
  • The content provider must be marked as synchronizable in the manifest (again, the XML fragment is mangled because of the limitations of the blog engine):
    [provider class=".SimpleStringDataProvider" android:authorities="aexp.syncexample.SimpleString"
    android:syncable="true"/]
  • The content provider must be able to provide a temporary version of itself. The method signature is the following:

    public ContentProvider getTemporaryInstance()

    The synchronization engine manipulates the temporary instance of the content provider and changes done on the temporary content provider are merged by another method the content provider must implement:

    public MergeResult merge(SyncContext context, ContentProvider diffs, boolean readOnly)

    This latter method gets the temporary content provider instance after one synchronization step is finished on it and merges it with the main content provider. Meanwhile, collisions may be detected (data changed incompatibly on both sides since the last synchronization) and as a result, synchronization may even be interrupted by the user.
  • And last, but not least, the synchronizable content provider must be able to return the SyncAdapter that is able to synchronize the provider.

    public SyncAdapter getSyncAdapter(Context context)

    The SyncAdapter instance abstracts the entire synchronization engine. The example program provides a dummy implementation.
This is not enough. The database structure of the content provider must be designed in such a way that the SyncAdapter supported by the content provider is able to manipulate it. There are different synchronization strategies and android.provider.SyncColumns provides a number of columns that may be useful implementing them. The content provider in the example program supports the most common one, the timestamp-based synchronization. In this method, the "version" of a data item is expressed by a high-resolution timestamp. Each item timestamped after the timestamp of the last synchronization is candidate for sync processing.

The interaction between the sync engine and the sync adapter can be followed on the the log below:

D/SyncExample( 614): syncDB
D/SyncExample( 614): After syncDB

D/Sync ( 467): running sync operation type: 1 url: content://aexp.syncexample.SimpleString
D/ActivityThread( 467): Installing external provider aexp.syncexample.SimpleString: aexp.syncexample.SimpleStringDataProvider

D/ActivityThread( 467): Installing external provider sync: aexp.syncexample.SyncSettingsProvider
I/Sync ( 467): Starting sync for aexp.syncexample.SimpleString
D/SyncAdapterStub( 614): syncStarting
D/SyncAdapterStub( 614): isReadOnly

D/SyncAdapterStub( 614): getServerDiffs, contentProvider: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, tempContentProvider: android.content.ContentProvider$Transport@400144a0 D/SyncAdapterStub( 614): writeSyncData: aexp.syncexample.SimpleString
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: null

D/SyncAdapterStub( 614): syncEnding

I/Sync ( 467): Ending sync for aexp.syncexample.SimpleString
D/Sync ( 467): finished sync operation type: 1 url: content://aexp.syncexample.SimpleString


  • In the first step, getServerDiffs is called on the SyncAdapter, using the temporary provider as input. In case of the timestamp-based adapter, the client-server communication must establish the timestamp at which this client and server synchronized last, send client changes after that timestamp and obtain changes from the server into the temporary provider.
  • In the second step, the provider's merge method is called with the temporary provider as parameter. At this point the content provider must reconcile the client-side database with the temporary database. During the merging, collisions may be detected when the client and the server both changed a data item since the last synchronization in non-compatible way. Eventually collisions may need to be escalated to the user who may even interrupt the synchronization process.
  • After this step, we have the reconciled database on the client side. The client now needs to notify the server about collision resolution and sends the client diffs one more time using the SyncAdapter's sendClientDiffs method. This step is not present in the log because our primitive synchronization example did not produce collisions.

After all this suffering, I guess, you would like to see a real synchronization session. I have to disappoint you: beside some scattered SyncML DS-related classes, I was not able to find any real sync adapter. Implementing a real synchronization engine takes a lot of time but I happen to have a SyncML-based engine implemented and I intend to put it under Android's sync framework. So stay tuned. :-)

15 comments:

Cyrus said...

Nice work. I tried your sample code on a Mac with m3-rc37, and unfortunately, there's no log output from the content service after:

D/SyncExample(629): syncDB
D/SyncExample(629): After syncDB

It doesn't look like the sync adapter methods are being called either. Not sure whether this is a logger issue or if there's a discrepancy in the SDK, but I'll keep digging.

Cyrus

Cyrus said...

I also built and run the sync sample on Windows with m3-rc37, with the same result... Are there any other setup requirements aside from those mentioned in your blog article? Thanks!

john said...

Did you ever finish a SyncML based engine for Android?

I want to be able to sync my G1 with my company's Oracle Calendar (it'd be an extra bonus to have this imply that my calendar data was in turn being synchronized to Google Calendar, but that's a secondary result).

Another bonus would be if it was read-only (ie. the phone should not upload new events, nor delete events, from Oracle calendar), and if I could choose which of my Google Calendars would be the recipient of the Oracle Calendar data (ex: I have 3 Google Calendar Calendars: Home, Work, Birthdays ... I want Oracle Calendar to sync into "Work", which it does now via Oracle -> Nokia -> iSync -> Spanning Sync -> Google).

Nokia's SyncML client say you can make the sync be one way, but it doesn't actually work in practice (it errors out if you try to make it "server to phone" only).

If you did this as an application (via the Android App Store, or some other store), I'd pay a reasonable price for it. Any updated information would be bonus.

(being able to sync with iSync and/or Thunderbird contacts would be great too)

清风居士 said...

That is a great work.
Can anyone still download the attached example code? I failed all the time with network error.

Can't someone share it with me by email to the below email address:

funyoung AT gmail.com

Thank you in advance.

Unknown said...

Thanks a lot for important info...now i'm able to delete google accounts through adb.

But can u plz tell how to do it in code, i want to drop table accounts.db which exists at path: data\system\accounts.db.

It would be really helpful for my app.

Gabor Paller said...

Parul, the Android security framework prevents you dropping the table. Basically your app does not have the Linux privileges to do so.

vicky said...

This is great, I need something like this, but i am not able to open the example link (getting network error every time)

Can some one please share this on the below mail id OR some other alternate link to access the examples :

vicky DOT gladiator AT GMAIL DOT COM

Thanks,

Android said...

Works also with Samsung Galaxy S - execpt two little strange exceptions.The 'd' key does nothing, the 'g' key brings the phone into the mode where it needs to be unlocked by gesture. (small/capital the same)

Unknown said...

android.content.ContentService is not available in my application what can i do for it..??

Gabor Paller said...

Sindhu, this entry is obsolete. Check out the SampleSyncAdapter example program among the stock Android sample programs.

Unknown said...

Hi Gabor Paller,I'm new to java plz explain full details abt sync. I'm using Android 2.2. Still i'm not getting contentService..

Gabor Paller said...

Sindhu, again I propose that you check the example program I mentioned. The API has changed completely.

sathish said...

i'm new to android development, and developing database application i want to synchronize my sq lite db to remote db how it is possible i'm using android 2.2

Prabhu Ganapathylingam said...

Hi,

I am new to android. I have downloaded your code. I am using android sdk 2.3. It is showing many errors. can you send me a updated code.

Gabor Paller said...

prabhu, it is an outdated code, pre-1.0. The sync subsystem in post-1.0 Android has been completely reengineered, this post will not help you with that one.