Friday, February 29, 2008

Autostarting services

Some privileged services start automatically in Android while the lowly applications I have written so far had to be started manually. I was not satisfied with the situation so I went after the Android bootup sequence.

The Linux layer under the JVM boots up normally, using the init script but I was not interested in that. I found that the Java services are all started by the System server (android.server.SystemServer) which does launch each system server in a hardwired way.

...
Log.i("SystemServer", "Starting Alarm Manager.");
sm.addService("alarm", new AlarmManagerService(context));
Log.i("SystemServer", "Starting Window Manager.");
wm = WindowManagerService.main(context, power);
sm.addService("window", wm);
...


There is no way getting our service autostarted this way but fortunately there is the android.intent.action.BOOT_COMPLETED intent which is broadcasted when the boot completes. One needs android.permission.RECEIVE_BOOT_COMPLETED permission to receive this intent.

Download the example program from here.

The example program has two parts: an intent receiver catching the BOOT_COMPLETED intent and a primitive service that counts down from 10 in every 3 seconds and logs the counter value. When it counts down, the service dies completely, this is necessary as there is no simple way to uninstall packages in the SDK so this service is not allowed to cause trouble later. Please, note the use of Handler for the timed countdown: you can't stay for too long in onStart() otherwise the application manager will shoot down the service.

Check the boot log below and you will find how the service is started. Each ... represents deleted log entries because meanwhile the boot process is happening and other services write the log.

D/BootCompletedIntentReceiver( 576): android.intent.action.BOOT_COMPLETED
...
D/BootCompletedIntentReceiver( 576): BackgroundService started
...
D/ActivityThread( 576): Creating service aexp.persistentapp.BackgroundService
D/BackgroundService( 576): onStart
I/ActivityManager( 510): Stopping service: {aexp.persistentapp/aexp.persistentapp.BackgroundService}
D/ActivityThread( 576): Stopping service aexp.persistentapp.BackgroundService@400bcb18
D/BackgroundService( 576): onDestroy
...
D/BackgroundService( 576): Counter: 10
...
D/BackgroundService( 576): Counter: 9
D/BackgroundService( 576): Counter: 8
D/BackgroundService( 576): Counter: 7
....
D/BackgroundService( 576): Counter: 6
D/BackgroundService( 576): Counter: 5
...

Friday, February 22, 2008

Android is the most modern application platform

Exceptionally, in this blog entry I intend to talk about philosophy instead of hard software code. There are so many mobile application platforms and some of them like Symbian is pretty widespread. So why the excitement about one more platform which is in addition Java-based - there are a number of Java-based mobile platform starting with J2ME, OSGi R4, Limo and so on. Everybody has one's own answer and I think the Android Developer Challenge is a great factor. However, one paper I wrote about 2 years ago (shameless self-advertisement ...) comes to my mind constantly when I work with Android. In that paper I analyzed mobile middleware patterns (get the presentation for free, the paper unfortunately costs money. Write an e-mail to me and I send you a copy). There are 4 major patterns in mobile middleware (quote from the introduction of the paper).

Context-awareness

Traditional middleware is strictly layered meaning that it shields the applications from events concerning the lower level of the stack. For example if a remote invocation cannot be sent, the middleware layer may retry and eventually send a general error to the service user. Context-awareness means that there is no such shielding and the application is aware of the environment situation. Context changes are inherently asynchronous and are often delivered in the form of events. It is important to note that only the application can decide what context events are important and how to handle them. For example when a business application notices that its cheap and fast proximity connection is no longer available, the application can revert using slower and more expensive cellular connection. The same option may not be available for a game which is not allowed to use more costly bearer and in case of disconnection, an error should be sent to the end user or the application may switch to standalone mode.

Reflection

Reflection generally means that the program is able to make computations on its own structure during its execution (retrieve the current structure, evaluate the structure against environmental constraints then update the structure if necessary). Reflection is a crucial technique in mobile computing, especially, if the application is expected to be context-aware. Even moderate number of context states yield large number of context state combinations. If the application and the middleware are built in monolithic fashion, the application and middleware code must be prepared for all possible context state combinations which quickly becomes intractable. Also, the memory footprint of the monolithic middleware increases with each context state combination handled. In order to keep the footprint minimal and the design of the system clear, the middleware needs to be decomposed into a collection of smaller components. The application chooses just the components it needs and composes the middleware that serves the application the best. In case of context changes, the application evaluates the context transition and possibly changes the instantiated components and/or their configuration.

Off-line access

Disconnection is an inherent property of mobile computing. The reason for disconnection can be physical (no coverage) or social (mobile access is too expensive or not acceptable in the given situation). In order to provide acceptable user experience, operation in disconnected mode must be available. The key technology to achieve disconnected operation is the relocation of relevant data and code to the mobile device. Data relocation can be achieved by pretty established data synchronization techniques.

Asynchronous communication

Networked computing is dominated by solutions following Remote Procedure Call (RPC) semantics. RPC mimics the procedure call on a single processor, the calling procedure is suspended for the duration of the call and execution continues in the called procedure. Therefore RPC is inherently synchronous. Mobile transport networks are characterized by long and variable delays and frequent transmission errors. In this environment communication must be asynchronous ( event-based or messagebased terms are also used for the same concept). This affects the communication semantics the middleware uses. Instead of procedure call semantics, communicationrelated events are delivered to the application.

The paper goes on and designs such a framework for OSGi but that's history. So you can imagine my surprise when I started to work with Android and saw the intent and the related service framework that realizes context-awareness and reflection. The decomposition of the Android application into intent handler activities implement one key element of the reflective vision - standalone components that can be reconnected as the environmental situation demands it - and the intent delivery framework is perfectly suitable for context-aware computing. The intent matching logic in the current Android implementation is pretty simple (exact match of action and categories) but it takes not much imagination to extend it with more sophisticated matching logic if the target platform's capabilities (e.g. power consumption, processor speed) allow it. Intent delivery is slow but service binding based on intent matching - so that the service is selected by an intent and then the selected service is called repeatedly - is sufficiently fast.

In addition, off-line operation - synchronizable content providers - are part of the platform. The only key element Android misses is a comprehensive asynchronous communication framework like queues. The XMPP service that has just been removed can be such a service.

Android is not just another Java-based mobile platform but actually the only platform that adopts the results of the mobile middleware research therefore it is interesting in itself, without the Android Developer Challenge and its prizes. Its fate depends mostly on business factors but it is still important to note that the Android platform's architecture is several steps ahead of the competing platforms.

Now back to coding. :-)

Update: Mr. Fan found this post so interesting that he translated it to Chinese. Here is the Chinese translation (I couldn't check it :-))

Wednesday, February 20, 2008

Location providers - climb the Triglav with Android

Android SDK as it is clearly encourages developers to deal with location-aware applications. Both the location provider and the map part of the system is sufficiently documented and is functional which is pretty much in contrast with e.g. the synchronization and bluetooth parts. Good tutorials have been written about how to use the Android location framework so I wanted to try something else. I wanted to see, how to introduce one's own location data into the system.

Android SDK comes with one built-in location provider called "gps". This location provider moves along a predetermined route - obviously around Google premises. I was curious whether it is possible to introduce one's own location provider into the system therefore I took apart android.server.LocationProviderService. I found that it is relatively easy to introduce one's own location track which is then presented as new location provider to Android applications. Creating a new location provider software module turned out to be more complicated and eventually I did not try that out. LocationProviderService is able to handle location providers with their own Java logic but unfortunately the classes of these providers are loaded by the own classloader of LocationProviderService. This means that it is not possible to deploy the location provider software module as a package (apk), one has to fiddle with the software base of the Android system.

I did not do it because, to my pleasant surprise, Android's own location provider turned to be a pretty versatile simulator itself. Deploying own location providers based on track data really works like a charm. The LocationProviderService loads providers from under the /data/misc/location directory (device-based path). Each subdirectory under this directory is a location provider and the name of the location provider equals to the name of the subdirectory. The files in the subdirectory define the provider. As an example, you can go to the tools subdirectory of the example program package and run pull_gps.bat (emulator needs to be running when you do this). This batch file will fetch the three files associated with the "gps" provider as gps_location, gps_nmea and gps_properties (their original names on the device are location, nmea and properties, respectively, under the subdirectory called gps). The properties file describes the properties of the location provider in an easy to understand textual format. The location file is the last known location and the nmea is the track info in NMEA format. Well, I am not familiar with location format and I don't know what NMEA format is. There are two other possibilities: instead of "nmea" file, you can have "kml" or "track". In case of "kml", the track data is in KML format while "track" is a very simple textual format. I went for the latter and didn't try the former one - erm, I left that execise to the interested reader. ;-)

You can download the example program from here.

The "track" format is simple. It looks like this:
time longitude latitude altitude [bearing speed]

Saturday, February 16, 2008

The rocky road to m5

Even though I slept over the release of m5-rc14 SDK release because of my hurdles with the synchronization framework, I lost no time test driving it. I ported the UI part of the sync example program to M5 which contains three activities and a content provider. The port was much more time consuming than I expected.

I knew in advance from the release notes that I will have troubles with the content provider because android.net.ContentURI has gone and was replaced by android.net.Uri. What I did not expect, however, is that because of minor schema changes, the layout and manifest XML files also require quite an amount of handwork. What was previously

[ListView id="@+id/android:list"


now has to become


[ListView android:id="@+id/android:list"

(XML mangling because of the blog engine limitations)
This required 7 changes in the layout files of the rather primitive application. The Android manifest was also changed for good. Gone are the class and value attributes and now everything is a name.

Before:
[activity class=".SyncExample2" android:label="SyncExample2"
After:
[activity android:name="SyncExample2" android:label="SyncExample2"]
Overall, 6 changes in the 21 lines of the Android manifest file.

Updating the content provider was really painful. The replacement of android.net.ContentURI with android.net.Uri and other class changes required 20 changes in the very simple content provider with 229 lines. I marked all changes as comments so you can review them in the code.

You can download the example program from here.

And when I thought that everything was done, it turned out that the NotificationManager (that I used fortunately at two places only) has also changed. Transient notifications are widgets nowadays and are created like this:

Toast.makeText(this, "message", Toast.LENGTH_SHORT).show();

In order not to make it boring, the documentation remarks merrily that android.widget.Toast may not be the final name of the thing. And I did not even mention the changes of the undocumented API that the content provider list views use - well, one uses undocumented API at one's own risk.

Overall, seemingly minor changes of API "beatification" cause significant work even on smaller programs. As a bonus, you may marvel at the new UI. Installed applications went to the main screen.



And look at this horizontal menu!

Thursday, February 14, 2008

Synchronization in Android

Note: this example program was written before the new m5-rc14 release of the SDK was published and therefore it refers to m3-rc37a SDK release. I intend to investigate the sync issue in the new release too.

The first thing I noted about Android content providers that they support synchronization. This fact has huge implications and shows, how well the Android mobile platform is designed. Synchronization is a key enabler in mobile applications but is often "engineered in" later into the platforms. This approach is about as realistic as "adding security later" to a non security-aware platform.

It is not surprising for an Android-hardened developer that synchronization does not work in the m3-rc37a release and the example program presented here is not able to accomplish any real synchronization. This statement would discourage lesser programmers but not the ones dealing with the Android platform! Decyphering the Android synchronization framework is an interesting exercise in itself even though that framework may change in the future. Key elements are already there and there are some interesting findings regarding synchronization as a service.

You can download the example program from here.



The naive observation that led me to deal with the Android synchronization framework is that the log clearly indicates a running synchronization service. The synchronization service (actually called android.content.ContentService) listens to data connection/disconnection events and logs these state changes. In order to actually activate the sync part of the service (other part of the service deals with the XMPP connection), the following steps should be taken.
  • The sync part of the service must be enabled by manipulating the sync_enabled property in the settings table. This is done by calling android.provider.Settings.Gservices.putString( cr,"sync_enabled","true" ). You noticed it well: the G in Gservices refers to Google. Clearly, Android sync framework programmers considered synchronization a Google service. In order for this call to succeed, the application must have android.permission.WRITE_SETTINGS permission (check the manifest of the example program).
  • Account information must be provided. Interestingly again, the service handling this logic is called GoogleLoginService. ContentService obtains account information from GoogleLoginService and if there are no accounts specified, synchronization stops. This is a bit frightening but in m3-rc37a release the account information is stored in a local SQLite table. Do the following:
    adb shell
    cd data/data/com.google.android.providers.googleapps/databases
    sqlite3 accounts.db

    INSERT INTO "accounts" VALUES(1,'user','password',1);

    .quit
  • The sync engine stores its settings, history and stats in a content provider under the content://sync/ tree. Strangely, this content provider is not implemented at all. The example program provides a naive implementation which is good enough to make the sync service work.
The sync service is ready for work now. In Android, only synchronizable data providers can be synchronized and this makes this whole exercise more valuable than just pure hacking. Content providers must jump some fences to become synchronizable. Pretty few built-in content providers of the SDK are synchronizable (you can examine these lists using the example program's "List sync providers" and "List content provider" menu items).

The steps necessary to make a content provider synchronizable are the following:
  • The content provider must be marked as synchronizable in the manifest (again, the XML fragment is mangled because of the limitations of the blog engine):
    [provider class=".SimpleStringDataProvider" android:authorities="aexp.syncexample.SimpleString"
    android:syncable="true"/]
  • The content provider must be able to provide a temporary version of itself. The method signature is the following:

    public ContentProvider getTemporaryInstance()

    The synchronization engine manipulates the temporary instance of the content provider and changes done on the temporary content provider are merged by another method the content provider must implement:

    public MergeResult merge(SyncContext context, ContentProvider diffs, boolean readOnly)

    This latter method gets the temporary content provider instance after one synchronization step is finished on it and merges it with the main content provider. Meanwhile, collisions may be detected (data changed incompatibly on both sides since the last synchronization) and as a result, synchronization may even be interrupted by the user.
  • And last, but not least, the synchronizable content provider must be able to return the SyncAdapter that is able to synchronize the provider.

    public SyncAdapter getSyncAdapter(Context context)

    The SyncAdapter instance abstracts the entire synchronization engine. The example program provides a dummy implementation.
This is not enough. The database structure of the content provider must be designed in such a way that the SyncAdapter supported by the content provider is able to manipulate it. There are different synchronization strategies and android.provider.SyncColumns provides a number of columns that may be useful implementing them. The content provider in the example program supports the most common one, the timestamp-based synchronization. In this method, the "version" of a data item is expressed by a high-resolution timestamp. Each item timestamped after the timestamp of the last synchronization is candidate for sync processing.

The interaction between the sync engine and the sync adapter can be followed on the the log below:

D/SyncExample( 614): syncDB
D/SyncExample( 614): After syncDB

D/Sync ( 467): running sync operation type: 1 url: content://aexp.syncexample.SimpleString
D/ActivityThread( 467): Installing external provider aexp.syncexample.SimpleString: aexp.syncexample.SimpleStringDataProvider

D/ActivityThread( 467): Installing external provider sync: aexp.syncexample.SyncSettingsProvider
I/Sync ( 467): Starting sync for aexp.syncexample.SimpleString
D/SyncAdapterStub( 614): syncStarting
D/SyncAdapterStub( 614): isReadOnly

D/SyncAdapterStub( 614): getServerDiffs, contentProvider: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, tempContentProvider: android.content.ContentProvider$Transport@400144a0 D/SyncAdapterStub( 614): writeSyncData: aexp.syncexample.SimpleString
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: null

D/SyncAdapterStub( 614): syncEnding

I/Sync ( 467): Ending sync for aexp.syncexample.SimpleString
D/Sync ( 467): finished sync operation type: 1 url: content://aexp.syncexample.SimpleString


  • In the first step, getServerDiffs is called on the SyncAdapter, using the temporary provider as input. In case of the timestamp-based adapter, the client-server communication must establish the timestamp at which this client and server synchronized last, send client changes after that timestamp and obtain changes from the server into the temporary provider.
  • In the second step, the provider's merge method is called with the temporary provider as parameter. At this point the content provider must reconcile the client-side database with the temporary database. During the merging, collisions may be detected when the client and the server both changed a data item since the last synchronization in non-compatible way. Eventually collisions may need to be escalated to the user who may even interrupt the synchronization process.
  • After this step, we have the reconciled database on the client side. The client now needs to notify the server about collision resolution and sends the client diffs one more time using the SyncAdapter's sendClientDiffs method. This step is not present in the log because our primitive synchronization example did not produce collisions.

After all this suffering, I guess, you would like to see a real synchronization session. I have to disappoint you: beside some scattered SyncML DS-related classes, I was not able to find any real sync adapter. Implementing a real synchronization engine takes a lot of time but I happen to have a SyncML-based engine implemented and I intend to put it under Android's sync framework. So stay tuned. :-)

Sunday, February 3, 2008

Double life of a service

Earlier (with much less experience in Android services) I wrote about the two kinds of services behind the android.app.Service abstraction: a long-running service whose lifecycle is independent of the activity that started it (Context.startService) and a service bound by other services and activities by its AIDL interface (Context.bindService). The lifecycle of this second type of service depends on the lifecycles of the entities that bind it, if the service has no more bound client, the application manager is free to destroy it. Meanwhile, the first type of service exists until the clients that started the service stop it or until the service stops itself. In extremely low resource conditions the application manager may shut down the first type of service too but this is rare.

I don't know how about you but the difference between the two types always seemed to be unnatural for me. For starter, I can pretty much imagine a number of useful and practical services that are long-running and clients want to communicate with them at the same time. For example a P2P module would be such a service; it would run in the background doing discovery-related or other long-running tasks and occassionally client applications would connect to it to accomplish P2P operations.

It turns out that this difference is artificial. Android services can exhibit both properties: they can be long-running and can be bound and unbound with AIDL interface operations. This is possible because there is only one service instance, independently of the method the service was started with. I discovered it recently and as there was a topic about it in the Android groups, I decided to quickly put together an example program about it.

You can download the example program from here.

You can download the example program updated for SDK 1.5 from here.

Our example is simple. There is a service that holds a counter and can return that counter by means of an AIDL interface. The service need to be bound by Context.bindService() to obtain client-side interface stub. This is not very useful, however, because the counter does not change, it needs to count forward. In order to do that, we start the same service with Context.startService(). It turns out that the onStart() that follows is invoked on the same service instance. Hence we have a service that conforms to both types: it is long-running and serves interface invocations at the same time.

Simple? I thought so. I implemented a version of the example program and it worked flawlessy except that the service could not be stopped. It turns out that a dual service like that can be stopped with Context.stopService() call only if both lifecycles models allow it. This means that stopService() has no effect on a bound service!

Let's take an example. The client activity allows binding/unbinding, starting/stopping the service manually.



Now let's see what happens if the service is bound, unbound then started and stopped.

D/DUALSERVICECLIENT( 568): bindService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICECLIENT( 568): onServiceConnected
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): unbindService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@40044928
D/DUALSERVICE( 568): onDestroy
D/DUALSERVICECLIENT( 568): startService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICE( 568): onStart
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): stopService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@40048cd0
D/DUALSERVICE( 568): onDestroy

The service lifecycles work nicely: when the service is bound, its onCreate is called, then it is destroyed (onDestroy()) when it is unbound. The same goes for startService-stopService except that onStart() is called after onCreate(), as it should happen.

Now let's see what happens if the sequence is bind-start-stop-unbind!

D/DUALSERVICECLIENT( 568): bindService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICECLIENT( 568): onServiceConnected
D/DUALSERVICECLIENT( 568): startService()
D/DUALSERVICE( 568): onStart
D/DUALSERVICECLIENT( 568): stopService()
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): unbindService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@4004a430
D/DUALSERVICE( 568): onDestroy


ActivityManager reports twice that the service was stopped. Of these two, only the second one is real (leads to the invocation of onDestroy()). As the service abstraction has no onStop() method (that would be counterpart of onStart()), the service is not notified that it was stopped. You can examine that with getting the counter: it continues counting after the service was stopped.

This is annoying enough but a method like stopCounting() on the AIDL interface would solve the problem. My opinion, however, is that the lack of service stop notification will cause problems with less tricky long-running services too. For example a voice recorder service will not be able to correctly free the resources (e.g. close the output file correctly) if it is just stopped by one of its clients or if the ActivityManager shuts it down due to low resource condition. I don't know how hard it would be ti implement stop notification callback (finalizers are the toughest kind of callbacks) but this enhancement was hacked into every application model that missed it originally (like Java applications, for example).