Saturday, December 13, 2008

The Dalvik opcodes

I wanted to continue with my adventures with the Android test framework but I ran into some troubles. With pre-1.0 SDKs my solution was simple in these cases: take apart the SDK's android.jar and decomplile the relevant classes. In 1.0 SDK, however, all the classes in android.jar are just stubs, at least in the version on the PC filesystem. The real classes are in DEX format, on the emulated device's file system.

That's sad news because the DEX format is not particularly well documented. More exactly: undocumented. There are some descriptions floating on the Internet but they are obsolete and inaccurate. Conveniently, the dx tool in Android SDK has some less used options that effectively document this format.

Dx is the utility that turns Java class files into DEX files. Every Android developer uses it regularly, although not everybody may be aware of its existence because the tool is invoked by automatically generated make/ant files. Dx has an option that dumps the content of the DEX file in human-readable format while generating the DEX file. This is the batch script I use to get that dump:

set BASEDIR=
javac %1\*.java
dx --dex --verbose --verbose-dump --dump-to=%BASEDIR%\%1\dexdump.txt --output=%BASEDIR%\%1\classes.dex %BASEDIR%\%1

Put your Java files into a subdirectory (e.g. test1) and invoke the batch script with the name of the directory. Beside the familiar classes.dex, dexdump.txt will be generated. This dump file is so verbose that reverse engineering of the DEX file format becomes something of a feasible project.

With using some test Java classes, the official Android opcode list and a lot of time, my first step was to document the Android opcodes. This is the bytecode the Dalvik virtual machine uses instead of the Java bytecode. If you are familiar with Java bytecode, you will see that the opcode set is pretty similar. Significant difference is that the Dalvik opcodes are register-based while Java bytecode is stack-based.

Click here to access the Dalvik opcode list.

The next step will be to put together a DEX disassembler. That will take some time, see you in 2009 with that!

Tuesday, December 2, 2008

Instrumentation and JUnit

JUnit has been nicely integrated into Android. JUnit classes are specialized to facilitate common Android testing tasks. In this post, I will talk about my experiences with InstrumentationTestRunner and the facilities it provides to integrate Android instrumentation with JUnit test execution.

As we have seen previously, the difficulty in JUnit-instrumentation integration stems from the fact that an instrumentation is an entire Android application, started, managed and terminated for testing purposes. One has to work quite a bit to make sure that the test controller is not terminated along with the application under test. InstrumentationTestRunner solves this problem by launching the Dalvik VM in special, instrumentation mode.

You can download the example program from here.

The test subject is our well known Calculator program that was renamed as Calculator2 so that we do not confuse it with the version coming from previous application packages, other than that it is the same simple program. Under aexp.calculator.tests, you will find two JUnit test classes. There are things to note, however.
  • Instead of TestCase, the test classes inherit from ActivityInstrumentationTestCase, templated by the activity class under test.
  • The constructor of the test class defines the activity under test class precisely.
One such test class or collection of test classes can be executed by InstrumentationTestRunner. Execute the following command (from command prompt).

adb shell am instrument -w aexp.calculator/android.test.InstrumentationTestRunner

The console of the command prompt reads like this:

aexp.calculator.tests.AddFunctionalTests:...
aexp.calculator.tests.SubFunctionalTests:...
Test results for InstrumentationTestRunner=......
Time: 11.448

OK (6 tests)

Meanwhile, the emulator main window flashes with action. Calculator2 is launched 4 times, each time the key input is automatically provided by the test case. All this is arranged by the Android specialization of TestCase/TestRunner.

Please observe the AndroidManifest.xml of the project. Check out the uses-library tag and the use of instrumentation tag, particularly the targetPackage attribute that refers to the application package under test (and not the Java package name of the test classes). InstrumentationTestRunner automagically looks for test classes that fit the test execution criteria (again, check out InstrumentationTestCase documentation for options), there was no need to organize the test classes into suites.

Note that the arrangement of this project is not typical. I mixed the test classes and the activity under test into the same application. This eliminates a lot of deployment problems that caused so much trouble for so many people when playing with the Android test framework. The typical deployment, however, is to place the test classes and application under test into separate application packages. I will get to that in the next post.

Sunday, November 23, 2008

JUnit in Android

Now that we have the instrumentation framework in our basket and we are able to connect instrumentation objects with a test runner, it is time to put a bit of structure into out tests. The number of test cases grows quicky. If there is no way to organize them or they depend on each other in a subtle way, test case maintenance soon becomes a nightmare. Experience shows that in case of a resource-constrained project, test effort is the first to suffer cutbacks and if the test cases are not maintained properly, the team or the developer may soon feel that the test set is too much hassle compared to the value the tests provide. The result is a fallback to the default, "chaotic" way of development with the predictable drop of quality.

The popular JUnit test framework is integrated into Android. JUnit, if used properly, brings two major benefits to the test implementation.
  • JUnit enforces the hierarchical organization of the test cases
  • The pattern JUnit is based on guarantees the independence of the test cases and minimizes their interference.
Entire books have been written about JUnit and unit testing therefore I will restrict myself to the minimum in explaining the test framework. JUnit's main building block is the TestCase class. Actual test case classes are inherited from TestCase. One TestCase child class contains one or more test methods, these test methods are the real test cases and the TestCase descendants are really already test groups. By convention, the test method names are prefixed with "test" (like testThis(), testThat(), etc.). JUnit can discover the test methods by itself if this convention is respected, otherwise the test method names have to be declared one by one.

The most important impact of using JUnit is the way one test method is executed. For each test method, the TestCase descendant class is instantiated, its setUp() method is invoked, then the test method in question is invoked, then the tearDown() method is invoked then the instance becomes garbage. This is repeated for each test method. This guarantees that test methods are independent of each other, their interference is minimized. Of course, "clever" programmers can work around this pattern and implement test cases that depend on each other but this will bring us back to the mess that was to be eliminated in the first place. JUnit also provides ways to collect test classes into test suites and organize these suites hierarchically.

JUnit has been included into Android and Android-specific extensions were provided in the android.test package. Android test cases classes should inherit from android.test.AndroidTestCase instead of junit.framework.TestCase. The main difference between the two is that AndroidTestCase knows about the Context object and provides method to obtain the current Context. As many Android API methods need the Context object, this makes coding of Android test cases much easier. AndroidTestCase already contains one test method that tests the correct setup of the test case class, the testAndroidTestCaseSetupProperly method. This adds one test method to every AndroidTestCase descendant but this fact rarely disturbs the programmer.

Those familiar with the "big", PC-based JUnit implementations will miss a UI-oriented TestRunner. The android.test.AndroidTestRunner does not have any user interface (not even console-based UI). If you want to get any readable feedback out of it, you have to handle callbacks from the test runner.

You can download the example program from here.

Our example program contains a very simple test runner UI, implemented as an Activity. The UI provides just the main progress indicators, details are written into the log so this implementation is nowhere near to the flexible test runners of the "big" JUnit. The test suite contains two test case classes. One of them (the MathTest) is really trivial. It demonstrates, however, the hidden testAndroidTestCaseSetupProperly() test method that explains the two extra test cases (the test runner will display 5 test cases when we have really only 3). The main lesson from the other test case class (ContactTest) is that individual test cases must be made independent from each other. This test case class inserts and deletes the test contact for each test method execution, cleaning up its garbage after each test execution. This is the effort that must be made to minimize interference. ExampleSuite organizes these two test classes into a suite. As we respected the JUnit test method naming conventions, we can leave the discovery of test methods to JUnit.



At this point, we have everything to write structured, easy to maintain tests for Android, using the Android test instrumentation mechanism to drive the UI. This will be my next step.

Wednesday, November 19, 2008

Controlling instrumentation

Those who are familiar with test frameworks could have felt something missing after the previous instrumentation example. The test did not finish very elegantly, both the test launcher and the test program were terminated and the results should be fished out from the logs. The reason is that when the instrumentation finishes, it terminates the entire application, not just the activity that was instrumented. Although application as a notion exists since the beginning of Android's public life, developers usually concentrate on Activities. Application is a bunch of artifacts packed into one apk file, having one manifest file, running in the same JVM slice (seemingly independent JVM simulated for each application by the Dalvik virtual machine), represented by the same Application object. Instrumentation works on application level (as opposed to activity level) and when an instrumentation is finished, the entire application is terminated. That's why our instrumentation launcher did not survive the end of instrumentation - as it lived in the same application, it was also terminated along with the instrumentation object and the target application.

If we want to build a decent automated test framework, we have to control it from a separate application. This example does just that.

You can download the example program from here.

In the package, you will find two application directories. The instrumentation1 directory contains our friends, the slightly modified calculator test target, the instrumentation object and the instrumentation launcher that we will not use now. The instrcontroller directory contains a separate application, our instrumentation controller that starts the instrumentation and receives the test result. Like this:


The only interesting part is the communication between the instrumentation object and the launcher. The instrumentation object has a finish() method that is seemingly capable of sending back a result Bundle to the instrumentation's starter but I was not able to figure out, how that bundle is received by the instrumentation's starter (my Android newsgroup thread also remained unanswered). Therefore I had no better idea than to create a not too elegant do-it-yourself method of sending back the test result using broadcast intents. If anyone knows how to do it right, please notify me.

And before I finish, something completely different.

I received a mail from the Phoload people that they launched their Android content and they asked me to advertise this fact in one of my posts. Certainly, it is a new market, there is a lot of risk taking here so the early birds definitely deserve help - however small mine can be. I did have my concerns, however, how this Phoload site would coexist e.g. with Android Marketplace but the Phoload people are convinced that they can carve out a nice piece of market by offering mobile applications for multiple platforms and by providing easier navigation and more visible place for applications. Let they be right and have success!

Sunday, November 16, 2008

Test-driving Android GUI

It happened again that I received an e-mail from somebody asking whether there is a way to automate the testing of Android user interfaces. I did not know so I went after the issue and to my utter surprise, I discovered that an entire unit test framework has been developed into Android while I looked the other way. In this blog entry, I will describe my experiences with the Android instrumentation framework.

You can download the example program from here.

The idea behind Android instrumentation framework is that there are instrumentation components that resemble a lot the usual Activities. The purpose of these instrumentation components, however, is to test-drive other, normal Activities. Let's see the example program!

Install the package in the bin directory as usual:

adb install instrumentation-debug.apk

If the package has been installed before, you first have to uninstall it. This is new in the 1.0 version of the SDK, previously the new installation would simply overwrite the old one.

adb uninstall aexp.instrumentation

Two applications appear in the application folder. "Calculator" is a primitive calculator application used as test target. You can launch it independently and try it yourself. If you launch "Calculator instrumentation launcher", you will be confronted with a button to launch Calculator instrumentation. Once you push it, the calculator launches but its input keypresses will come from the instrumentation component. You will see the input fields filled up automatically, the addition button pushed and the result displayed. Then both the calculator instance and the launcher will disappear but if you check the logs (adb logcat), you will see that the test was successful.


The key instruction is in CalculatorInstrumentationLauncher.java where the instrumentation component is launched:

startInstrumentation(
new ComponentName(
CalculatorInstrumentationLauncher.this,
CalculatorInstrumentation.class ),
null,
null );

This launches CalculatorInstrumentation component which is a descendant of android.app.Instrumentation. The instrumentation component resembles a normal Activity, except that it is able to drive user input of other activities. This time we launch the Calculator, feed in keypresses (sendCharacterSync()) and invoke one of its method to obtain the result. Note the usage of runOnMainSync() method when we invoke Calculator's method to obtain the result.

It is definitely cool that Android has a user interface-oriented unit testing framework built into the platform. Maybe I am ignorant but I am not aware of any other platform that has this capability. In order to Test Driven Development (TDD), we need a proper unit testing framework and Android has even JUnit integrated. More about that later.

Tuesday, November 4, 2008

Again about synchronization

Once I wrote a post about synchronization in Android on this blog. Although that post is rather obsolete now because the synchronization-related Android API changed a lot, old sins did come back to haunt me. Rushang Shah from CompanionLink Software wrote a kind letter to me because of that old blog entry asking my opinion about their Android-based synchronization solution.

Disclaimer: Although I have no relationship whatsoever with CompanionLink, I do advertise their product now because of that nice letter. :-)

The product is an Android-based synchronization solution for PIM and CRM data. Try it, it is a great product but I use it as a pretext to emphasize again my mania about synchronization as a key mobile enabler. I have already written a post about it, there is also a more detailed paper involved there. My claim is that synchronization should not be considered as part of PIM or CRM systems, it is a general enabler because mobility inherently requires disconnected operation due to varying network conditions. Therefore my dream synchronization solution is not restricted to predefined data types but can be used as a building block in any application.

In order to demonstrate my point, here is a larger Android client-Java server application I wrote some time back. The application itself probably cannot be executed on the latest Android platform because the API changed since and I was lazy to bring up the application to the latest API. It does demonstrate, however, that synchronization needs can be identified in any mobile application, for any type of mobile data.

So Rushang, if you listen to me, you turn your product into a general-purpose synchronization component.

Tuesday, May 13, 2008

Expandable lists

Multilevel, expandable lists are useful user interface elements and Android does support two-level expandable lists. This fact would not be worth a blog entry if there were no traps using the widget.

Click here to download the example program.

The example program implements a simple, two-level expandable list to display color shade information.



The first thing to note is that our Activity extends android.app.ExpandableListActivity. Otherwise setting up the list is pretty similar to ordinary ListActivity classes. The tricky part is the android.widget.SimpleExpandableListAdapter.

SimpleExpandableListAdapter expListAdapter =
new SimpleExpandableListAdapter(

this,

createGroupList(), // groupData describes the first-level entries

R.layout.child_row, // Layout for the first-level entries
new
String[] { "colorName" }, // Key in the groupData maps to display

new int[] { R.id.childname }, // Data under "colorName" key goes into this TextView
createChildList(), // childData describes second-level entries
R.layout.child_row, // Layout for second-level entries

new String[] { "shadeName", "rgb" }, // Keys in childData maps to display

new int[] { R.id.childname, R.id.rgb } // Data under the keys above go into these TextViews

);


First of all, this adapter requires somewhat complicated (although pretty well-documented) data structures. There are two of them, one describing the first-level group elements, the other describing the second-level string elements.

Here is the first one:
List->Map
Every map describes one first-level group element. The keys in the Map are arbitrary but are specified for SimpleExpandableListAdapter so that it can map the keys in the Map to widget IDs. In our case, the data under key name "colourName" will be set as the value of the TextView with "childname" ID.

The second one is a bit more complicated.
List->List->Map
Every entry in the first List represents a group. Entries in the second List represent entries of the group and each such entry is a Map that describes one child, similarly to the description of groups. In our case, the child Map contains two entries (keys "shadeName" and "rgb") that go to TextViews ("childname" and "rgb").

So far so good. Unfortunately, SimpleExpandableListAdapter has its tricks. First of all, the adapter takes the description of one row from a layout (child_row in our case, located under res/layout/child_row.xml). When the adapter draws the expand button onto the row, it does not consider the content of the row, it simply draws the button over the row. That's the reason child_row.xml defines generous left padding for the first TextView. While this behaviour can be considered just an oddity, the fact that SimpleExpandableListAdapter must be created with identical views for first- and second-level lines is a plain bug that existed at least since m3-rc37a. Having group- and child rows with different styles is a cool and useful feature that is supported by SimpleExpandableListAdapter but does not work.

Replace this fragment:

R.layout.child_row, // Layout for the first-level entries
new String[] { "colorName" }, // Key in the groupData maps to display

new int[] { R.id.childname }, // Data under "colorName" key goes into this TextView


with this one:

R.layout.group_row, // Layout for the first-level entries
new String[] { "colorName" }, // Key in the groupData maps to display

new int[] { R.id.groupname }, // Data under "colorName" key goes into this TextView


and you will see, how the expandable list gets confused.

Monday, April 28, 2008

Custom views

Now that I entertained myself (and hopefully the readers of the blog as well) with custom adapters and animations, I set out to an even more adventurous exercise. In Android, entirely new view classes could be created and used in conjunction with XML layout files, similarly to built-in views. Views have their own lifecycle. Intererested readers should consult the documentation of android.view.View class. Very shortly: views may handle the event when their inflation is finished, when they are being measured by the layout manager, when they are being laid out, when their size change, when they gain or lose focus, when they are attached to and detached from a window and when the window they are attached to changes visibility. This is pretty complex so I chose an easier way to introduce custom views by extending a built-in view.

The scenario which absolutely requires custom views is the following: we have a list of alarm events and some of them requires the user's acknowledgement. There is a flashing exclamation mark beside such alarms. Like this (except that the exclamation mark is flashing:)


You can download the example program from here.

We will support the list with a custom adapter, as we have seen previously. A single row for this adapter looks like this (located under res/layout/alarm_row.xml, XML mangling because of blog engine limitation).

[?xml version="1.0" encoding="utf-8"?]
[LinearLayout
xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:orientation="horizontal"]

[TextView android:id="@+id/alarmtext"
android:textSize="16px"
android:layout_width="280px"
android:layout_height="wrap_content"/]

[aexp.customview.AlarmingView
android:id="@+id/alarming"
android:layout_width="wrap_content"
android:layout_height="wrap_content"/]
[/LinearLayout]

TextView is our old friend but the second view in the row is our own. If you check the aexp.customview.AlarmingView class, you will see that it is a descendant of ImageView. We still have to define three constructors (one for the creation from code, 2 others for inflating from XML) but the rest of the view lifecycle handling is inherited from ImageView. The flashing effect is achieved by periodically updating the drawable resource displayed by the underlying ImageView.

There is one point worth highlighting. There is a single private class handling the periodic flashing (coded as singleton for all AlarmingView instances to save resources and achieve synchronous flashing) but that class does not use the java.lang.Thread mechanism. If you code it that way (I did first :-( ), the application is shut down with an error message that views are not thread-safe and can be manipulated only from the thread that created them. Hence, we create the private class instance when the first AnimatedView is created and therefore the Handler instance in the private class will also be created in the context of the thread that creates the AnimatedView in the first place. Then we request this Handler instance to update the state of our views periodically.

Tuesday, April 22, 2008

Animated views

I have to admit, I am aesthetically challenged. Creating a nice GUI is not my game and I respected those who could design such things. Because of this limitation of mine, I generally look down on glittering UIs with contempt. But even I was moved - even if only a bit - by Android's animation capabilities.

Android is able to animate any View object and each and every View has a startAnimation() method that launches animation. Animations exist as independent objects that can be applied to the Views. Animation objects can even be populated from XML resources that live in the res/anim directory.

You can download the example program from here.

Let's look at the animation file below (again, XML mangling due to the limitations of the blog engine). This file can be found as res/anim/magnify.xml in the example program bundle.

[?xml version="1.0" encoding="utf-8"?]
[scale xmlns:android="http://schemas.android.com/apk/res/android"
android:fromXScale="0.5"
android:toXScale="2.0"
android:fromYScale="0.5"
android:toYScale="2.0"
android:pivotX="0%"
android:pivotY="50%"
android:startOffset="0"
android:duration="400"
android:fillBefore="true" /]

This is a scaling animation. The attributes mean the followoing:
  • The content of the view is actually shrinken first (fromXScale, fromYScale=0.5) to half before it starts to grow to double its standard size and 4 times of the initial size (0.5->2.0). This is done both in X and Y axis.
  • The pivot point from which the object grows is 0% in the X axis. This means that the leftmost points of the object will stay in place and the object grows rightward to double size. Meanwhile, the pivot point on the Y axis is the middle of the object (50%), this means that the object will grow up and down from its center line.
  • The animation starts immediately (startOffset=0) and finishes in 400 msec.
Once we have an animation object, we can apply it to any view. In the example program, I created a simple list and applied the animation to the TextView list elements whenever one list element is selected (either by moving the selection with the arrow keys or clicking at list elements). Two points are worth noting:
  • Just because we animate, the list layout is not recalculated. Hence the generous top and bottom paddings around the list elements. There is enough space provided for the TextView to grow.
  • In the list row layout (res/layout/row.xml), the layout_width is set to fill_parent. This is seemingly random choice but actually, the program does not work well with wrap_content as layout_width. Whenever a list element is selected, its color changes to highlight. If the width is wrap_content, then the size of the TextView (hence the higlighted screen area) is just the length of the text. When the animation starts, the text grows past this size and the end of the text becomes unreadable. This is very important therefore to allow the list row to occupy the entire available width because then the entire row will be highlighted and the animation cannot grow larger than the highlighted area.
At last, two pictures about the beginning and the end of the animation.


Monday, April 21, 2008

Custom widget adapter based on XML layout

Lex from anddev.org pointed out an annoying property of my custom Weather adapter example: the view row the Weather objects are transformed to are encoded in Java instead of being defined in XML layout resource. This makes the weather display style harder to modify and every application using the WeatherAdapter has the same weather screen layout.

You can download the example program from here.

I created a new version of WeatherAdapter that takes an additional parameter in its constructor, an ID of a composite view (a LinearLayout in our case) that has three children views having the IDs "city", "temperature" and "sky" respectively. The code is flexible enough to handle the case when one or more of these child views are missing, in this case the field will simply not set from the Weather object. Check the res/layout/weather_row.xml file for an example.

The new version is indeed more flexible which is demonstrated by the bit more complicated display of weather rows. Thanks for the feedback, Lex, it really made this example program more valuable.


Friday, April 18, 2008

Custom widget adapters

In an earlier post, I wrote about the SimpleAdapter and how SimpleAdapter allows significant flexibility when laying out list items. SimpleAdapter is great but then I became curious what it takes to write an adapter.

Adapters are simple devices. On one side of the adapter is a data structure like a Java object storing data. SimpleAdapter handles Java objects that can be meaningfully translated into Strings by invoking the objects' toString() method (every Java object supports that but for quite many of them, the toString() format is not meaningful for the end user). On the other side of the adapter, there is a View that the data structure was transformed into. That View is displayed to the user. As we use Adapters to supports list views, the Adapter handles lists of Java objects (that are eventually transformed into a list of Views).

Android's built-in adapters are sufficiently versatile but it is often handy to create a custom adapter. Let's look at the following example.



You can download the example code from here.

The example is a simple weather display. A weather entry consists of 3 data items: name of the city (String), temperature in the city in degrees (integer) and an icon showing whether the sky is sunny, overcast or it is raining. The first two entries could be handled using SimpleAdapter but the third cannot: SimpleAdapter does not handle icons. Therefore we create our own adapter, called WeatherAdapter that takes list of Weather objects storing the weather info and turns that list into Views that can be rendered as list rows.

The most important part of the trick happens in a private class in WeatherAdapter.java called WeatherAdapterView. WeatherAdapter does nothing more than manages a list of WeatherAdapterViews. WeatherAdapterView is the View the Weather data object is mapped to. It is itself a composite View, composed by a LinearLayout. The LinearLayout is set up programmatically (as opposed to an XML layout) and contains two TextViews and one ImageView. The ImageView encapsulates the icon. It is worth checking how the icon images are referenced in Weather.java: as the icons are in the res/drawable directory, R.java has integer IDs for them. The getSkyResource() method in the Weather class just returns this resource ID based on the sky member variable of the Weather class.

Sunday, March 16, 2008

My first meeting with the SimpleAdapter widget

Such a shame. Normally, Android developers have their SimpleAdapter adventures right at the beginning of their Android development career, meanwhile I met this beast just now. I wouldn't say that this UI feature is overdocumented and that there are truckloads of example programs out there (if there are, Google does not find them) so I decided to share my experiences with whoever cares to read this blog entry.

It all started with a very simple wish that I wanted to create a list like this:


Each list entry has two lines and the lines are styled differently. Fortunately, there is an example program at the ListActivity documentation page which happens not to work and takes the input behind the list from a database. So it took me some time to find my friend, the SimpleAdapter widget and to find out, how to use it.

You can download the example program from here.

The root of the problem is that each element of this list is a complex view, consisting of two TextViews (XML mangling because of blog engine limitations):

[?xml version="1.0" encoding="utf-8"?]
[LinearLayout
xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:orientation="vertical"]

[TextView android:id="@+id/text1"
android:textSize="16px"
android:textStyle="bold"
android:layout_width="fill_parent"
android:layout_height="wrap_content"/]

[TextView android:id="@+id/text2"
android:textSize="12px"
android:textStyle="italic"
android:layout_width="fill_parent"
android:layout_height="wrap_content"/]
[/LinearLayout]

The corresponding SimpleAdapter maps an element from list of objects to this view.

SimpleAdapter notes = new SimpleAdapter(
this,
list,
R.layout.main_item_two_line_row,
new String[] { "line1","line2" },
new int[] { R.id.text1, R.id.text2 } );

Now this is not a trivial line of code. What it says is:
  • "list" is a reference to an object implementing the List interface. Each element in this list is an object implementing the Map interface. In our implementation example, the list is ArrayList and the elements in this list are HashMaps.
  • The layout describing one row in the main list is defined in layout/main_item_two_line_row.xml file.
  • In each individual HashMap, there are two key-value pairs. The keys are "line1" and "line2", the corresponding values are arbitrary objects whose toString() method yields the value displayed in the list row. In our case, the values are Strings.
  • Values stored with key "line1" will be displayed in the TextView whose id is "text1". Similarly, "line2" Map key is associated with "text2" TextView.
You can try the program following the instructions in the Start here entry. You can add items to the list using the menu.

Tuesday, March 11, 2008

Observing content

Android documentation makes the point about content providers that they are perfect tool for hiding data access details. The Android content provider API has a clever little feature, however, that allows applications (and services) to observe changes of a dataset. The use case is that some component manipulates the persistent dataset that other components depend upon. The usual pattern is that the component makes the changes then invokes some sort of notification interface so that the dependent components are aware that the dataset was changed. If such a strong coupling is not possible, the dependent components may even need to poll for changes.

The Android content provider framework allows a much more elegant design. As datasets are identified by unique URIs, it is possible to ask for notifications if a certain URI is changed. The framework is the following.

  1. Well-behaving content providers are expected to notify the content resolver if they do something that may potentially change their dataset. This includes insert and update operations but registering database cursors returned by query operations is also necessary because cursors may be used to update data.
  2. Components interested in dataset changes register at the content resolver. If any URI is manipulated that they registrated for, they get notification.
You can download the example program from here.

Click here to access the example program adapted to Android SDK 2.3.3.

The content observer framework depends on the cooperation of content providers and content observers. Content providers are expected to notify content resolver that they have updated the dataset. You can observe this in our well-known simple provider, SimpleStringDataProvider.java.

In insert method:

getContext().getContentResolver().notifyChange(uri, null);
This notifies all the observers registered for that particular URI that change happened.

In query method:

c.setNotificationUri(getContext().getContentResolver(), url);

where c is reference to a Cursor object. If the data behind the cursor's position is updated (Cursor.update*(), Cursor.commitUpdates()), the given URI will be notified.

The other side of the picture is the observer. In this simple example, the observer is located in the DPObserver activity. As the observer is unregistered using the observer object reference, the observer is registered at onStart() and unregistered at onStop() because after onStop() the state of the activity may be lost.

In registerContentObservers:

ContentResolver cr = getContentResolver();
stringsObserver = new StringsContentObserver( handler );
cr.registerContentObserver( SimpleString.Strings.CONTENT_URI, true, stringsObserver );


We requested notification for the content URI allocated to the Strings provider plus all its descendants. This is important because e.g. the Strings provider generates URIs for newly allocated items like the following: content://aexp.dpobserver.SimpleString/strings/[id] where [id] is an integer number. These URIs are not equals to the Strings provider's base URI, these are descendants of the base URI.

The observer object is a child of the android.database.ContentObserver class. Its onChange() method is called by the content resolver if there is a change of the data behind the URI the observer was registered for. Unfortunately the URI that triggered the invocation is not passed to the method. The onChange() method is called in the context of a Handler. In our case, this Handler uses the UI thread of the activity.

You can try the application by launching the DPObserver activity and monitoring the log (adb logcat from the command prompt). Whenever you add a new data item (by clicking a menu item), the onChange() will generate a log message.

Now comes the interesting bit. We actually registered another observer, that one for the URI of content://contacts/people and descendants which belongs to the Contacts application. Launch now the Contacts application and add a new user. You will see in the log that our activity got called (except if it was stopped by the application manager because it went into the background). At this point, we could access the Contacts content provider and find out, what entry was added. It would be much more easier if we had the URI of the added entry but finding the new entry is possible by looking for the entry with the highest ID. We could hook onto Contacts database manipulations with our own functionality. Isn't that interesting? ;-)

Friday, February 29, 2008

Autostarting services

Some privileged services start automatically in Android while the lowly applications I have written so far had to be started manually. I was not satisfied with the situation so I went after the Android bootup sequence.

The Linux layer under the JVM boots up normally, using the init script but I was not interested in that. I found that the Java services are all started by the System server (android.server.SystemServer) which does launch each system server in a hardwired way.

...
Log.i("SystemServer", "Starting Alarm Manager.");
sm.addService("alarm", new AlarmManagerService(context));
Log.i("SystemServer", "Starting Window Manager.");
wm = WindowManagerService.main(context, power);
sm.addService("window", wm);
...


There is no way getting our service autostarted this way but fortunately there is the android.intent.action.BOOT_COMPLETED intent which is broadcasted when the boot completes. One needs android.permission.RECEIVE_BOOT_COMPLETED permission to receive this intent.

Download the example program from here.

The example program has two parts: an intent receiver catching the BOOT_COMPLETED intent and a primitive service that counts down from 10 in every 3 seconds and logs the counter value. When it counts down, the service dies completely, this is necessary as there is no simple way to uninstall packages in the SDK so this service is not allowed to cause trouble later. Please, note the use of Handler for the timed countdown: you can't stay for too long in onStart() otherwise the application manager will shoot down the service.

Check the boot log below and you will find how the service is started. Each ... represents deleted log entries because meanwhile the boot process is happening and other services write the log.

D/BootCompletedIntentReceiver( 576): android.intent.action.BOOT_COMPLETED
...
D/BootCompletedIntentReceiver( 576): BackgroundService started
...
D/ActivityThread( 576): Creating service aexp.persistentapp.BackgroundService
D/BackgroundService( 576): onStart
I/ActivityManager( 510): Stopping service: {aexp.persistentapp/aexp.persistentapp.BackgroundService}
D/ActivityThread( 576): Stopping service aexp.persistentapp.BackgroundService@400bcb18
D/BackgroundService( 576): onDestroy
...
D/BackgroundService( 576): Counter: 10
...
D/BackgroundService( 576): Counter: 9
D/BackgroundService( 576): Counter: 8
D/BackgroundService( 576): Counter: 7
....
D/BackgroundService( 576): Counter: 6
D/BackgroundService( 576): Counter: 5
...

Friday, February 22, 2008

Android is the most modern application platform

Exceptionally, in this blog entry I intend to talk about philosophy instead of hard software code. There are so many mobile application platforms and some of them like Symbian is pretty widespread. So why the excitement about one more platform which is in addition Java-based - there are a number of Java-based mobile platform starting with J2ME, OSGi R4, Limo and so on. Everybody has one's own answer and I think the Android Developer Challenge is a great factor. However, one paper I wrote about 2 years ago (shameless self-advertisement ...) comes to my mind constantly when I work with Android. In that paper I analyzed mobile middleware patterns (get the presentation for here, the paper here). There are 4 major patterns in mobile middleware (quote from the introduction of the paper).

Context-awareness

Traditional middleware is strictly layered meaning that it shields the applications from events concerning the lower level of the stack. For example if a remote invocation cannot be sent, the middleware layer may retry and eventually send a general error to the service user. Context-awareness means that there is no such shielding and the application is aware of the environment situation. Context changes are inherently asynchronous and are often delivered in the form of events. It is important to note that only the application can decide what context events are important and how to handle them. For example when a business application notices that its cheap and fast proximity connection is no longer available, the application can revert using slower and more expensive cellular connection. The same option may not be available for a game which is not allowed to use more costly bearer and in case of disconnection, an error should be sent to the end user or the application may switch to standalone mode.

Reflection

Reflection generally means that the program is able to make computations on its own structure during its execution (retrieve the current structure, evaluate the structure against environmental constraints then update the structure if necessary). Reflection is a crucial technique in mobile computing, especially, if the application is expected to be context-aware. Even moderate number of context states yield large number of context state combinations. If the application and the middleware are built in monolithic fashion, the application and middleware code must be prepared for all possible context state combinations which quickly becomes intractable. Also, the memory footprint of the monolithic middleware increases with each context state combination handled. In order to keep the footprint minimal and the design of the system clear, the middleware needs to be decomposed into a collection of smaller components. The application chooses just the components it needs and composes the middleware that serves the application the best. In case of context changes, the application evaluates the context transition and possibly changes the instantiated components and/or their configuration.

Off-line access

Disconnection is an inherent property of mobile computing. The reason for disconnection can be physical (no coverage) or social (mobile access is too expensive or not acceptable in the given situation). In order to provide acceptable user experience, operation in disconnected mode must be available. The key technology to achieve disconnected operation is the relocation of relevant data and code to the mobile device. Data relocation can be achieved by pretty established data synchronization techniques.

Asynchronous communication

Networked computing is dominated by solutions following Remote Procedure Call (RPC) semantics. RPC mimics the procedure call on a single processor, the calling procedure is suspended for the duration of the call and execution continues in the called procedure. Therefore RPC is inherently synchronous. Mobile transport networks are characterized by long and variable delays and frequent transmission errors. In this environment communication must be asynchronous ( event-based or messagebased terms are also used for the same concept). This affects the communication semantics the middleware uses. Instead of procedure call semantics, communicationrelated events are delivered to the application.

The paper goes on and designs such a framework for OSGi but that's history. So you can imagine my surprise when I started to work with Android and saw the intent and the related service framework that realizes context-awareness and reflection. The decomposition of the Android application into intent handler activities implement one key element of the reflective vision - standalone components that can be reconnected as the environmental situation demands it - and the intent delivery framework is perfectly suitable for context-aware computing. The intent matching logic in the current Android implementation is pretty simple (exact match of action and categories) but it takes not much imagination to extend it with more sophisticated matching logic if the target platform's capabilities (e.g. power consumption, processor speed) allow it. Intent delivery is slow but service binding based on intent matching - so that the service is selected by an intent and then the selected service is called repeatedly - is sufficiently fast.

In addition, off-line operation - synchronizable content providers - are part of the platform. The only key element Android misses is a comprehensive asynchronous communication framework like queues. The XMPP service that has just been removed can be such a service.

Android is not just another Java-based mobile platform but actually the only platform that adopts the results of the mobile middleware research therefore it is interesting in itself, without the Android Developer Challenge and its prizes. Its fate depends mostly on business factors but it is still important to note that the Android platform's architecture is several steps ahead of the competing platforms.

Now back to coding. :-)

Update: Mr. Fan found this post so interesting that he translated it to Chinese. Here is the Chinese translation (I couldn't check it :-))

Wednesday, February 20, 2008

Location providers - climb the Triglav with Android

Android SDK as it is clearly encourages developers to deal with location-aware applications. Both the location provider and the map part of the system is sufficiently documented and is functional which is pretty much in contrast with e.g. the synchronization and bluetooth parts. Good tutorials have been written about how to use the Android location framework so I wanted to try something else. I wanted to see, how to introduce one's own location data into the system.

Android SDK comes with one built-in location provider called "gps". This location provider moves along a predetermined route - obviously around Google premises. I was curious whether it is possible to introduce one's own location provider into the system therefore I took apart android.server.LocationProviderService. I found that it is relatively easy to introduce one's own location track which is then presented as new location provider to Android applications. Creating a new location provider software module turned out to be more complicated and eventually I did not try that out. LocationProviderService is able to handle location providers with their own Java logic but unfortunately the classes of these providers are loaded by the own classloader of LocationProviderService. This means that it is not possible to deploy the location provider software module as a package (apk), one has to fiddle with the software base of the Android system.

I did not do it because, to my pleasant surprise, Android's own location provider turned to be a pretty versatile simulator itself. Deploying own location providers based on track data really works like a charm. The LocationProviderService loads providers from under the /data/misc/location directory (device-based path). Each subdirectory under this directory is a location provider and the name of the location provider equals to the name of the subdirectory. The files in the subdirectory define the provider. As an example, you can go to the tools subdirectory of the example program package and run pull_gps.bat (emulator needs to be running when you do this). This batch file will fetch the three files associated with the "gps" provider as gps_location, gps_nmea and gps_properties (their original names on the device are location, nmea and properties, respectively, under the subdirectory called gps). The properties file describes the properties of the location provider in an easy to understand textual format. The location file is the last known location and the nmea is the track info in NMEA format. Well, I am not familiar with location format and I don't know what NMEA format is. There are two other possibilities: instead of "nmea" file, you can have "kml" or "track". In case of "kml", the track data is in KML format while "track" is a very simple textual format. I went for the latter and didn't try the former one - erm, I left that execise to the interested reader. ;-)

You can download the example program from here.

The "track" format is simple. It looks like this:
time longitude latitude altitude [bearing speed]

Saturday, February 16, 2008

The rocky road to m5

Even though I slept over the release of m5-rc14 SDK release because of my hurdles with the synchronization framework, I lost no time test driving it. I ported the UI part of the sync example program to M5 which contains three activities and a content provider. The port was much more time consuming than I expected.

I knew in advance from the release notes that I will have troubles with the content provider because android.net.ContentURI has gone and was replaced by android.net.Uri. What I did not expect, however, is that because of minor schema changes, the layout and manifest XML files also require quite an amount of handwork. What was previously

[ListView id="@+id/android:list"


now has to become


[ListView android:id="@+id/android:list"

(XML mangling because of the blog engine limitations)
This required 7 changes in the layout files of the rather primitive application. The Android manifest was also changed for good. Gone are the class and value attributes and now everything is a name.

Before:
[activity class=".SyncExample2" android:label="SyncExample2"
After:
[activity android:name="SyncExample2" android:label="SyncExample2"]
Overall, 6 changes in the 21 lines of the Android manifest file.

Updating the content provider was really painful. The replacement of android.net.ContentURI with android.net.Uri and other class changes required 20 changes in the very simple content provider with 229 lines. I marked all changes as comments so you can review them in the code.

You can download the example program from here.

And when I thought that everything was done, it turned out that the NotificationManager (that I used fortunately at two places only) has also changed. Transient notifications are widgets nowadays and are created like this:

Toast.makeText(this, "message", Toast.LENGTH_SHORT).show();

In order not to make it boring, the documentation remarks merrily that android.widget.Toast may not be the final name of the thing. And I did not even mention the changes of the undocumented API that the content provider list views use - well, one uses undocumented API at one's own risk.

Overall, seemingly minor changes of API "beatification" cause significant work even on smaller programs. As a bonus, you may marvel at the new UI. Installed applications went to the main screen.



And look at this horizontal menu!

Thursday, February 14, 2008

Synchronization in Android

Note: this example program was written before the new m5-rc14 release of the SDK was published and therefore it refers to m3-rc37a SDK release. I intend to investigate the sync issue in the new release too.

The first thing I noted about Android content providers that they support synchronization. This fact has huge implications and shows, how well the Android mobile platform is designed. Synchronization is a key enabler in mobile applications but is often "engineered in" later into the platforms. This approach is about as realistic as "adding security later" to a non security-aware platform.

It is not surprising for an Android-hardened developer that synchronization does not work in the m3-rc37a release and the example program presented here is not able to accomplish any real synchronization. This statement would discourage lesser programmers but not the ones dealing with the Android platform! Decyphering the Android synchronization framework is an interesting exercise in itself even though that framework may change in the future. Key elements are already there and there are some interesting findings regarding synchronization as a service.

You can download the example program from here.



The naive observation that led me to deal with the Android synchronization framework is that the log clearly indicates a running synchronization service. The synchronization service (actually called android.content.ContentService) listens to data connection/disconnection events and logs these state changes. In order to actually activate the sync part of the service (other part of the service deals with the XMPP connection), the following steps should be taken.
  • The sync part of the service must be enabled by manipulating the sync_enabled property in the settings table. This is done by calling android.provider.Settings.Gservices.putString( cr,"sync_enabled","true" ). You noticed it well: the G in Gservices refers to Google. Clearly, Android sync framework programmers considered synchronization a Google service. In order for this call to succeed, the application must have android.permission.WRITE_SETTINGS permission (check the manifest of the example program).
  • Account information must be provided. Interestingly again, the service handling this logic is called GoogleLoginService. ContentService obtains account information from GoogleLoginService and if there are no accounts specified, synchronization stops. This is a bit frightening but in m3-rc37a release the account information is stored in a local SQLite table. Do the following:
    adb shell
    cd data/data/com.google.android.providers.googleapps/databases
    sqlite3 accounts.db

    INSERT INTO "accounts" VALUES(1,'user','password',1);

    .quit
  • The sync engine stores its settings, history and stats in a content provider under the content://sync/ tree. Strangely, this content provider is not implemented at all. The example program provides a naive implementation which is good enough to make the sync service work.
The sync service is ready for work now. In Android, only synchronizable data providers can be synchronized and this makes this whole exercise more valuable than just pure hacking. Content providers must jump some fences to become synchronizable. Pretty few built-in content providers of the SDK are synchronizable (you can examine these lists using the example program's "List sync providers" and "List content provider" menu items).

The steps necessary to make a content provider synchronizable are the following:
  • The content provider must be marked as synchronizable in the manifest (again, the XML fragment is mangled because of the limitations of the blog engine):
    [provider class=".SimpleStringDataProvider" android:authorities="aexp.syncexample.SimpleString"
    android:syncable="true"/]
  • The content provider must be able to provide a temporary version of itself. The method signature is the following:

    public ContentProvider getTemporaryInstance()

    The synchronization engine manipulates the temporary instance of the content provider and changes done on the temporary content provider are merged by another method the content provider must implement:

    public MergeResult merge(SyncContext context, ContentProvider diffs, boolean readOnly)

    This latter method gets the temporary content provider instance after one synchronization step is finished on it and merges it with the main content provider. Meanwhile, collisions may be detected (data changed incompatibly on both sides since the last synchronization) and as a result, synchronization may even be interrupted by the user.
  • And last, but not least, the synchronizable content provider must be able to return the SyncAdapter that is able to synchronize the provider.

    public SyncAdapter getSyncAdapter(Context context)

    The SyncAdapter instance abstracts the entire synchronization engine. The example program provides a dummy implementation.
This is not enough. The database structure of the content provider must be designed in such a way that the SyncAdapter supported by the content provider is able to manipulate it. There are different synchronization strategies and android.provider.SyncColumns provides a number of columns that may be useful implementing them. The content provider in the example program supports the most common one, the timestamp-based synchronization. In this method, the "version" of a data item is expressed by a high-resolution timestamp. Each item timestamped after the timestamp of the last synchronization is candidate for sync processing.

The interaction between the sync engine and the sync adapter can be followed on the the log below:

D/SyncExample( 614): syncDB
D/SyncExample( 614): After syncDB

D/Sync ( 467): running sync operation type: 1 url: content://aexp.syncexample.SimpleString
D/ActivityThread( 467): Installing external provider aexp.syncexample.SimpleString: aexp.syncexample.SimpleStringDataProvider

D/ActivityThread( 467): Installing external provider sync: aexp.syncexample.SyncSettingsProvider
I/Sync ( 467): Starting sync for aexp.syncexample.SimpleString
D/SyncAdapterStub( 614): syncStarting
D/SyncAdapterStub( 614): isReadOnly

D/SyncAdapterStub( 614): getServerDiffs, contentProvider: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: aexp.syncexample.SimpleStringDataProvider@40014470
D/SIMPLESTRINGDATAPROVIDER( 614): merge, tempContentProvider: android.content.ContentProvider$Transport@400144a0 D/SyncAdapterStub( 614): writeSyncData: aexp.syncexample.SimpleString
D/SIMPLESTRINGDATAPROVIDER( 614): merge, diffs: null

D/SyncAdapterStub( 614): syncEnding

I/Sync ( 467): Ending sync for aexp.syncexample.SimpleString
D/Sync ( 467): finished sync operation type: 1 url: content://aexp.syncexample.SimpleString


  • In the first step, getServerDiffs is called on the SyncAdapter, using the temporary provider as input. In case of the timestamp-based adapter, the client-server communication must establish the timestamp at which this client and server synchronized last, send client changes after that timestamp and obtain changes from the server into the temporary provider.
  • In the second step, the provider's merge method is called with the temporary provider as parameter. At this point the content provider must reconcile the client-side database with the temporary database. During the merging, collisions may be detected when the client and the server both changed a data item since the last synchronization in non-compatible way. Eventually collisions may need to be escalated to the user who may even interrupt the synchronization process.
  • After this step, we have the reconciled database on the client side. The client now needs to notify the server about collision resolution and sends the client diffs one more time using the SyncAdapter's sendClientDiffs method. This step is not present in the log because our primitive synchronization example did not produce collisions.

After all this suffering, I guess, you would like to see a real synchronization session. I have to disappoint you: beside some scattered SyncML DS-related classes, I was not able to find any real sync adapter. Implementing a real synchronization engine takes a lot of time but I happen to have a SyncML-based engine implemented and I intend to put it under Android's sync framework. So stay tuned. :-)

Sunday, February 3, 2008

Double life of a service

Earlier (with much less experience in Android services) I wrote about the two kinds of services behind the android.app.Service abstraction: a long-running service whose lifecycle is independent of the activity that started it (Context.startService) and a service bound by other services and activities by its AIDL interface (Context.bindService). The lifecycle of this second type of service depends on the lifecycles of the entities that bind it, if the service has no more bound client, the application manager is free to destroy it. Meanwhile, the first type of service exists until the clients that started the service stop it or until the service stops itself. In extremely low resource conditions the application manager may shut down the first type of service too but this is rare.

I don't know how about you but the difference between the two types always seemed to be unnatural for me. For starter, I can pretty much imagine a number of useful and practical services that are long-running and clients want to communicate with them at the same time. For example a P2P module would be such a service; it would run in the background doing discovery-related or other long-running tasks and occassionally client applications would connect to it to accomplish P2P operations.

It turns out that this difference is artificial. Android services can exhibit both properties: they can be long-running and can be bound and unbound with AIDL interface operations. This is possible because there is only one service instance, independently of the method the service was started with. I discovered it recently and as there was a topic about it in the Android groups, I decided to quickly put together an example program about it.

You can download the example program from here.

You can download the example program updated for SDK 1.5 from here.

Our example is simple. There is a service that holds a counter and can return that counter by means of an AIDL interface. The service need to be bound by Context.bindService() to obtain client-side interface stub. This is not very useful, however, because the counter does not change, it needs to count forward. In order to do that, we start the same service with Context.startService(). It turns out that the onStart() that follows is invoked on the same service instance. Hence we have a service that conforms to both types: it is long-running and serves interface invocations at the same time.

Simple? I thought so. I implemented a version of the example program and it worked flawlessy except that the service could not be stopped. It turns out that a dual service like that can be stopped with Context.stopService() call only if both lifecycles models allow it. This means that stopService() has no effect on a bound service!

Let's take an example. The client activity allows binding/unbinding, starting/stopping the service manually.



Now let's see what happens if the service is bound, unbound then started and stopped.

D/DUALSERVICECLIENT( 568): bindService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICECLIENT( 568): onServiceConnected
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): unbindService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@40044928
D/DUALSERVICE( 568): onDestroy
D/DUALSERVICECLIENT( 568): startService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICE( 568): onStart
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): stopService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@40048cd0
D/DUALSERVICE( 568): onDestroy

The service lifecycles work nicely: when the service is bound, its onCreate is called, then it is destroyed (onDestroy()) when it is unbound. The same goes for startService-stopService except that onStart() is called after onCreate(), as it should happen.

Now let's see what happens if the sequence is bind-start-stop-unbind!

D/DUALSERVICECLIENT( 568): bindService()
D/ActivityThread( 568): Creating service aexp.dualservice.DualService
D/DUALSERVICE( 568): onCreate
D/DUALSERVICECLIENT( 568): onServiceConnected
D/DUALSERVICECLIENT( 568): startService()
D/DUALSERVICE( 568): onStart
D/DUALSERVICECLIENT( 568): stopService()
I/ActivityManager( 465): Stopping service: {aexp.dualservice/aexp.dualservice.DualService}
D/DUALSERVICECLIENT( 568): unbindService()
D/ActivityThread( 568): Stopping service aexp.dualservice.DualService@4004a430
D/DUALSERVICE( 568): onDestroy


ActivityManager reports twice that the service was stopped. Of these two, only the second one is real (leads to the invocation of onDestroy()). As the service abstraction has no onStop() method (that would be counterpart of onStart()), the service is not notified that it was stopped. You can examine that with getting the counter: it continues counting after the service was stopped.

This is annoying enough but a method like stopCounting() on the AIDL interface would solve the problem. My opinion, however, is that the lack of service stop notification will cause problems with less tricky long-running services too. For example a voice recorder service will not be able to correctly free the resources (e.g. close the output file correctly) if it is just stopped by one of its clients or if the ActivityManager shuts it down due to low resource condition. I don't know how hard it would be ti implement stop notification callback (finalizers are the toughest kind of callbacks) but this enhancement was hacked into every application model that missed it originally (like Java applications, for example).

Tuesday, January 29, 2008

Hacking into Android's Bluetooth support

I have been mentioning that wicked Bluetooth service for two posts but eventually something else was more interesting. Well, finding something more interesting was easy because it turned out pretty soon that the Bluetooth support in m3-rc37a version of the emulator is half-ready at best. I cannot be sure but it seems that the hcid daemon on which Android's Bluez port is based is completely missing or at least is not launched properly along with other system services.

Is there a point then to deal with Bluetooth given the current state of the emulator? IMHO, yes, because when the shiny new version of the emulator comes out, applications will be ready to run. At the present state of affairs this approach means that we have to live with a bit (lot!) of hacking.

For the moment, I was interested in the discovery process. Discovery in the present implementation does not work but that did not prevent me simulating it. By taking apart org.bluez.IBluetoothService and android.server.BluetoothService classes with the jad tool, it quickly became evident that the discovery process is based on 4 intent messages.
  • org.bluez.intent.action.DISCOVERY_STARTED - delivered when the discovery process starts.
  • org.bluez.intent.action.REMOTE_DEVICE_FOUND - delivered when one device is discovered. It has 3 parameters in the Intent's Bundle: "address" (String) is the Bluetooth address of the device, "class" (Integer) is the class of the Bluetooth device and "rssi" (Short) is the Received Signal Strength Indicator.
  • org.bluez.intent.action.REMOTE_NAME_UPDATED - delivered when the user-friendly name of one device is discovered. This message follows an earlier REMOTE_DEVICE_FOUND message and carries the user-friendly name ("name" (String)) along with the Bluetooth address ("address" (String)).
  • org.bluez.intent.action.DISCOVERY_COMPLETED - delivered when the discovery process finishes.
You can download the example program from here.

The discovery simulator itself is an interesting piece of software because of some common Android programming tricks it employs. The discovery simulator is located in the BtDetectSimulatorService class that the main activity launches with a startService call (it is therefore the other type of service, not the one seen at the service invocation entry). Normally, the discovery would just be started with a method call on org.bluez.IBluetoothService but the implementation is not that ready. The service abuses the half-implemented Bluetooth classes (like the android.server.BluetoothService class, don't ever do what I did here!), starts the Bluetooth service and sends out intent messages. These intent messages are captured by an intent receiver (BtIntentReceiver) and are used to update the UI of the main activity.

Two small details are interesting here. The intent receiver and the main activity call each other like ordinary Java objects. The code exploits the fact that the main activity is a singleton and just connects the activity and the intent receiver instances by Java references. This is possible because these two entities are part of the same package, hence they run in the same VM instance. Here is the proof (619 is the PID of the VM process):

D/BTINTENTRECEIVER( 619): org.bluez.intent.action.DISCOVERY_STARTED
D/BTDETECT( 619): Action = org.bluez.intent.action.DISCOVERY_STARTED


The other interesting bit is the way BtDetectSimulatorService delivers the events. There is a delay among the intent broadcasts (Bluetooth discovery may be a lengthy process). This delay could not be coded with the good old Thread.sleep call in onStart method because onStart would have run too long and the Android application manager would kill the process. In addition, no broadcasts would be delivered until onStart ends which means that all the broadcasts would be delivered at once. Instead, observe how an android.os.Handler instance is used to schedule Runnable tasks after a certain delay.

And at last, the fun part. The image below shows, how Android's own Phone application crashes terribly when our application stops the Bluetooth service (it was us who started it, so we stop it nicely). That characterizes pretty much the readiness of the Bluetooth support in the current emulator.

Saturday, January 26, 2008

About binders

I wanted to go back to my Bluetooth exercise once again but then I stumbled into this beautiful sample program from John Guilfoyle. I decided immediately that before playing again with that wicked Bluetooth service, I have to test-drive again the service framework and the low-level mechanisms behind it.

My Android experiences with services were so far restricted to repeating the sample program in the documentation. This time I wanted to try something more ambitious: connecting two applications through services. In order to do that, it is worth getting behind the facade of the Android service framework. This thread contains a lot of information that I try to recap here.

The workhorse of the Android IPC is the "IBinder class". As its name implies, android.os.IBinder is really an interface, implemented by the real worker objects, namely android.os.BinderNative and android.os.BinderProxy. The binder provides a simple functionality of synchronous method invocation. Calling the transact() method on the binder is able to transmit a function code, a java.os.Parcel instance and some flags. The result is returned in another Parcel instance. Look for this method signature:

public final boolean transact(int code, Parcel data, Parcel reply, int flags ) throws DeadObjectException;

The client calls the transact method and the other side of the binder, in the other side of the process separation will receive a callback.

protected boolean onTransact(int code, Parcel data, Parcel reply, int flags);
The IPC has a thread pool for each process and the onTransact method is invoked in the context of a thread in the IPC thread pool. Parcel is the abstraction of the data the IPC framework is able to serialize and deserialize, you can look at the documentation of the android.os.Parcel object to see, what data types are supported. The most interesting data type is the IBinder itself (readStrongBinder, writeStrongBinder) so it is possible to pass binder endpoints to another processes. The interface mechanism that the aidl tool provides is really simple after that. Aidl takes the interface description and turns it into Java code that serializes and deserializes invocation parameters and return values to and from Parcels. Take for example our trusty IAdderService which has a method with this signature:

int add( in int i1, in int i2 );

Aidl has to generate a code that serializes two integers into the request Parcel and deserializes the return integer from the response Parcel in transact() because this is the client side of the stub. In onTransact method, as it is the receiver side of the call, it has to deserialize the input parameters, invoke the service-side implementation of the method then serialize the result. And it does exactly that (excerpts from the IAdderService.java generated by the aidl tool):

public int add(int i1, int i2) throws android.os.DeadObjectException { android.os.Parcel _data = android.os.Parcel.obtain();
android.os.Parcel _reply = android.os.Parcel.obtain();

int _result;

try {

_data.writeInt(i1);

_data.writeInt(i2);

mRemote.transact(Stub.TRANSACTION_add, _data, _reply, 0);

_result = _reply.readInt();

}

...

public boolean onTransact(int code, android.os.Parcel data, android.os.Parcel reply, int flags) {
try {
switch (code)

{

case TRANSACTION_add:

{
int _arg0;
_arg0 = data.readInt();

int _arg1;

_arg1 = data.readInt();

int _result = this.add(_arg0, _arg1);

reply.writeInt(_result);
return true;
}

}

The service framework does not provide any means to find out, what is the serialization and deserialization stub on the client and the service side which I consider a drawback. This is a reason why it is impossible to discover the service interfaces (one can retrieve the client-side binder but not the interface stub, therefore it is impossible to know, how to talk to the service without knowing the interface stub from some other, independent source). The framework simply assumes that the client- and the server-side stubs match, if not, then it is the programmer's fault.

I was now ready to realize my dream to connect two applications by means of service invocations. The best way is, of course, Intent-based invocation but I wanted to work that around just for the sake of hacking. As Dianne Hackbod pointed out, however, Android application lifecycle makes the direct connection impossible. Depending on resource situation, Android apps just come and go and therefore the binders connecting them may stop functioning without any warning. One has to go through services because service dependencies are respected by the application manager. So I created the following simple application (actually, two applications and one service).

You can download the example program from here.

The task is to receive a callback from Application 2 (called SClientApp) by Application 1 (called HostappClient). The connection between the two applications goes through a service (HostappServiceImpl, co-located with HostappClient). Both applications bind this service in a usual way. In addition to that, HostappClient provides the HostappServiceImpl with a callback binder. Why binder, why not interface? It could be interface and it would be more elegant to do so. The Windows version of the aidl tool in the current version (m3-rc37a) has an annoying bug that makes the import statement almost unusable so there is no way to import our own interface classes. This is not a problem, however, if we know the relationship of binders and interfaces. We just pass the binder reference and apply the interface stubs onto it when the binder arrives. The call flow is the following:
  • HostappClient binds HostappServiceImpl. When the service binding callback arrives, we pass the binder of HostAppClient's ISetTextService to HostappServiceImpl.
  • Now HostappClient launches SClientApp as subActivity. The first thing SClientApp does is to bind HostAppServiceImpl. When the service is bound succesfully, SClientApp passes the hugely valuable text result to HostAppServiceImpl which in turn calls HostappClient's ISetTextService whose binder we have received previously.
  • One interesting bit remained: the use of android.os.Handler in HostappClient. When HostappClient receives the service callback, it cannot directly manipulate the UI, only the UI thread is allowed to do that. The service callback runs in an IPC thread, not the UI thread, therefore it uses a Handler instance to pass a Runnable to the UI thread which is then executed there (the Handler executes the Runnable in the context of the thread it was created in and our Handler instance was created in the context of the UI thread).

Huh, pretty complicated instead of returning that hugely important string in the intent invocation bundle? And that is not the end, because our example program is not perfect. It is not prepared for low-memory situations when each of our three players may be destroyed and resurrected by the application manager. The worst-case scenario is this:
  • HostappClient is launched and when it binds HostappServiceImpl, these two entities are in the memory.
  • When HostappClient makes an intent invocation to SClientApp, both HostappClient and HostappService may be destroyed by the application manager.
  • SClientApp now binds HostappServiceImpl and these two are in the memory now. The IBinder instance passed by HostappClient is now lost.
  • When SClientApp finishes, SClientApp and HostappServiceImpl may both be destroyed again. Our hugely valuable text data passed by SClientApp to HostappServiceImpl is lost then.
  • HostappClient is resurrected but it has nothing to receive from HostappServiceImpl.
A robust implementation of HostappServiceImpl would store setMyText() invocations in a persistent storage, e.g. SQLite database if the callback to HostappClient fails (DeadObjectException or appService==null in HostappServiceImpl.setMyText). When the binder is set by HostappClient's invocation to setCallback(), HostappServiceImpl would check its storage and would immediately send back setMyText() callbacks to HostappClient from the persistent storage. This exercise is left, however, to the interested reader. :-)