Friday, October 25, 2013

Thursday, July 4, 2013

Converting classes with the Parcelable protocol

Many good tutorials can be found about the Parcelable protocol in Android. If the Android framework has to serialize a set of values (either for AIDL-based IPC or for Intent-based message passing), it can serialize basic types or objects that implement the Parcelable protocol. In short, Parcelable objects are able to serialize their states into a Parcel object and they can retrieve their state from the serialized form.

But what happens if the two classes (the one which is serializing and the one which is retrieving its state from serialized form) are not the same? If you remember my post about classloading in Android, there is no way loading a class from another APK. If you pass a serialized object to another application whose codebase comes from a different APK, the target application has to rely on its own version of the class with the same name. Android framework does not verify that the sender and receiving classes are the same, they just have to have the same name (including package name).

Click here to download the example application.

There are two apps in the package. Actually, it is one of my first example programs, here is the description from a 2007 post of this blog. There are two applications, intentsender and intentreceiver. Intentsender has an internal intentreceiver plus it can talk to the external intentreceiver app. Both applications have a version of class. In this example they are just slightly different - they log different messages.

What happens when you click on the "exp / int" (explicit Intent with internal intentreceiver) button. Both of these activities are residing in the intentsender APK therefore the class which will be used to serialize and deserialize the object are the same.

D/TwoNumbers1( 1090): serializing in TwoNumbers#1: i1: 2; i2: 3
D/TwoNumbers1( 1090): deserializing in TwoNumbers#1: i1: 2; i2: 3

But if you click on the "exp / ext" (explicit Intent with external intentreceiver) button, the message has to pass the APK boundary (it is processed in the intentreceiver app). Intentreceiver has nothing else but its own version of The log demonstrates it:

D/TwoNumbers1( 1207): serializing in TwoNumbers#1: i1: 4; i2: 5
D/TwoNumbers2( 1166): deserializing in TwoNumbers#2: i1: 4; i2: 5

Android framework did not care that the two TwoNumbers classes were not the same, it just called them blindly to serialize and deserialize. Actually, if you look into the Parcel implementation, you will see that the serialized version of the Parcelable only stores the fully qualified class name so it is not surprising that intentreceiver loaded whatever version of the class it could find under the given name. Fortunately, the two versions were compatible (they produced and consumed the same serialized stream) so there were no problems.

Why is this useful other than misleading yourself with classes that look the same but they aren't? If you want to expose just part of your serialized object to the receiver application, this trick comes handy. Otherwise it is best to avoid it.

Thursday, June 20, 2013

Tap detection supported by the gyroscope

Once upon a time I made a Londroid presentation that briefly mentioned the possibility of detecting taps on the phone's body with an accelerometer (it is in slide 31). You may rightly ask, why do you want to detect taps with the accelerometer when there is the touchscreen? First, the touchscreen may be switched off by the screensaver or the user may have deactivated it because he or she is in a call or put the phone into a position where the touchscreen may be touched unintentionally. Then the user may want to tap the back of the phone. Lastly, the user may be interested in the strength of the tap, not only its location or the fact that the tap happened.

Let's recall that graph in that old Londroid presentation.

The graph shows the absolute value of the acceleration vector while the phone's body was tapped twice. The base rate is the gravity acceleration (about 10 m/sec^2) and the acceleration caused by the taps was added (in the actual configuration, subtracted) from the gravity acceleration, resulting two characteristic spikes.

But where was the phone's body tapped? By looking solely on the accelerometer's data, I don't think there is a way to find it out. But fortunately we have other sensors and the gyroscope is particularly relevant. Here is the well-known image of the sensor axes in Android.

In case of the gyroscope, the sensor measures rotation around the axes. If you tap the device in the right lower corner, we can expect a short rotation around the x and y axes if the phone supported by a reasonably flexible base (e.g. a human hand and not a block of granite).

But can this be really detected? I made a series of measurements to prove it. Note here that I was just interested in the theoretical possibility and did not want to do a product-grade implementation. So I tested only on my Galaxy Nexus.

First here is the absolute acceleration value for 3 consecutive taps.

As expected, the time of the tap can be safely detected. These taps were all in the lower right corner so let's see what the gyroscope measured along the x and y axes.

X axis, just for two taps:

Y axis, just for two taps:

As you can see, the taps are detected by the gyroscope too as short rotations. In case of both axes, there was a spike of positive rotation velocity first.

Now let's see what happens, if we tap the lower left corner.

X axis, just for two taps:

Y axis, just for two taps:

As you see, the X graph has the same character but the form of the Y graph has changed. Instead of a marked positive spike, there is a small negative spike, followed by a positive spike.

Let's see if the experience is repeated in the upper left corner.

X axis, just one tap:

Y axis, just one tap:

Now the character of the spikes on both axes changed. A small positive spike is followed by a large negative spike.

This is not a kind of movement the gyroscope was designed for so it is premature to state that this waveform pattern will be exactly the same for all the Android devices sporting gyroscope out there. But I found these measurements convincing enough to support the statement that it is possible to implement rudimentary tap location using only the accelerometer and the gyroscope in the device.

Sunday, April 28, 2013

Advantages and limitations of PhoneGap for sensor processing

This is my Droidcon Tunis 2013 presentation about the effect of the web application model on battery consumption.

Test programs related to the slideset are available here. You have to be logged into Sfonge website to access those.

Sunday, March 10, 2013

Data capture application for car speed, accelerometer, compass and gyro fusion experiments

In my previous post I wrote about my enthusiasm of integrating car sensors with the mobile device's own sensors. The first step was to implement a data capture application that saves sensor data into a file. This time I decided to save the gyro, accelerometer and compass sensor data from the phone and the speed as measured by the car's speedometer. The most interesting aspect here is how we obtain the speed data.

There are two levels of interfaces playing role here. OBD2 interface and the protocols accessible over it is the first level offered by the car's on-board electronics. This interface is not trivial to talk to but fortunately all the complexity of OBD2-related protocols is implemented by the super-popular ELM327 microcontroller from ELM Electronics (or its compatible copies). On one side the ELM327 chip talks to the car's ECU using the OBD2 interface. On the other side, it offers a modem-like serial interface. Popular connection option is USB but for our Android client, Bluetooth is a better option. ELM327-compatible adapters are available at a low cost, for example I bought the very basic version below.

ELM327 has a very simple, yet powerful interface. In the first phase the chip is set up by issuing a series of AT commands reminiscent of the Hayes modem commands. The best place for reference is the ELM327 specification. The most important command is the ATS0 command. This instructs the chip to auto-detect the protocol used on the OBD2 interface. Once this auto-detection is successful, the car sensors can be queried by sending the PID code of the sensor in question. For example for the car's speed one sends:


The ELM327 chip then talks to the car's ECU, obtains the sensor data and returns a response like this:


Where "4" is the ID for response, 10D is the PID of the sensor and 0C is the value. Some OBD2 PIDs have complex coding but fortunately for the speed sensor 0C simply means 12 km/h.

Click here to download the example program.

Before you start using this test application, make the Bluetooth pairing of your device with the sensor from the Android Bluetooth settings. Either you can use the real thing (ELM327 interface connected to a real car) or you can use Ryan C. Gordon's great OBDSim application that can run on a Linux box and can simulate a Bluetooth-connected ELM327. In theory, OBDSim can also run on Windows, I tested only on Linux. Setting it up is not trivial but it saves a great deal of time if you develop for the ELM327 interface.

Then you can start the test application. First connect to your ELM327 device from the menu ("Connect a device - secure"). If the connection works properly, you should see a series of AT commands and responses flashing on the screen then the speed appears (e.g. 0 km/h if you do it on a real car). This time you can start the sampling ("Sampling" check box) and the application will save all the sensor information into a file in the SDCARD in CSV format. The name of the file is gyrocarcapture_<date>.csv and its format is very simple. One warning here: sadly the timestamp format used for the speed sensor and the device's own sensor are different: the sensor timestamps are coming from the SensorEvent object and the time stamp for the speed data comes from System.currentTimeMillis(). As device sensor events are very frequent, I use the last device sensor timestamp for the speed sensor too when processing these files. The speed sensor sampling frequency is approximately 1 Hz.

The application can be extended easily to sample other car sensors too. However, in its present form it is good enough to detect if the driver slows down in front of a speed bump (accelerometer-car speed sensor fusion).

Or to figure whether the driver takes the bend too quickly (gyroscope-car speed sensor fusion).

Friday, March 1, 2013

Call for papers for Droidcon Tunis 2013

Last year I had a presentation at Droidcon Tunis. The conference was inspiring with a clever and enthusiastic audience and the presentation became a success with over 16000 views since the end of the conference. Not bad for a sensor processing topic with some scary math in it. Plus my name became a landmark in Tunis which was pretty much a new sensation for me.

Call for papers for Droidcon Tunis 2013 is now open but sadly, I can't submit anything. My presentation material is not ready because some topics I could speak about are not public, others are not mature enough. But you out there, you should consider submitting papers to this conference. Last year's conference was fun - also my first time in Africa. I am sure that this year will be just better.

Friday, February 22, 2013

Car sensors

My conclusion from the speed bump post was that the built-in accelerometer in the Android phone is probably not enough to detect the obstacle. Alexander Kosenkov commented that maybe the accelerometer vector can be processed in a more sophisticated way but my experience is that the car generates such an amount of vibrations that makes extracting relatively faint signals from the accelerometer rather hard if not impossible.

While I was contemplating on this issue, I got a mail from Edward Pultar, president of Valarm, a start-up based in Los Angeles. These guys created an Android application and associated web services that started as a theft-prevention tool and is being repositioned as a general platform for all sorts of sensor data acquisition and processing. Read their press kit about the exact features of their products.

The most fascinating idea in their product is the use of OBD2 interface to connect to the car's own sensors. OBD2 with a Bluetooth plug is widely used by car fanatics to obtain diagnostic data including speedometer, tachometer, engine temperature, on-board GPS (if the car is equipped with it) and many other sensors. Valarm's idea was to use this wealth of data for use cases other than car diagnostics, e.g. monitoring whether the car is driven recklessly. They do sensor fusion with the car sensors and with the phone's own sensors. For example cars don't normally have accelerometers that can detect excessive vehicle vibration caused by bumpy roads or faulty tires but a smartphone has. The phone is also equipped with a wealth of communication options, large memory and is an application platform. It is therefore an ideal device to aggregate data from different sensors (internal and external) and to send the data to web services.

This may sound like a futuristic research paper but to my utter surprise, even my own car which is well in its teenager years has the OBD2 interface. All I need is a commercially available Bluetooth plug (connection by cable is also possible but is less practical) and the car will talk to my Android phone. This makes Alexander's idea about incorporating the speed profile for the speed bump detection pretty easy to implement.

Tuesday, February 19, 2013

Detecting speed bumps with accelerometer

Somebody came in from the internet some months ago and asked whether speed bumps (also called sleeping policemen) can be detected with an Android phone. You know, speed bumps like this:

I was cautious because there's a lot of traps here. Obviously the detected signal will differ depending on the form factor of the speed bump, the type of the car and even the way the phone is fixed to the car (or the lack of fixing). But I was also curious and as car-related phone applications are very fashionable today, I took the challenge. Meanwhile the guy who brought up the problem disappeared so I feel free to share what I achieved during the last one and half months.

The solution that I produced is far from perfect. First and foremost, I did not make any effort to target other phones than my trusty Nexus S. Phones which have sampling frequency of around 50 Hz in SENSOR_DELAY_FASTEST mode will work fine, significantly different sampling frequencies are not supported. It is more important, however, that I was not able to create a foolproof algorithm for the problem.

Initially I thought that some sort of rotation could be measured when the front of the car is lifted by the speed bump but I was not able to detect this rotation from the sensor signal. The car generates all sorts of vibrations and it is not easy to extract this relatively small rotation from the noisy background. So I went back to the acceleration spike which is generated when the car hits the bump and I used just the acceleration's absolute value as described in this presentation. The trouble is that this spike is pretty reminiscent of a class of other spikes like knocks, etc. For example here is a beautiful double street bump somewhere in Budapest.

One would think that signals like these could be easily detected. Now comes, however, a bumpy road with potholes, also typical in Budapest.

There are no street bumps in the second signal but the potholes generate "knocks" that are hard to distinguish from the "knocks" caused by street bumps.

There is no perfect solution to this problem, at least I could not find any. Eventually I obtained the best result by calculating the cross-correlation between the acceleration signal and a pre-recorded speed bump signal. I re-sampled this representative signal for +/- 10% of its initial frequency and calculated the cross-correlation with all the three variants (initial, +10% frequency, -10% frequency). Then I used a sliding window to calculate the power of the cross-correlation signal. This is the pre-recorded sample I used for cross-correlation:

Cross-correlating then calculating the power in a sliding window yields this signal for the double speed bump (different colors show the power of the 3 variants of the re-sampled signal):

And this is the result of the same calculation for the bumpy road:

Using a limit of about 150, the two signals can be separated and no fake bumps are detected even on the bumpy road.

Click here to download the example program.

There are a lot of problems with this approach. First, speed bumps come in all shapes and sizes and this method can detect speed bumps that are similar to the one used to generate the reference signal. Then there is the problem of the car itself. I did not have the chance to test with a lot of different cars but I bet the different suspension systems will yield drastically different signals in different car types. Anyway, I thought somebody may find this post and this post can start some thinking.