Advantages and limitations of PhoneGap for sensor processing

This is my Droidcon Tunis 2013 presentation about the effect of the web application model on battery consumption.

Test programs related to the slideset are available here. You have to be logged into Sfonge website to access those.

Data capture application for car speed, accelerometer, compass and gyro fusion experiments

In my previous post I wrote about my enthusiasm of integrating car sensors with the mobile device's own sensors. The first step was to implement a data capture application that saves sensor data into a file. This time I decided to save the gyro, accelerometer and compass sensor data from the phone and the speed as measured by the car's speedometer. The most interesting aspect here is how we obtain the speed data.

There are two levels of interfaces playing role here. OBD2 interface and the protocols accessible over it is the first level offered by the car's on-board electronics. This interface is not trivial to talk to but fortunately all the complexity of OBD2-related protocols is implemented by the super-popular ELM327 microcontroller from ELM Electronics (or its compatible copies). On one side the ELM327 chip talks to the car's ECU using the OBD2 interface. On the other side, it offers a modem-like serial interface. Popular connection option is USB but for our Android client, Bluetooth is a better option. ELM327-compatible adapters are available at a low cost, for example I bought the very basic version below.

ELM327 has a very simple, yet powerful interface. In the first phase the chip is set up by issuing a series of AT commands reminiscent of the Hayes modem commands. The best place for reference is the ELM327 specification. The most important command is the ATS0 command. This instructs the chip to auto-detect the protocol used on the OBD2 interface. Once this auto-detection is successful, the car sensors can be queried by sending the PID code of the sensor in question. For example for the car's speed one sends:

010D

The ELM327 chip then talks to the car's ECU, obtains the sensor data and returns a response like this:

410D0C

Where "4" is the ID for response, 10D is the PID of the sensor and 0C is the value. Some OBD2 PIDs have complex coding but fortunately for the speed sensor 0C simply means 12 km/h.

Click here to download the example program.

Before you start using this test application, make the Bluetooth pairing of your device with the sensor from the Android Bluetooth settings. Either you can use the real thing (ELM327 interface connected to a real car) or you can use Ryan C. Gordon's great OBDSim application that can run on a Linux box and can simulate a Bluetooth-connected ELM327. In theory, OBDSim can also run on Windows, I tested only on Linux. Setting it up is not trivial but it saves a great deal of time if you develop for the ELM327 interface.

Then you can start the test application. First connect to your ELM327 device from the menu ("Connect a device - secure"). If the connection works properly, you should see a series of AT commands and responses flashing on the screen then the speed appears (e.g. 0 km/h if you do it on a real car). This time you can start the sampling ("Sampling" check box) and the application will save all the sensor information into a file in the SDCARD in CSV format. The name of the file is gyrocarcapture_<date>.csv and its format is very simple. One warning here: sadly the timestamp format used for the speed sensor and the device's own sensor are different: the sensor timestamps are coming from the SensorEvent object and the time stamp for the speed data comes from System.currentTimeMillis(). As device sensor events are very frequent, I use the last device sensor timestamp for the speed sensor too when processing these files. The speed sensor sampling frequency is approximately 1 Hz.

The application can be extended easily to sample other car sensors too. However, in its present form it is good enough to detect if the driver slows down in front of a speed bump (accelerometer-car speed sensor fusion).

Or to figure whether the driver takes the bend too quickly (gyroscope-car speed sensor fusion).

Call for papers for Droidcon Tunis 2013

Last year I had a presentation at Droidcon Tunis. The conference was inspiring with a clever and enthusiastic audience and the presentation became a success with over 16000 views since the end of the conference. Not bad for a sensor processing topic with some scary math in it. Plus my name became a landmark in Tunis which was pretty much a new sensation for me.

Call for papers for Droidcon Tunis 2013 is now open but sadly, I can't submit anything. My presentation material is not ready because some topics I could speak about are not public, others are not mature enough. But you out there, you should consider submitting papers to this conference. Last year's conference was fun - also my first time in Africa. I am sure that this year will be just better.

Car sensors

My conclusion from the speed bump post was that the built-in accelerometer in the Android phone is probably not enough to detect the obstacle. Alexander Kosenkov commented that maybe the accelerometer vector can be processed in a more sophisticated way but my experience is that the car generates such an amount of vibrations that makes extracting relatively faint signals from the accelerometer rather hard if not impossible.

While I was contemplating on this issue, I got a mail from Edward Pultar, president of Valarm, a start-up based in Los Angeles. These guys created an Android application and associated web services that started as a theft-prevention tool and is being repositioned as a general platform for all sorts of sensor data acquisition and processing. Read their press kit about the exact features of their products.

The most fascinating idea in their product is the use of OBD2 interface to connect to the car's own sensors. OBD2 with a Bluetooth plug is widely used by car fanatics to obtain diagnostic data including speedometer, tachometer, engine temperature, on-board GPS (if the car is equipped with it) and many other sensors. Valarm's idea was to use this wealth of data for use cases other than car diagnostics, e.g. monitoring whether the car is driven recklessly. They do sensor fusion with the car sensors and with the phone's own sensors. For example cars don't normally have accelerometers that can detect excessive vehicle vibration caused by bumpy roads or faulty tires but a smartphone has. The phone is also equipped with a wealth of communication options, large memory and is an application platform. It is therefore an ideal device to aggregate data from different sensors (internal and external) and to send the data to web services.

This may sound like a futuristic research paper but to my utter surprise, even my own car which is well in its teenager years has the OBD2 interface. All I need is a commercially available Bluetooth plug (connection by cable is also possible but is less practical) and the car will talk to my Android phone. This makes Alexander's idea about incorporating the speed profile for the speed bump detection pretty easy to implement.

Detecting speed bumps with accelerometer

Somebody came in from the internet some months ago and asked whether speed bumps (also called sleeping policemen) can be detected with an Android phone. You know, speed bumps like this:

I was cautious because there's a lot of traps here. Obviously the detected signal will differ depending on the form factor of the speed bump, the type of the car and even the way the phone is fixed to the car (or the lack of fixing). But I was also curious and as car-related phone applications are very fashionable today, I took the challenge. Meanwhile the guy who brought up the problem disappeared so I feel free to share what I achieved during the last one and half months.

The solution that I produced is far from perfect. First and foremost, I did not make any effort to target other phones than my trusty Nexus S. Phones which have sampling frequency of around 50 Hz in SENSOR_DELAY_FASTEST mode will work fine, significantly different sampling frequencies are not supported. It is more important, however, that I was not able to create a foolproof algorithm for the problem.

Initially I thought that some sort of rotation could be measured when the front of the car is lifted by the speed bump but I was not able to detect this rotation from the sensor signal. The car generates all sorts of vibrations and it is not easy to extract this relatively small rotation from the noisy background. So I went back to the acceleration spike which is generated when the car hits the bump and I used just the acceleration's absolute value as described in this presentation. The trouble is that this spike is pretty reminiscent of a class of other spikes like knocks, etc. For example here is a beautiful double street bump somewhere in Budapest.

One would think that signals like these could be easily detected. Now comes, however, a bumpy road with potholes, also typical in Budapest.

There are no street bumps in the second signal but the potholes generate "knocks" that are hard to distinguish from the "knocks" caused by street bumps.

There is no perfect solution to this problem, at least I could not find any. Eventually I obtained the best result by calculating the cross-correlation between the acceleration signal and a pre-recorded speed bump signal. I re-sampled this representative signal for +/- 10% of its initial frequency and calculated the cross-correlation with all the three variants (initial, +10% frequency, -10% frequency). Then I used a sliding window to calculate the power of the cross-correlation signal. This is the pre-recorded sample I used for cross-correlation:

Cross-correlating then calculating the power in a sliding window yields this signal for the double speed bump (different colors show the power of the 3 variants of the re-sampled signal):

And this is the result of the same calculation for the bumpy road:

Using a limit of about 150, the two signals can be separated and no fake bumps are detected even on the bumpy road.

Click here to download the example program.

There are a lot of problems with this approach. First, speed bumps come in all shapes and sizes and this method can detect speed bumps that are similar to the one used to generate the reference signal. Then there is the problem of the car itself. I did not have the chance to test with a lot of different cars but I bet the different suspension systems will yield drastically different signals in different car types. Anyway, I thought somebody may find this post and this post can start some thinking.

Universal orientation, gravity and motion acceleration sensor

We have seen a number of fusion possibilities for 2 sensors.

• We have seen accelerometer supported by gyroscope so that gravity and motion acceleration may be reliably separated. See this Droidcon Tunis presentation for more details.
• We have seen accelerometer supported by the compass to achieve the same goal. There were some limitations that made this solution less flexible than the one based on accelerometer-gyroscope fusion.
• We have seen the compass supported by the accelerometer to compensate for the tilt of the device.

Time has come to put all of these together and to take a step toward the Holy Grail of sensor processing. We want a sensor that measures gravity, motion acceleration and orientation reliably, no matter what the circumstances are. If we achieve this, then we have a good basis for the much researched in-door navigation problem because the linear acceleration may be used for step counting while the orientation can be used to calculate the vector of movement. Errors will accumulate but if we have occasional reference points like WiFi beacons, reasonably good results can be achieved.


So what prevents us realizing this vision? We already obtained a pretty reliable motion acceleration estimation from the accelerometer-gyroscope fusion. The orientation is the thing we don't have. Using the compass-accelerometer fusion we can eliminate the tilt problem. If we throw in the gravity calculation from the accelerometer-gyroscope fusion and use the gravity estimation instead of the raw accelerometer data for the compass tilt compensation then we have also compensated the motion acceleration. The only thing we don't have is the effect of external magnetic fields which may fool the compass and can be pretty nasty in-door in the presence of household appliances, large transformators or other electric machinery. That we will compensate with a compass-gyroscope fusion.


The idea of the compensation of the external magnetic field using the gyroscope is very similar to the compensation of the motion acceleration. If we assume that no external magnetic field is present (beside the Earth's magnetic field), we just use the compass data (and then post-compensate against tilt by the gravity vector which needs accelerometer-gyroscope fusion). If we suspect that there may be external magnetic field present, we use the gyroscope rotation data to rotate a previous, trusted compass vector (that is supposed to measure only the Earth's magnetic field) and use that as compass estimation. The difference between this compass estimation and the measured compass value is the estimation for the external magnetic field.


The trick is the detection whether an external magnetic field may be present. In this prototype the same trick was used as with the accelerometer: if the length of the compass vector is too long or too short compared to the reference measurement, then we have an external magnetic field to compensate.

Here is the summary of the sensor fusions used in this prototype:

• Accelerometer-gyroscope fusion provides gravity and motion acceleration.
• Compass-gravity fusion provides compass tilt compensation. The gravity is already a derived measurement coming from the accelerometer-gyroscope fusion.
• Compass-gyroscope fusion provides orientation and external magnetic field.

Click here to download the example program.

The example program starts with a complex stationary and dynamic calibration process. When stationary, it measures the reference gravity and magnetic field values. In this stage it is very important that the device is not subject to motion acceleration or external magnetic field (beside the Earth's magnetic field). In the dynamic calibration phase we calculate the bias of the compass. We do this by instructing the user to rotate the device and measuring compass vectors in each special position. These are supposed to be points on a sphere. Once we collected the measurement points, we calculate the center of the sphere by solving a set of linear equations. The Jama package was incorporated into the example program for matrix manipulations.

Once the calibration is finished, the algorithm outlined above is executed. The measurement data is visualized in a 2D coordinate system - the z coordinate is not drawn. This is just my laziness, I did not take the effort to implement a 3D visualization engine. It is important to note, however, that the motion acceleration (yellow) and the external magnetic field (cyan) vectors are really 3D, they are just not drawn as such. The white vector is the final orientation compensated against tilt, motion acceleration and external magnetic field.

The prototype does have limitations. While it survives pretty nicely short-term disturbances from external magnetic fields (try with a small magnet), in longer term (e.g. after 5-10 minutes) it gets properly lost. For example when riding the Budapest underground there are strong and varying magnetic fields generated by the train's traction motors during the entire journey. If the compensation algorithm picks up a wrong reference vector, it may get lost for the rest of the journey until it is able to pick up the Earth's undisturbed magnetic field.

Better motion control using accelerometer/gyroscope sensor fusion

This is my Droidcon Tunis 2012 presentation. Thanks for the great audience in Tunis!

Better motion control using accelerometer/gyroscope sensor fusion

View more presentations from Gabor Paller.