Infrared imaging with Android devices

One of the most evident sensors of Android devices is the camera. An ordinary smartphone's camera is able to capture a lot of interesting information but has its limitations too. Most evidently, its viewing angle depends on the position of the device (so it is not fixed and hard to measure) and its bandwidth is (mostly) restricted to the visible light. It is therefore an exciting idea to connect special cameras to Android devices.




In this post, I will present an integration of FLIR Lepton Long-wavelength Infrared Camera to an Android application over Bluetooth Low Energy connection. Long-wavelength IR (LWIR) cameras are not new. Previously, however, they were priced in the thousands of dollars range (if not higher). Lepton is still pricey (currently about 300 USD) but its price is low enough so that mere mortals can play with it. FLIR sells a smartphone integration product (called FLIR One) but it is currently only available for iPhone and locks the camera to one device. Our prototype allows any device with BLE connection to access this very special camera.

The prototype system presented here needs a relatively long list of external hardware components and it is also not trivial to prepare these components. This list is the following:

• An Android phone with Bluetooth 4.0 capability. I used Nexus 5 for these experiments.
• An FLIR Lepton module. My recommendation is the FLIR Dev Kit from Sparkfun that has the camera module mounted on a breakout panel that is much easier to handle than the original FLIR socket.
• A BeagleBone Black card with an SD Card >4GB.
• A BLED112 BLE dongle from Silicon Labs (formerly Bluegiga).

The software for the prototype can be downloaded in two packages.

• This package contains the stuff for the embedded computer.
• This package is the Android application that connects to it.
  

Once you got all these, prepare the ingredients.

1. Hook up the FLIR camera with the BeagleBone Black

Fortunately the BBB's SPI interface is completely compatible with the Lepton's so the "hardware" just needs a couple of wires. Do the following connections (P9 refers to the BBB's P9 extension port).

FLIR	BBB
CS	P9/28 (SPI1_CS0)
MOSI	P9/30 (SPI1_D1)
MISO	P9/29 (SPI1_D0)
CLK	P9/31 (SPI1_SCLK)
GND	P9/1 (GND)
VIN	P9/4 (DC, 3.3V)

2. Prepare the BBB environment

I use Snappy Ubuntu. Grab the SD card and download the image as documented here. Before flashing the SD card, we have to update the device tree in the image so that the SPI port is correctly enabled. Unpack bt_ircamera.zip that you have just downloaded and go to the dt subdirectory. There you find a device tree file that I used for this project. Beside the SPI1 port, it also enables some serial ports. These are not necessary for this project but may come handy.

Compile the device tree:

dtc -O dtb -o am335x-boneblack.dtb am335x-boneblack.dts

The output is the binary device tree (am335x-boneblack.dtb) that needs to be put into the kernel image file. Let's suppose that the downloaded image file is ubuntu-15.04-snappy-armhf-bbb.img and you have an empty directory at /mnt/img. Then do the following:

fdisk -l ubuntu-15.04-snappy-armhf-bbb.img

Look for the first partition and note the partition image name and the offset:

ubuntu-15.04-snappy-armhf-bbb.img1   *        8192      139263       65536    c  W95 FAT32 (LBA)
...

Note that the actual partition image name may differ depending on the Snappy image you downloaded. Calculate the offset as 8192*512=4194304
Now mount the partition:

mount -o loop,offset=4194304 ubuntu-15.04-snappy-armhf-bbb.img /mnt/img

Then copy the dtb into the image, unmount and write the image to SD card (on my computer the SD card interface is /dev/sdc, check before you issue the dd command!):

cp am335x-boneblack.dtb /mnt/img/a/dtbs
umount /mnt/img
dd if=ubuntu-15.04-snappy-armhf-bbb.img of=/dev/sdc bs=32M

Now you have an SD card that you can insert into the BBB and boot from it. Once you reached the Ubuntu prompt and logged in (ubuntu/ubuntu), there's one thing more: the Snappy prototype application depends on the libpng package which is not part of the default Snappy image. But before you do it, check whether the SPI device was enabled correctly:
root@localhost:~# ls /dev/spidev1.0                                            
/dev/spidev1.0

Now about the png library. Download the armhf image from this location:

wget http://ports.ubuntu.com/pool/main/libp/libpng/libpng12-0_1.2.50-1ubuntu2_armhf.deb


Copy it to the BBB (update your card's IP address according to your network policies):
scp libpng12-0_1.2.50-1ubuntu2_armhf.deb ubuntu@192.168.1.115:~

Then go to the BBB console and install the deb package:
sudo mount -o remount,rw /
sudo dpkg -i libpng12-0_1.2.50-1ubuntu2_armhf.deb
sudo mount -o remount,ro /

3. Prepare the BLE dongle

The BLED112 stores the GATT tree in its firmware, hence in order to provide the GATT services that connect the BBB with the Android device, a new firmware needs to be generated and installed in the dongle. The config files are located in the config subdirectory in the bt_ircamera.zip archive. Follow the steps in this post to generate and install the new firmware. Once you are done, you can simply plug the dongle into the USB port of the BBB.

4. Install the prototype applications

The prototype system has two parts. The application running on the BBB acts as BLE server, fetches images from the FLIR camera and transmits them over BLE. The Android application acts as BLE client, fetches images from the BLE server and displays them. The BBB part is located in bt_ircamera.zip and the Android part is in IRCamera.zip. The latter is just the source part of the Android Studio project tree - I omitted all the garbage that Android Studio generates into the project folders. For the BBB installation, follow the instructions in this blog post. Launch the BBB application like this as root:
/apps/ircamera.sideload/1.0.0/bin/ircamera /dev/ttyACM0
and you are ready to go. On the Android side, select the BLE node with the name "test", connect, click the "Take picture" button, wait for the image to download and there you are. Note that the images are saved on the SD card, which means that they also appear in the stock "Photos" application.

Now at last we can get to the technical issues with this prototype. One interesting aspect is that there is no standard BLE service that provides the functionalities - image capture triggering, image fetching - our system needs. That's not a problem, we defined our own BLE service. It is easiest to follow this service in irc_gattBLED112.xml (bt_ircamera.zip, config subdirectory).

The service has a custom UUID, generated randomly:

<service uuid="274b15a3-b9cd-4e5e-94c4-1248b42b82f8" advertise="true">

Also, its 3 GATT characteristics are in the non-standard UUID domain:

<characteristic uuid="00000000-b9cd-4e5e-94c4-1248b42b82f8" id="irc_len">
...
<characteristic uuid="00000001-b9cd-4e5e-94c4-1248b42b82f8" id="irc_offs">
...
<characteristic uuid="00000002-b9cd-4e5e-94c4-1248b42b82f8" id="irc_pic">

The interaction goes like the following. The BLE client connects and reads the irc_len characteristic. This characteristic is tagged as "user" on the BLE server side meaning that the BLE application must generate the value on the fly, when the attribute is read. In our case, reading this attribute fetches an image from the FLIR camera, converts it into PNG format and stores it in the apps' data folder, returning only the PNG file size. The Android application now can fetch the image piece by piece. First the Android application writes the irc_offs characteristic to inform the BLE server, what is the starting location of the fragment it wants to fetch. Then it reads the irc_pic characteristic which returns a maximum of 20 bytes of image data. This makes the image download very slow (takes about 10-20 second to download a general 5-6 Kbyte image to the Android application) but the restriction comes from a BLE protocol layer. Maybe the old RFCOMM from Bluetooth Classic would have been actually a better option for this application.

Update: I updated the client/server application to make the download faster (it is still quite slow). In order to speed up, I removed the explicit setting of the file offset (so the irc_offs characteristic is not used anymore). This made the download faster but there's still room for improvement.

Other than the issue with fragment size, both applications are pretty straighforward. Maybe the colors of the IR image are worth discussing a bit. The FLIR camera returns a matrix of 80x60 pixels, each pixel has a depth of 12 bit. Grayscale presentation is the most evident option but most displays have only 256 gray colors. In order to make the IR shades more visible, I used fake coloring. The algorithm is very simple: after the image is fetched, the maximum and the minimum IR intensity is calculated and the range between the two are mapped into a rainbow gradient of 400 colors.

Hacking into Android's Bluetooth support

I have been mentioning that wicked Bluetooth service for two posts but eventually something else was more interesting. Well, finding something more interesting was easy because it turned out pretty soon that the Bluetooth support in m3-rc37a version of the emulator is half-ready at best. I cannot be sure but it seems that the hcid daemon on which Android's Bluez port is based is completely missing or at least is not launched properly along with other system services.

Is there a point then to deal with Bluetooth given the current state of the emulator? IMHO, yes, because when the shiny new version of the emulator comes out, applications will be ready to run. At the present state of affairs this approach means that we have to live with a bit (lot!) of hacking.

For the moment, I was interested in the discovery process. Discovery in the present implementation does not work but that did not prevent me simulating it. By taking apart org.bluez.IBluetoothService and android.server.BluetoothService classes with the jad tool, it quickly became evident that the discovery process is based on 4 intent messages.

• org.bluez.intent.action.DISCOVERY_STARTED - delivered when the discovery process starts.
• org.bluez.intent.action.REMOTE_DEVICE_FOUND - delivered when one device is discovered. It has 3 parameters in the Intent's Bundle: "address" (String) is the Bluetooth address of the device, "class" (Integer) is the class of the Bluetooth device and "rssi" (Short) is the Received Signal Strength Indicator.
• org.bluez.intent.action.REMOTE_NAME_UPDATED - delivered when the user-friendly name of one device is discovered. This message follows an earlier REMOTE_DEVICE_FOUND message and carries the user-friendly name ("name" (String)) along with the Bluetooth address ("address" (String)).
• org.bluez.intent.action.DISCOVERY_COMPLETED - delivered when the discovery process finishes.

You can download the example program from here.

The discovery simulator itself is an interesting piece of software because of some common Android programming tricks it employs. The discovery simulator is located in the BtDetectSimulatorService class that the main activity launches with a startService call (it is therefore the other type of service, not the one seen at the service invocation entry). Normally, the discovery would just be started with a method call on org.bluez.IBluetoothService but the implementation is not that ready. The service abuses the half-implemented Bluetooth classes (like the android.server.BluetoothService class, don't ever do what I did here!), starts the Bluetooth service and sends out intent messages. These intent messages are captured by an intent receiver (BtIntentReceiver) and are used to update the UI of the main activity.

Two small details are interesting here. The intent receiver and the main activity call each other like ordinary Java objects. The code exploits the fact that the main activity is a singleton and just connects the activity and the intent receiver instances by Java references. This is possible because these two entities are part of the same package, hence they run in the same VM instance. Here is the proof (619 is the PID of the VM process):

D/BTINTENTRECEIVER( 619): org.bluez.intent.action.DISCOVERY_STARTED
D/BTDETECT( 619): Action = org.bluez.intent.action.DISCOVERY_STARTED

The other interesting bit is the way BtDetectSimulatorService delivers the events. There is a delay among the intent broadcasts (Bluetooth discovery may be a lengthy process). This delay could not be coded with the good old Thread.sleep call in onStart method because onStart would have run too long and the Android application manager would kill the process. In addition, no broadcasts would be delivered until onStart ends which means that all the broadcasts would be delivered at once. Instead, observe how an android.os.Handler instance is used to schedule Runnable tasks after a certain delay.

And at last, the fun part. The image below shows, how Android's own Phone application crashes terribly when our application stops the Bluetooth service (it was us who started it, so we stop it nicely). That characterizes pretty much the readiness of the Bluetooth support in the current emulator.