Infrared imaging with Android devices

One of the most evident sensors of Android devices is the camera. An ordinary smartphone's camera is able to capture a lot of interesting information but has its limitations too. Most evidently, its viewing angle depends on the position of the device (so it is not fixed and hard to measure) and its bandwidth is (mostly) restricted to the visible light. It is therefore an exciting idea to connect special cameras to Android devices.




In this post, I will present an integration of FLIR Lepton Long-wavelength Infrared Camera to an Android application over Bluetooth Low Energy connection. Long-wavelength IR (LWIR) cameras are not new. Previously, however, they were priced in the thousands of dollars range (if not higher). Lepton is still pricey (currently about 300 USD) but its price is low enough so that mere mortals can play with it. FLIR sells a smartphone integration product (called FLIR One) but it is currently only available for iPhone and locks the camera to one device. Our prototype allows any device with BLE connection to access this very special camera.

The prototype system presented here needs a relatively long list of external hardware components and it is also not trivial to prepare these components. This list is the following:

• An Android phone with Bluetooth 4.0 capability. I used Nexus 5 for these experiments.
• An FLIR Lepton module. My recommendation is the FLIR Dev Kit from Sparkfun that has the camera module mounted on a breakout panel that is much easier to handle than the original FLIR socket.
• A BeagleBone Black card with an SD Card >4GB.
• A BLED112 BLE dongle from Silicon Labs (formerly Bluegiga).

The software for the prototype can be downloaded in two packages.

• This package contains the stuff for the embedded computer.
• This package is the Android application that connects to it.
  

Once you got all these, prepare the ingredients.

1. Hook up the FLIR camera with the BeagleBone Black

Fortunately the BBB's SPI interface is completely compatible with the Lepton's so the "hardware" just needs a couple of wires. Do the following connections (P9 refers to the BBB's P9 extension port).

FLIR	BBB
CS	P9/28 (SPI1_CS0)
MOSI	P9/30 (SPI1_D1)
MISO	P9/29 (SPI1_D0)
CLK	P9/31 (SPI1_SCLK)
GND	P9/1 (GND)
VIN	P9/4 (DC, 3.3V)

2. Prepare the BBB environment

I use Snappy Ubuntu. Grab the SD card and download the image as documented here. Before flashing the SD card, we have to update the device tree in the image so that the SPI port is correctly enabled. Unpack bt_ircamera.zip that you have just downloaded and go to the dt subdirectory. There you find a device tree file that I used for this project. Beside the SPI1 port, it also enables some serial ports. These are not necessary for this project but may come handy.

Compile the device tree:

dtc -O dtb -o am335x-boneblack.dtb am335x-boneblack.dts

The output is the binary device tree (am335x-boneblack.dtb) that needs to be put into the kernel image file. Let's suppose that the downloaded image file is ubuntu-15.04-snappy-armhf-bbb.img and you have an empty directory at /mnt/img. Then do the following:

fdisk -l ubuntu-15.04-snappy-armhf-bbb.img

Look for the first partition and note the partition image name and the offset:

ubuntu-15.04-snappy-armhf-bbb.img1   *        8192      139263       65536    c  W95 FAT32 (LBA)
...

Note that the actual partition image name may differ depending on the Snappy image you downloaded. Calculate the offset as 8192*512=4194304
Now mount the partition:

mount -o loop,offset=4194304 ubuntu-15.04-snappy-armhf-bbb.img /mnt/img

Then copy the dtb into the image, unmount and write the image to SD card (on my computer the SD card interface is /dev/sdc, check before you issue the dd command!):

cp am335x-boneblack.dtb /mnt/img/a/dtbs
umount /mnt/img
dd if=ubuntu-15.04-snappy-armhf-bbb.img of=/dev/sdc bs=32M

Now you have an SD card that you can insert into the BBB and boot from it. Once you reached the Ubuntu prompt and logged in (ubuntu/ubuntu), there's one thing more: the Snappy prototype application depends on the libpng package which is not part of the default Snappy image. But before you do it, check whether the SPI device was enabled correctly:
root@localhost:~# ls /dev/spidev1.0                                            
/dev/spidev1.0

Now about the png library. Download the armhf image from this location:

wget http://ports.ubuntu.com/pool/main/libp/libpng/libpng12-0_1.2.50-1ubuntu2_armhf.deb


Copy it to the BBB (update your card's IP address according to your network policies):
scp libpng12-0_1.2.50-1ubuntu2_armhf.deb ubuntu@192.168.1.115:~

Then go to the BBB console and install the deb package:
sudo mount -o remount,rw /
sudo dpkg -i libpng12-0_1.2.50-1ubuntu2_armhf.deb
sudo mount -o remount,ro /

3. Prepare the BLE dongle

The BLED112 stores the GATT tree in its firmware, hence in order to provide the GATT services that connect the BBB with the Android device, a new firmware needs to be generated and installed in the dongle. The config files are located in the config subdirectory in the bt_ircamera.zip archive. Follow the steps in this post to generate and install the new firmware. Once you are done, you can simply plug the dongle into the USB port of the BBB.

4. Install the prototype applications

The prototype system has two parts. The application running on the BBB acts as BLE server, fetches images from the FLIR camera and transmits them over BLE. The Android application acts as BLE client, fetches images from the BLE server and displays them. The BBB part is located in bt_ircamera.zip and the Android part is in IRCamera.zip. The latter is just the source part of the Android Studio project tree - I omitted all the garbage that Android Studio generates into the project folders. For the BBB installation, follow the instructions in this blog post. Launch the BBB application like this as root:
/apps/ircamera.sideload/1.0.0/bin/ircamera /dev/ttyACM0
and you are ready to go. On the Android side, select the BLE node with the name "test", connect, click the "Take picture" button, wait for the image to download and there you are. Note that the images are saved on the SD card, which means that they also appear in the stock "Photos" application.

Now at last we can get to the technical issues with this prototype. One interesting aspect is that there is no standard BLE service that provides the functionalities - image capture triggering, image fetching - our system needs. That's not a problem, we defined our own BLE service. It is easiest to follow this service in irc_gattBLED112.xml (bt_ircamera.zip, config subdirectory).

The service has a custom UUID, generated randomly:

<service uuid="274b15a3-b9cd-4e5e-94c4-1248b42b82f8" advertise="true">

Also, its 3 GATT characteristics are in the non-standard UUID domain:

<characteristic uuid="00000000-b9cd-4e5e-94c4-1248b42b82f8" id="irc_len">
...
<characteristic uuid="00000001-b9cd-4e5e-94c4-1248b42b82f8" id="irc_offs">
...
<characteristic uuid="00000002-b9cd-4e5e-94c4-1248b42b82f8" id="irc_pic">

The interaction goes like the following. The BLE client connects and reads the irc_len characteristic. This characteristic is tagged as "user" on the BLE server side meaning that the BLE application must generate the value on the fly, when the attribute is read. In our case, reading this attribute fetches an image from the FLIR camera, converts it into PNG format and stores it in the apps' data folder, returning only the PNG file size. The Android application now can fetch the image piece by piece. First the Android application writes the irc_offs characteristic to inform the BLE server, what is the starting location of the fragment it wants to fetch. Then it reads the irc_pic characteristic which returns a maximum of 20 bytes of image data. This makes the image download very slow (takes about 10-20 second to download a general 5-6 Kbyte image to the Android application) but the restriction comes from a BLE protocol layer. Maybe the old RFCOMM from Bluetooth Classic would have been actually a better option for this application.

Update: I updated the client/server application to make the download faster (it is still quite slow). In order to speed up, I removed the explicit setting of the file offset (so the irc_offs characteristic is not used anymore). This made the download faster but there's still room for improvement.

Other than the issue with fragment size, both applications are pretty straighforward. Maybe the colors of the IR image are worth discussing a bit. The FLIR camera returns a matrix of 80x60 pixels, each pixel has a depth of 12 bit. Grayscale presentation is the most evident option but most displays have only 256 gray colors. In order to make the IR shades more visible, I used fake coloring. The algorithm is very simple: after the image is fetched, the maximum and the minimum IR intensity is calculated and the range between the two are mapped into a rainbow gradient of 400 colors.