Kinect + Android

Tue 4/26/11

After many researches, we found a guide from Japan (conveniently in English) that allowed us to work with Kinect on our Beagleboard running Android.

This guide used these open-source software:

OpenFrameworks
ofxKinect(Kinect for OpenFrameworks)
ofxAndroid(Android for OpenFrameworks)

Steps 3 and 4 in the guide were the necessary steps to achieve our goal.

Step 3. Change Android's configuration

              we added the following line

mount usbfs none /proc/bus/usb -o devmode=0666

              into the init.rc file (after all file permissions and before services)

Step 4. Upgrade Android's kernel

              libusb MUST neet "USB FS". And Kinect MUST neet "High Speed
              USB(USB2.0)".

              Check the kernel option in .config

CONFIG_USB_DEVICEFS=y

              Copy new kernel onto sd card

Upgrading the kernel involved recompiling the it and took quite a while for us to finish (upwards of 20-30 minutes).

Now that Kinect can be recognized through the USB on Beagleboard, we downloaded the Full "OpenFrameworks x kinect x Android" Development Environment for beagleboard provided by the guide as well. The extracted files included an openFrameworks sample that we compiled into an .apk file. Using the same procedures to install Beats on Android+Beagleboard in the previous post, the openFrameworks sample installed and ran smoothly.

At first glance, the sample looks promising with fairly good and stable frame rates. However, the new OpenFrameworks with ofxKinect addon would require us to learn how to code with the new library and reprogram the functionalities we achieved with Kinect+Ubuntu.

New SD card + Android Running Beats on Beagleboard

Sun 4/25/11

We procured a new micro SD card from RadioShack and repeated the same steps in our previous post to boot the Beableboard into Android. As a precaution, a fan kept the board cool at all times to hopefully avoid the same overheat problem that occurred before.

Recall the several network and installation issues that prevented us from getting Phil's Beats to work. This time, we managed to discover a hackish way to open and run the application's .apk file without the need for internet access on the Beagleboard (credits to Phil for finding this out). It involved three simple steps:

1. Placing the .apk package onto the SD card's 3rd partition. (This is done on a pc)

2. Insert SD card into Beagleboard and boot into Android.
    Select web browser and enter
 
    file:///sdcard//<appname>.apk
 
    in the address bar.

3. Follow the on screen steps to complete installation and open the application.

We were delighted to see that the Beats game had a great performance and ran at the maximum 60 fps almost constantly!

Our next milestone will be getting Kinect to work with Android like it did with Ubuntu. However, this is likely to be more challenging given the relatively poor documentation on this topic.

Beagleboard Not Fast Enough for Ubuntu

Sat 4/23/11

With Kinect working on Beagleboard running Ubuntu, we proceeded to download and installed Stepmania Simulator (Pump It Up game-mode). A screenshot can be seen below.

Stepmania (SM-SSC) Running on Beagleboard with Ubuntu

To our disappointment, the game runs at very low frame rate (less than 5 fps), so it is virtually unplayable.

This drove us to reconsider running Android again, especially after seeing the blazing speed of 2D and 3D graphics demos that came with the Texas Instruments' pre-built Android image.

New board + Ubuntu 10.10

Fri 4/22/11

After obtaining a new Beagleboard from Kevin, we decided to experiment with Ubuntu 10.10 following the instruction that can be found here. Because we suspected a bad power supply, the Beagleboard is powered through its mini USB port this time around.

We downloaded the Texas Instruments OMAP3 preinstalled netbook image here. With the SD card inserted in our Linux PC, we then ran the following command to write the image onto the SD card.

sudo sh -c 'zcat ./ubuntu-netbook-10.10-preinstalled-netbook-armel+<omap image>.img.gz >/dev/<device name>'

Due to a known bug, two update files needed to be copied onto the SD card partition.

$ cd /media/6B57-ED78

$ wget http://people.canonical.com/~rsalveti/maverick/xM/ulmage -O ulmage

$ cd /media/87e09b0b-4ba6-4476-920b-2bb4445e9ae8/boot

$ wget http://people.canonical.com/~rsalveti/maverick/xM/vmlinuz-2.6.35-22-omap

Beagleboard + Ubuntu 10.10

We immediately followed the same procedures as previously on a Linux PC to set up Kinect with the Beagleboard. The results were as expected when we ran our modified glview-demo code.

However, the response time was not very promising, approximately half a second between physical input to display output. This prompted us to reconsider running Android again seeing how fast other applications performed.

Late Post 3: Beagleboard Overheat?!

Tue 4/12/11

Now that Android is up and running on the Beagleboard, we wanted to try and run Beats (an android music-rhythm game developed by Phil). However, we encountered several issues in our download and install process.

The Android on Beagleboard becomes extremely unresponsive when download was initiated. Later, we managed to finally finish the download. However, the installation simply results in crashing the board. After several trials, the Beagleboard suddenly stopped booting even though all LED lights still turns on. 

We touched the Beagleboard chip and found it to be burning hot. Unplugging the power supply cools the chip immediately, but it overheats again quickly in ~20 seconds after the power is reconnected.

Late Post 2: Getting Kinect to work on PC

Sat 4/16/11

Setting up Kinect on Linux desktop was fairly straightforward. The platform that we used was Fedora 14. We followed the openkinect guide using libfreenect library that can be found here under the section for Linux.

After we fired up the glview test demo, we can see two video frames (depth and regular video).

Following that, we modified the glview.c to implement the functionality that we desired, detecting the four corners and middle of the dance pad. First, we removed the regular video output as it would take extra resources. Comparison file between original and modified can be downloaded here: glview-depth-only.diff

To better distinguish the sections of detection, we made changes to the code further more and averaged out the depth information in blocks.The result was a pixelated figure whose pixels were easily countable. The depth information is summed each detection zone. When a summed depth is above the set threshold, the program outputs the corresponding zone in the Terminal.

Comparison File: glview-demo-day.diff

Finally, we removed all graphical output leaving only text output indicating which pads are detected. The text outputs are top-left, top-right, bottom-left, bottom-right, and middle ("TL", "TR", "BL", "BR", "MM").

Comparison File: glview-text-only.diff

Late Post 1: Running Android on Beagleboard!

Fri 4/15/11

We successfully booted Android 2.2 on our Beagleboard. We followed the 4 step instruction found here to prepare our SD card.

  Step 1. Insert SD card on Linux PC

  Step 2. Download pre-build Android 2.2 image for Beagleboard-XM here

  Step 3. Putting Android onto SD card

              first locate downloaded Android image and extract

tar -xzvf Beagleboard-XM.tar.gz

cd Beagleboard-XM.tar.gz

              run provided script to prepare SD card

sudo ./mkmmc-android /dev/sd<device>

  Step 4. Plug SD card into Beagleboard and power up

Booting Android was faster than expected, and the interface performed rather smoothly. We immediately went on to test out its Internet connectivity.

Ethernet worked out of the box but was unstable, mainly due to the network registration requirement by Penn. A few adjustments and trials got us onto the Internet. At this point, the Beagleboard had been increasingly hot on touch.

The Idea

Our idea stemmed from considering simulation of a theremin using Kinect. That lead us to thinking about doing other cool things related to music with hand motion. Finally, it ended up with using Kinect as a controller to play Pump It Up with your hands instead of your stomping feet as seen below.

Concept art

Actual dance pad for comparison

To run the game, we will be using a beagleboard with Linux installed. Hopefully, the board will be fast enough for running the game and processing Kinect inputs at the same time.