Tuesday, May 10, 2011

Report + File Downloads!

Copied from:
(Keripo = Philip Peng)


Finished up writing the full report with instructions, etc. Includes all relevant documents, source code modifications, photos, and of course a video of the actual demo ; )

Report: 2011-05-09 ESE350 Final Project - Report.pdf
Video: Beats, Advanced Rhythm Game - 1.5.5b Kinect PoC Demo [Android]
Files: http://beatsportable.com/static/kinect/
Blog: Dance With Your Hands

For installation instructions, download all the files in the "install/" folder and skip down to "5. b)" in the report.

Happy hacking!



Monday, May 9, 2011


Following the successful implementation of our project base goal, we proceeded to add in the multi-input feature, allowing the player to input more than one directional input at once.

On top of that, the slowdown issue was fixed and the game is essentially 100% playable now! Check out our demo video above!

Demo day setup
Demo day setup
Demo day setup

Integrating Modded ofsample Functionalities with Beats

Fri 4/13/11

Integrating Beats with Kinect was much trickier than we originally thought. The main obstacle we faced was that the Kinect refused to become active when we booted Beats up. After many hours of trial and error, we finally managed to keep the Kinect active (evident from its green LED) continuously while running Beats.

The next step is to modify the Beats source code to respond appropriately when the Kinect detects the corresponding inputs. This was accomplished fairly painlessly with Phil modifying his own codes. Following another lengthy episode of debuggings and adjustments, we finally arrive at SUCCESS!


The Beats+Kinect ran at about 15-20 frames per second. We tried to narrow down what was causing the major slowdown but could not find the source as we were thoroughly exhausted by then. Due to lag, the player would have to raise their hand into the Kinect's detection zone about half a second prior to the arrows reaching to the bottom. However, the game was ultimately playable, and we were able to achieve 10+ combos with acceptable accuracy.

Modifying ofsample

Thur 4/28/11

As mentioned previously, the package we downloaded included the ofsample source code. After, many trials of figuring out what does what in the sample code, we made the appropriate changes and compiled the code in Eclipse. The code was altered to have similar functionalities as the code we wrote for Kinect + Ubuntu.

We made a minor modification, changing the 4 corner + 1 middle input boxes into 4 input boxes laid out horizontally. This is because of the switch we made from the StepMania simulator on Ubuntu to Beats on Android.

Modifying ofsample in Eclipse
Running openFrameworks Sample
Running openFrameworks Sample
Modified ofsample source codes: ofsample.zip
Modified ofsample compiled apk: ofsamplemod.apk

Saturday, April 30, 2011

Kinect + Android

Tue 4/26/11

After many researches, we found a guide from Japan (conveniently in English) that allowed us to work with Kinect on our Beagleboard running Android.

This guide used these open-source software:

ofxKinect(Kinect for OpenFrameworks)
ofxAndroid(Android for OpenFrameworks)

Steps 3 and 4 in the guide were the necessary steps to achieve our goal.

Step 3. Change Android's configuration

              we added the following line
mount usbfs none /proc/bus/usb -o devmode=0666
              into the init.rc file (after all file permissions and before services)

Step 4. Upgrade Android's kernel

              libusb MUST neet "USB FS". And Kinect MUST neet "High Speed

              Check the kernel option in .config
              Copy new kernel onto sd card

Upgrading the kernel involved recompiling the it and took quite a while for us to finish (upwards of 20-30 minutes).

Now that Kinect can be recognized through the USB on Beagleboard, we downloaded the Full "OpenFrameworks x kinect x Android" Development Environment for beagleboard provided by the guide as well. The extracted files included an openFrameworks sample that we compiled into an .apk file. Using the same procedures to install Beats on Android+Beagleboard in the previous post, the openFrameworks sample installed and ran smoothly.

At first glance, the sample looks promising with fairly good and stable frame rates. However, the new OpenFrameworks with ofxKinect addon would require us to learn how to code with the new library and reprogram the functionalities we achieved with Kinect+Ubuntu.

New SD card + Android Running Beats on Beagleboard

Sun 4/25/11

We procured a new micro SD card from RadioShack and repeated the same steps in our previous post to boot the Beableboard into Android. As a precaution, a fan kept the board cool at all times to hopefully avoid the same overheat problem that occurred before.

Recall the several network and installation issues that prevented us from getting Phil's Beats to work. This time, we managed to discover a hackish way to open and run the application's .apk file without the need for internet access on the Beagleboard (credits to Phil for finding this out). It involved three simple steps:

1. Placing the .apk package onto the SD card's 3rd partition. (This is done on a pc)

2. Insert SD card into Beagleboard and boot into Android.
    Select web browser and enter
    in the address bar.

3. Follow the on screen steps to complete installation and open the application.

We were delighted to see that the Beats game had a great performance and ran at the maximum 60 fps almost constantly!

Our next milestone will be getting Kinect to work with Android like it did with Ubuntu. However, this is likely to be more challenging given the relatively poor documentation on this topic.

Beagleboard Not Fast Enough for Ubuntu

Sat 4/23/11

With Kinect working on Beagleboard running Ubuntu, we proceeded to download and installed Stepmania Simulator (Pump It Up game-mode). A screenshot can be seen below.

Stepmania (SM-SSC) Running on Beagleboard with Ubuntu

To our disappointment, the game runs at very low frame rate (less than 5 fps), so it is virtually unplayable.

This drove us to reconsider running Android again, especially after seeing the blazing speed of 2D and 3D graphics demos that came with the Texas Instruments' pre-built Android image.