Copied from:
http://beatsportable.com/forum/viewtopic.php?f=4&t=1023&p=1346#p1346
(Keripo = Philip Peng)
=============================================================
Finished up writing the full report with instructions, etc. Includes all relevant documents, source code modifications, photos, and of course a video of the actual demo ; )
Report: 2011-05-09 ESE350 Final Project - Report.pdf
Video: Beats, Advanced Rhythm Game - 1.5.5b Kinect PoC Demo [Android]
Files: http://beatsportable.com/static/kinect/
Blog: Dance With Your Hands
For installation instructions, download all the files in the "install/" folder and skip down to "5. b)" in the report.
Happy hacking!
~Keripo
=============================================================
Tuesday, May 10, 2011
Monday, May 9, 2011
Demo!
Following the successful implementation of our project base goal, we proceeded to add in the multi-input feature, allowing the player to input more than one directional input at once.
On top of that, the slowdown issue was fixed and the game is essentially 100% playable now! Check out our demo video above!
 |
| Demo day setup |
 |
| Demo day setup |
 |
| Demo day setup |
Integrating Modded ofsample Functionalities with Beats
Fri 4/13/11
Integrating Beats with Kinect was much trickier than we originally thought. The main obstacle we faced was that the Kinect refused to become active when we booted Beats up. After many hours of trial and error, we finally managed to keep the Kinect active (evident from its green LED) continuously while running Beats.
The next step is to modify the Beats source code to respond appropriately when the Kinect detects the corresponding inputs. This was accomplished fairly painlessly with Phil modifying his own codes. Following another lengthy episode of debuggings and adjustments, we finally arrive at SUCCESS!
The Beats+Kinect ran at about 15-20 frames per second. We tried to narrow down what was causing the major slowdown but could not find the source as we were thoroughly exhausted by then. Due to lag, the player would have to raise their hand into the Kinect's detection zone about half a second prior to the arrows reaching to the bottom. However, the game was ultimately playable, and we were able to achieve 10+ combos with acceptable accuracy.
Integrating Beats with Kinect was much trickier than we originally thought. The main obstacle we faced was that the Kinect refused to become active when we booted Beats up. After many hours of trial and error, we finally managed to keep the Kinect active (evident from its green LED) continuously while running Beats.
The next step is to modify the Beats source code to respond appropriately when the Kinect detects the corresponding inputs. This was accomplished fairly painlessly with Phil modifying his own codes. Following another lengthy episode of debuggings and adjustments, we finally arrive at SUCCESS!
 |
| Success! |
The Beats+Kinect ran at about 15-20 frames per second. We tried to narrow down what was causing the major slowdown but could not find the source as we were thoroughly exhausted by then. Due to lag, the player would have to raise their hand into the Kinect's detection zone about half a second prior to the arrows reaching to the bottom. However, the game was ultimately playable, and we were able to achieve 10+ combos with acceptable accuracy.
Modifying ofsample
Thur 4/28/11
As mentioned previously, the package we downloaded included the ofsample source code. After, many trials of figuring out what does what in the sample code, we made the appropriate changes and compiled the code in Eclipse. The code was altered to have similar functionalities as the code we wrote for Kinect + Ubuntu.
We made a minor modification, changing the 4 corner + 1 middle input boxes into 4 input boxes laid out horizontally. This is because of the switch we made from the StepMania simulator on Ubuntu to Beats on Android.
Downloads:
Modified ofsample source codes: ofsample.zip
Modified ofsample compiled apk: ofsamplemod.apk
As mentioned previously, the package we downloaded included the ofsample source code. After, many trials of figuring out what does what in the sample code, we made the appropriate changes and compiled the code in Eclipse. The code was altered to have similar functionalities as the code we wrote for Kinect + Ubuntu.
We made a minor modification, changing the 4 corner + 1 middle input boxes into 4 input boxes laid out horizontally. This is because of the switch we made from the StepMania simulator on Ubuntu to Beats on Android.
 |
| Modifying ofsample in Eclipse |
 |
| Running openFrameworks Sample |
 |
| Running openFrameworks Sample |
Modified ofsample source codes: ofsample.zip
Modified ofsample compiled apk: ofsamplemod.apk
Saturday, April 30, 2011
Kinect + Android
Tue 4/26/11
After many researches, we found a guide from Japan (conveniently in English) that allowed us to work with Kinect on our Beagleboard running Android.
This guide used these open-source software:
OpenFrameworks
ofxKinect(Kinect for OpenFrameworks)
ofxAndroid(Android for OpenFrameworks)
Steps 3 and 4 in the guide were the necessary steps to achieve our goal.
Step 3. Change Android's configuration
we added the following line
Step 4. Upgrade Android's kernel
libusb MUST neet "USB FS". And Kinect MUST neet "High Speed
USB(USB2.0)".
Check the kernel option in .config
Upgrading the kernel involved recompiling the it and took quite a while for us to finish (upwards of 20-30 minutes).
Now that Kinect can be recognized through the USB on Beagleboard, we downloaded the Full "OpenFrameworks x kinect x Android" Development Environment for beagleboard provided by the guide as well. The extracted files included an openFrameworks sample that we compiled into an .apk file. Using the same procedures to install Beats on Android+Beagleboard in the previous post, the openFrameworks sample installed and ran smoothly.
After many researches, we found a guide from Japan (conveniently in English) that allowed us to work with Kinect on our Beagleboard running Android.
This guide used these open-source software:
OpenFrameworks
ofxKinect(Kinect for OpenFrameworks)
ofxAndroid(Android for OpenFrameworks)
Steps 3 and 4 in the guide were the necessary steps to achieve our goal.
Step 3. Change Android's configuration
we added the following line
mount usbfs none /proc/bus/usb -o devmode=0666into the init.rc file (after all file permissions and before services)
Step 4. Upgrade Android's kernel
libusb MUST neet "USB FS". And Kinect MUST neet "High Speed
USB(USB2.0)".
Check the kernel option in .config
CONFIG_USB_DEVICEFS=yCopy new kernel onto sd card
Upgrading the kernel involved recompiling the it and took quite a while for us to finish (upwards of 20-30 minutes).
Now that Kinect can be recognized through the USB on Beagleboard, we downloaded the Full "OpenFrameworks x kinect x Android" Development Environment for beagleboard provided by the guide as well. The extracted files included an openFrameworks sample that we compiled into an .apk file. Using the same procedures to install Beats on Android+Beagleboard in the previous post, the openFrameworks sample installed and ran smoothly.
At first glance, the sample looks promising with fairly good and stable frame rates. However, the new OpenFrameworks with ofxKinect addon would require us to learn how to code with the new library and reprogram the functionalities we achieved with Kinect+Ubuntu.
New SD card + Android Running Beats on Beagleboard
Sun 4/25/11
We procured a new micro SD card from RadioShack and repeated the same steps in our previous post to boot the Beableboard into Android. As a precaution, a fan kept the board cool at all times to hopefully avoid the same overheat problem that occurred before.
Recall the several network and installation issues that prevented us from getting Phil's Beats to work. This time, we managed to discover a hackish way to open and run the application's .apk file without the need for internet access on the Beagleboard (credits to Phil for finding this out). It involved three simple steps:
1. Placing the .apk package onto the SD card's 3rd partition. (This is done on a pc)
2. Insert SD card into Beagleboard and boot into Android.
Select web browser and enter
file:///sdcard//<appname>.apk
in the address bar.
3. Follow the on screen steps to complete installation and open the application.
We were delighted to see that the Beats game had a great performance and ran at the maximum 60 fps almost constantly!
Our next milestone will be getting Kinect to work with Android like it did with Ubuntu. However, this is likely to be more challenging given the relatively poor documentation on this topic.
We procured a new micro SD card from RadioShack and repeated the same steps in our previous post to boot the Beableboard into Android. As a precaution, a fan kept the board cool at all times to hopefully avoid the same overheat problem that occurred before.
Recall the several network and installation issues that prevented us from getting Phil's Beats to work. This time, we managed to discover a hackish way to open and run the application's .apk file without the need for internet access on the Beagleboard (credits to Phil for finding this out). It involved three simple steps:
1. Placing the .apk package onto the SD card's 3rd partition. (This is done on a pc)
2. Insert SD card into Beagleboard and boot into Android.
Select web browser and enter
file:///sdcard//<appname>.apk
in the address bar.
3. Follow the on screen steps to complete installation and open the application.
We were delighted to see that the Beats game had a great performance and ran at the maximum 60 fps almost constantly!
Our next milestone will be getting Kinect to work with Android like it did with Ubuntu. However, this is likely to be more challenging given the relatively poor documentation on this topic.
Beagleboard Not Fast Enough for Ubuntu
Sat 4/23/11
With Kinect working on Beagleboard running Ubuntu, we proceeded to download and installed Stepmania Simulator (Pump It Up game-mode). A screenshot can be seen below.
To our disappointment, the game runs at very low frame rate (less than 5 fps), so it is virtually unplayable.
This drove us to reconsider running Android again, especially after seeing the blazing speed of 2D and 3D graphics demos that came with the Texas Instruments' pre-built Android image.
With Kinect working on Beagleboard running Ubuntu, we proceeded to download and installed Stepmania Simulator (Pump It Up game-mode). A screenshot can be seen below.
 |
| Stepmania (SM-SSC) Running on Beagleboard with Ubuntu |
To our disappointment, the game runs at very low frame rate (less than 5 fps), so it is virtually unplayable.
This drove us to reconsider running Android again, especially after seeing the blazing speed of 2D and 3D graphics demos that came with the Texas Instruments' pre-built Android image.
Thursday, April 28, 2011
New board + Ubuntu 10.10
Fri 4/22/11
After obtaining a new Beagleboard from Kevin, we decided to experiment with Ubuntu 10.10 following the instruction that can be found here. Because we suspected a bad power supply, the Beagleboard is powered through its mini USB port this time around.
sudo sh -c 'zcat ./ubuntu-netbook-10.10-preinstalled-netbook-armel+<omap image>.img.gz >/dev/<device name>'
Due to a known bug, two update files needed to be copied onto the SD card partition.
We immediately followed the same procedures as previously on a Linux PC to set up Kinect with the Beagleboard. The results were as expected when we ran our modified glview-demo code.
However, the response time was not very promising, approximately half a second between physical input to display output. This prompted us to reconsider running Android again seeing how fast other applications performed.
After obtaining a new Beagleboard from Kevin, we decided to experiment with Ubuntu 10.10 following the instruction that can be found here. Because we suspected a bad power supply, the Beagleboard is powered through its mini USB port this time around.
We downloaded the Texas Instruments OMAP3 preinstalled netbook image here. With the SD card inserted in our Linux PC, we then ran the following command to write the image onto the SD card.
Due to a known bug, two update files needed to be copied onto the SD card partition.
$ cd /media/6B57-ED78
$ wget http://people.canonical.com/~rsalveti/maverick/xM/ulmage -O ulmage
$ cd /media/87e09b0b-4ba6-4476-920b-2bb4445e9ae8/boot
$ wget http://people.canonical.com/~rsalveti/maverick/xM/vmlinuz-2.6.35-22-omap
 |
| Beagleboard + Ubuntu 10.10 |
We immediately followed the same procedures as previously on a Linux PC to set up Kinect with the Beagleboard. The results were as expected when we ran our modified glview-demo code.
However, the response time was not very promising, approximately half a second between physical input to display output. This prompted us to reconsider running Android again seeing how fast other applications performed.
Sunday, April 24, 2011
Late Post 3: Beagleboard Overheat?!
Tue 4/12/11
Now that Android is up and running on the Beagleboard, we wanted to try and run Beats (an android music-rhythm game developed by Phil). However, we encountered several issues in our download and install process.
The Android on Beagleboard becomes extremely unresponsive when download was initiated. Later, we managed to finally finish the download. However, the installation simply results in crashing the board. After several trials, the Beagleboard suddenly stopped booting even though all LED lights still turns on.
We touched the Beagleboard chip and found it to be burning hot. Unplugging the power supply cools the chip immediately, but it overheats again quickly in ~20 seconds after the power is reconnected.
Late Post 2: Getting Kinect to work on PC
Sat 4/16/11
Setting up Kinect on Linux desktop was fairly straightforward. The platform that we used was Fedora 14. We followed the openkinect guide using libfreenect library that can be found here under the section for Linux.
Following that, we modified the glview.c to implement the functionality that we desired, detecting the four corners and middle of the dance pad. First, we removed the regular video output as it would take extra resources. Comparison file between original and modified can be downloaded here: glview-depth-only.diff
To better distinguish the sections of detection, we made changes to the code further more and averaged out the depth information in blocks.The result was a pixelated figure whose pixels were easily countable. The depth information is summed each detection zone. When a summed depth is above the set threshold, the program outputs the corresponding zone in the Terminal.
Comparison File: glview-demo-day.diff
Finally, we removed all graphical output leaving only text output indicating which pads are detected. The text outputs are top-left, top-right, bottom-left, bottom-right, and middle ("TL", "TR", "BL", "BR", "MM").
Comparison File: glview-text-only.diff
Late Post 1: Running Android on Beagleboard!
Fri 4/15/11
We successfully booted Android 2.2 on our Beagleboard. We followed the 4 step instruction found here to prepare our SD card.
Step 1. Insert SD card on Linux PC
Step 2. Download pre-build Android 2.2 image for Beagleboard-XM here
Step 3. Putting Android onto SD card
first locate downloaded Android image and extract
tar -xzvf Beagleboard-XM.tar.gz
cd Beagleboard-XM.tar.gz
run provided script to prepare SD card
sudo ./mkmmc-android /dev/sd<device>
Step 4. Plug SD card into Beagleboard and power up
Booting Android was faster than expected, and the interface performed rather smoothly. We immediately went on to test out its Internet connectivity.
Ethernet worked out of the box but was unstable, mainly due to the network registration requirement by Penn. A few adjustments and trials got us onto the Internet. At this point, the Beagleboard had been increasingly hot on touch.
Saturday, April 23, 2011
The Idea
Our idea stemmed from considering simulation of a theremin using Kinect. That lead us to thinking about doing other cool things related to music with hand motion. Finally, it ended up with using Kinect as a controller to play Pump It Up with your hands instead of your stomping feet as seen below.
 |
| Concept art |
 |
| Actual dance pad for comparison |
To run the game, we will be using a beagleboard with Linux installed. Hopefully, the board will be fast enough for running the game and processing Kinect inputs at the same time.
Subscribe to:
Posts (Atom)