Using the Kinect as an Input Device for Open Wonderland

By Matthew Schmidt

Microsoft’s Kinect is a device that needs little introduction. This 3D camera is capable of tracking body movements and gestures, and packs a capable webcam as well. Shortly after its release for the Xbox 360, some industrial developers were able to create drivers for the device. What followed was a veritable explosion of innovation, with reports of Kinect hacking cropping up in diverse fields such as medicine and robotics. There is even a blog dedicated to hacking the Kinect, aptly titled “Kinect Hacks.” Clearly, the Kinect is a powerful device that has the potential to impact how we think about human computer interaction.

The University of Missouri iSocial project, where I work as a post-doctoral fellow, was kind enough to purchase a Kinect for our team to experiment with. My initial experiments were simple and were focused mainly on getting the device to connect to a Ubuntu Linux computer and display its output. Since we were very early adopters and the open source drivers were still in very early development, this was more of a chore than I anticipated, but I was ultimately able to get it to work.

Kinect connected and outputting 3D (left) and webcam (right) images

3D (left) and webcam (right)

Success! Kinect connected and outputting 3D (left) and webcam (right) images.

Following this, I began to look for ways to use the Kinect to interface with Open Wonderland. The rest of this blog post outlines how I was able to do just this.

There are a few prerequisites. You will need a Microsoft Kinect and a newer computer running Windows 7. While other versions of Windows may work (and perhaps even other operating systems), I have not experimented with them.

The software I use to interface with the connect is FAAST, which stands for “Flexible Action and Articulated Skeleton Toolkit.” You can download the software at the FAAST website. Note that in order to use FAAST, you will need to download OpenNI v1.0.0.25, PrimeSense NITE v1.3.0.18, and hardware drivers for the Kinect. Instructions and links for doing this are located at the FAAST website.

To start capturing input from the Kinect, you must first calibrate the Kinect. To do this, stand in front of the Kinect and position your body in the skeleton calibration pose.

Calibration pose

Calibration pose

Once the software recognizes your skeleton, you are ready to begin experimenting with moving your avatar around in Open Wonderland using the Kinect as an input device.

When you load the FAAST software for the first time, the default key bindings will work for controlling your avatar in Open Wonderland, since they are mapped to W, A, S and D. You will not, however, be able to use your mouse, select objects, or do anything advanced. In order to move your avatar forward, simply lean forward. To move backwards, lean backwards. To rotate left, lean left. And to rotate right, lean right. The video below is a demonstration of me using the default key bindings to move my avatar.

As an aside, I’m using the free BB FlashBack screen recording software for screen capture.

More advanced functionality can be achieved by reading the documentation on the FAAST homepage and editing the FAAST.cfg file by hand. You can use the FAAST forums to ask questions and see others’ configuration files. I provide the configuration file that I used below. Here’s a video of me using it.

My next steps include coming up with a way to translate real-world gestures into avatar gestures and refining the parameters which control the avatar.

While this research is still in the early phases, it is clear that the Kinect indeed holds promise as a potential human interface device for Open Wonderland. There is still a long way to go, however, and problems to solve before the Kinect will be usable enough for everyday usage. In particular, the software which controls the Kinect and translates body movement into avatar movement will need to improve significantly. And, hopefully, a completely open source solution that is more feature rich than libfreenect will become available.


# FAAST 0.07 configuration file

[Sensor]

sensor_resolution 0
mirror_mode 0
smoothing_factor 0
skeleton_mode 0
focus_gesture 0

[Mouse]

mouse_enabled 1
mouse_control 0
mouse_body_part 1
mouse_origin 1
mouse_left_bound 12
mouse_right_bound 12
mouse_bottom_bound 12
mouse_forward_threshold 12
mouse_top_bound 12
mouse_relative_speed 30
mouse_movement_threshold 2
mouse_multiple_monitors 0

[Actions]

# mappings from input events to output events
# format: event_name threshold output_type event

left_arm_forwards 24 mouse_click left_button

turn_left 30 key_hold a
turn_right 30 key_hold a
walk 3 key_hold w
lean_backwards 15 key_hold s

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: