Breaking Out … Of Prison? Into Song?

February 26, 2009

As odd as it sounds, release testing is one of the most fun things we do. It’s a time when lots of new functionality comes together, so it’s exciting to see everyone’s hard work merged into a single build. But it’s also a time when unexpected, and sometimes very funny, things happen. Here are some tidbits from today’s test of the upcoming Wonderland v0.5 Dev3 release.

First, we searched around for a scene we could include as a default world since our new default world is still under construction and not ready for distribution yet. As you’ve seen in one of my previous posts (Wonderland 0.5 – The Prehistoric Age), we have been using a Stonehenge model for testing. This model is attractive, but the stones make it tricky to navigate. We eventually found a model in the Google 3D Warehouse with more open spaces and some nice arches. When you look at it facing the arches, it’s rather appealing:

3D model of arches

The problem is, the other walls in the model have windows with bars on them. Since the new avatar system integration is not yet complete, Dev3 only includes a single bald, gray-shirted avatar. While this look might appeal to our friend Ric who is doing important work with the prison system, it was not quite the look we were aiming for!

Wonderland inmates

We solved the problem by editing the model in Google SketchUp and removing all the walls other than the arches.

Moving on, we next tested the new move, rotate and scale controls. This went extremely well…even if one of us did take a bit of damage when impaled by the move tool :-)

Getting impaled by the move affordance

We were also able to see the SVG Whiteboard in world for the first time, complete with an elegant tool palette on the HUD or heads-up display. This refers to the glass pane that sits in front of the virtual world.

SVG Whiteboard

Too bad none of us can draw! The window frame looks clunky right now, but we should see move icons and a take-control button on the frame in Dev4. For now, you can right-click on the window, select "edit" from the context menu, and move the whiteboard using the move and rotate controls.

By far the best part of today’s session came when we tested the audio recorder. We were able to successfully select the audio recorder option from the cell palette and place it in world. We recorded some audio with it and saved it to a "tape." Then we were brave enough to try duplicating the recorder by right-clicking on it and selecting "duplicate" from the context menu. What do you know, we got a second recorder and both recorders worked! And how better to test playback but to sing some rounds. For your listening pleasure, I secretly recorded a snippet:

Wonderland v0.5 Audio Recorder Test from Nicole Yankelovich on Vimeo.

I’m running for cover now. Hopefully one day I’ll be forgiven for posting this video!


Successful test of WL v0.4 with 28 users

February 25, 2009

Thierry Bücheler, on behalf of the "ShanghAI Lectures" project team based at the University of Zurich, has contributed the following guest blog to share his team’s Wonderland experience with the rest of the community.


The Artificial Intelligence Lab at the University of Zurich (UZH), Switzerland, is extensively testing Wonderland 0.4 and 0.5 distributions in order to prepare a global lecture series called the "ShanghAI Lectures" (http://shanghailectures.org) starting in October 2009. The tests on the current release 0.4 are necessary since the team runs a "live" pilot (a joint seminar between two Swiss universities), starting on February 26 (and hence running on WL 0.4). In the context of the ShanghAI Lectures, many scientific plugins for Wonderland are written that track avatars (e.g., record voice conversations or avatar’s moving behavior) and allow virtual world researchers to collect data on virtual team behavior.

The test included 28 users on different HW/OS settings and was a success!

Wonderland v0.4 test

Details

Last Thursday a group of 28 people located in different spots throughout Switzerland met in the University of Zurich’s Wonderland world (http://wonderland.ifi.uzh.ch) to test the server’s performance under such load. Users were using very different HW/OS settings (e.g., Ubuntu 8.10, Windows XP, MacOS X on 32 and 64 bit machines). Some clients were below the minimum specs mentioned here: http://wiki.java.net/bin/view/Javadesktop/ProjectWonderlandAbout#ClientSystemRequirements (e.g., laptop computers with built-in graphic cards without 3D/OpenGL acceleration, running Windows XP on a Duo Core 1.6 GHz processor with 3 GB RAM). One interesting detail to mention: On the 64 bit dual-boot machines that were used, WL 0.4 did run, but only on the Windows installation, not on Ubuntu 8.10 (even though SUN Java was used in Ubuntu).

Users were logged in to the world for about 30 minutes, during which time there was a number of concurrent edits on OpenOffice documents, looking at the built-in video player and webcam application, all whilst chatting to each other through text and audio.

This test run was intended to stress the server in terms of number of users and audio recording. For this, the recording capabilities of jvoicebridge were employed, whereupon each user’s individual audio stream was saved to disk.

There was no issue with avatar control and text chat. Audio worked well until about 23 users were logged in, whereafter it became choppy. The reason for this behaviour is still to be determined, however whilst monitoring the server processes of the sgs, voicebridge and smc client, the sgs server seem to barely be coping, consuming nearly 100% of the cpu cycles whilst the voice-bridge consumption was only about 10-15%. The max CPU usage was at 140% with 28 concurrent users.

The server that was used has the following specs: Intel Pentium Core Duo CPU 3.40Ghz, 4 GB RAM, running debian etch.

The "ShanghAI Lectures" project team


Persistence in Wonderland 0.5

February 24, 2009

In the real world, we tend to take persistence for granted.  If I move a book into my office, I expect it will still be on my desk the next time I go looking for it.  In Wonderland 0.4, however, this type of world persistence doesn’t exist: whenever you restart the Wonderland server, it resets the world to its original state.  This can be quite frustrating for spaces like the team rooms shown below.  We would really like the team room to persist as users add documents, request comments, and generally live in the space. The room itself should evolve with and reflect the current state of the team that works there, as happens in real-world team rooms.

Team rooms in the real world (my office) and in Wonderland
Sun Labs Big Room Wonderland team room

In Wonderland 0.5, we have added several new features to help maintain world persistence.  From the user’s perspective, this means that Wonderland worlds will act more like the real world, which is usually a good thing.  But under the covers, there are some complexities that Wonderland developers and administrators will need to pay attention to.  The rest of this post will talk about the mechanics of world persistence in Wonderland 0.5.

Mechanisms for persistence

In Wonderland, we support two general mechanisms for persistence:

  • For short term persistence, Wonderland uses the Darkstar datastore. As you change the world, the state is continuously written to the Darkstar data store as serialized objects.  This happens as part of Darkstar’s built-in mechanisms for data integrity, so there is no special work on Wonderland’s part for this type of persistence.  Since Darkstar is event driven, whenever an event occurs, Darkstar retrieves the relevant objects from its datastore, and then stores the updated version when the event completes.  The catch is that, due to all this object serialization, data in the Darkstar datastore becomes difficult to manage if the server code changes.  Thus the Darkstar datastore is appropriate for saving the world state through server restarts, but not across different versions of the server or as code changes.  The Darkstar datastore is also an opaque storage mechanism — only live Darkstar applications can access it, so it cannot be used to share worlds.
  • For long term persistence, Wonderland uses an XML serialization mechanism we call WFS (which originally stood for "Wonderland File System", an acronym that may be a bit out of date at this point).  WFS describes a Wonderland world as a set of XML files, arranged in a tree structure just like the Wonderland cell hierarchy.  Each file describes the properties of a particular cell in the Wonderland world.  WFS is managed via a web service in the Wonderland web server.  Currently, writing WFS requires the world to be in a quiesced state with no users logged in, and typically requires a server restart.  Once it is written, however, WFS is independent of any Java code, so can be used to persist worlds when the server code changes.  It also doubles as a interchange format, so worlds can be modified and shared as a tree of WFS text files.

Given these two overlapping persistence mechanisms, we had to determine the rules for when each is used. Typically, our goal is to use the short-term Darkstar persistence whenever possible. So when the world is running, we use Darkstar persistence exclusively. The good news for us is that this mostly happens automatically: Darkstar persists state by default, so unless we explicitly remove it, the world will remain up-to-date.  The decision point comes during a server restart.  We need to decide what type of persistence to use based on whether the code or the desired world has changed:

  • A warm restart happens when there is an existing Darkstar database, and this database is up-to-date with both the world that is selected and the latest Wonderland code.  In the case of a warm start, the server reads the current state from the Darkstar datastore on startup.  For most Wonderland server code, this is completely transparent: objects in Darkstar persistence come back as if the server had never been stopped.
  • A cold restart happens when the Darkstar database is out-of-date with the Wonderland code, there is no existing database, or the administrator has chosen to load a fresh world.  In this case, the state of the world is read at startup from the web server via WFS.  This is a fairly heavy-weight mechanism, and involves each cell getting set up individually based on the state in WFS.  The world that is read in might reset the server back to some well-known previous state, or it may restore a snapshot of the state from just before the cold start.

Managing snapshots

The administrator’s window into this restart behavior is the Wonderland snapshot manager, pictured below. When you open the snapshot manager, you will see a list of initial world images and snapshots of existing worlds.  You can choose which image to load by selecting "make current" or you can restore a world back to the state stored on disk by clicking "restore."  Both of these operations will force a cold restart of the server, so the Darkstar datastore will be replaced with state stored in the selected XML files.

Wonderland snapshot manager
Wonderland snapshot manager

In addition to managing the current world image, you can also create a new snapshot.  Snapshots store the current state of the world in a WFS directory, but do not change the Darkstar datastore. You can select "restore" to reload the world from one of these snapshots. 

WFS files, both from initial worlds and snapshots, are stored in the Wonderland run directory in the "wfs" subdirectory.  On my system, that is in /Users/jkaplan/.wonderland-server/0.5-dev/wfs.

Finally, it is important to note that when you install a new module that contains server code, the system will also force a cold restart.  The desired behavior is that the system will automatically create a snapshot of the world, install the new server code, and the restart from the snapshot. This should bring the world back to a state that is very similar to what it was prior to the restart.  As of Wonderland 0.5 dev3, this behavior is not fully implemented, so you must manually create a snapshot before installing modules if you want
to preserve your world state.

Developing for persistence

For Wonderland developers, there should be no additional work needed to support the short-term Darkstar persistence.  The normal Darkstar rules about using ManagedObject and Serializable will ensure that your server cells are persisted properly in the Darkstar datastore.  Since Darkstar is event driven, a restart will be transparent to your objects: the object is not notified in any way of the restart, although you may notice that a long time has passed between receiving events.

Supporting long term storage in WFS does require some work on the developer’s part.  The CellMO class specifies two methods that the developer must implement to properly store data to WFS:

  • getServerState() is called by the container to read the current state of your cell as a CellServerState object.  The CellServerState must be serializable using JAXB APIs.  This allows the container to read the current state of your cell and convert it into an XML file.
  • setServerState is called by the container when your cell is restored from WFS.  During a cold start, your cell will be instantiated in its default state, but before the setLive() method is called, setServerState() will be called with a CellServerState object containing the state of the cell.  The CellMO object should read the CellServerState and set its internal state accordingly.

For more information on how to implement these methods, see part 1 of the Wonderland 0.5 dev3 tutorials, about how to build the server classes for a custom cell.

While testing, it is often useful to change the behavior of persistence during a restart.  The restart behavior is controlled by the wonderland.sgs.persistence  variable in the my.run.properties  file.  This variable can be set to one of three values to control the persistence:

  • NONE – this disables all persistence during a restart.  Every time the Darkstar server starts up, it will read the world description from WFS.  This is often useful for debugging, because the world is reset to a known state at every restart.
  • FALLBACK – in this mode, the server automatically detects changes to the Darkstar server code or the selected WFS, and does a cold restart if necessary.  So if there are no changes, the world will be persistent, but if there are any server changes, it will fall back to the currently selected WFS.
  • FULL – in this mode, the server automatically detects changes as in the FALLBACK case.  Before the Darkstar datastore is cleared, the system will automatically take a snapshot of the current world state, and then restart using that snapshot.   In this case, the world should appear completely persistent, even if server code is changed, since the most recent version of the world will be saved to WFS and then restored.

Note that as of the 0.5 dev3 release, the FULL option for persistence is not implemented, and FALLBACK is the default.


SHIFT Radio Appearance

February 13, 2009

I had the pleasure of making a Friday the 13th appearance on Sun Chief Gaming Officer Chris Melissinos’ SHIFT Radio show today:

SHIFT Radio Episode #30: Project Wonderland

As an extra bonus, Kevin Roebuck joined us on-air to talk about Wonderland in the education community.


Thinking About Accessibility

February 6, 2009

I attended a demo today in Saint Paul College’s virtual Wonderland classroom that showed how you can do a loose integration of a speech recognizer and the Wonderland chat pane. Computer Careers instructor Eric Level has a student who is deaf in his introductory Java class, so he decided to try using Dragon NaturallySpeaking to automatically transcribe everything he said in the class. To accomplish this, he brought up Wonderland on a laptop with Dragon installed and set input focus to the chat pane. Everything he said appeared in the chat pane, as shown below.

Speech recognition output in Wonderland chat pane

It’s a feature of the Dragon application that speech recognition results can be output to any application that accepts text input.

While this technique does work, it got me thinking about how one might design a more streamlined solution. One idea is to put the recognizer on the server with the voice bridge so that everyone’s audio can potentially be transcribed, not just the presenter’s. This would not only eliminate the need for a separate computer to devote to the recognizer, but it would also be possible to annotate the recognition stream with the names of the speakers since the voice bridge can determine who is speaking at any given time. The downside to recognition on the server is that the audio may not be as clear as it would be if captured locally on the client. Also, it might not be as easy to train the recognizer on the presenter’s voice.

From the user interface point of view, in 0.5 it might be interesting to build a "subtitle channel" that users could subscribe to. Rather than have the recognition output typed into the chat window which everyone sees, it could be displayed in an optional HUD panel, which could be resized, positioned, or magnified as needed.  This would leave the chat pane available for back-channel communication among in-world participants.

Providing accessibility to Wonderland for users with a range of disabilities is an important area of focus. For anyone who might be interested in doing a project in this area, here are a few open source resources to start with:

Sphinx-4 – open source speech recognizer

eSpeak or FreeTTS - open source speech synthesizers

GNOME Accessibility Project – accessibility solutions for graphical user interfaces


Video in Wonderland using MJPEG and VLC

February 2, 2009

When we created the video player for Wonderland we took what we thought was the most expedient approach by using the Java Media Framework. This turned out to be a real challenge. Not only was JMF no longer being maintained, but a lot of its behavior was bizarre and obscure. So, I was very happy to hear that a team at Ericsson in the Netherlands had built a video player that doesn’t use JMF but uses an open source video library instead. This is just a small part of an incredibly compelling Wonderland project from Ericsson that we’re hoping will be showcased in the "Cool stuff" category at JavaOne later this year.

Here’s a technical overview of the video player by Erik Reitsma, who seems to have a really fun job at Ericsson!


Several people on the mailing list and forum have asked about streaming video in Wonderland. Apparently some people have it working with specific webcams, but generic streaming of video is harder. Since also we liked to stream video from a webcam, we bought an AXIS webcam. Not the 212 PTZ, but a cheaper one, the AXIS 207 . And I could not get it working with the existing video player in Wonderland.

Instead of buying a more expensive webcam, I looked into what was really happening. The current video player uses JMF to decode the incoming video stream, and takes snapshots of the video. These snapshots are then drawn on a surface in the 3D world. So probably there was something going wrong in the JMF part. Since JMF is no longer developed or supported, I started looking for alternatives.

Our AXIS 207 can stream its video in two formats: MPEG-4 and MJPEG (http://en.wikipedia.org/wiki/Motion_JPEG). I would not think of writing an own MPEG-4 decoder in Java. However, MJPEG is basically a multipart stream, containing one JPEG image after another, plus some headers and boundaries. This is quite easy to decode, and it is not hard to put the JPEGs as a texture in a 3D surface.

So what I did was adapt the existing JMF video application to get images from a (pure Java) MJPEG decoder and render the received JPEG images. I used the MJPEG decoder from Cambozola (http://www.charliemouse.com:8080/code/cambozola/) for this, but it would not be too hard to write one from scratch. This simple solution works surprisingly well. The only remaining issue was, that we could only play MJPEG streams, and then even only MJPEG streams that were accessible through HTTP. Now that I had this working video player for our webcam, I wanted to use it to play other videos too.

Fortunately there exists an open source video player that can do just what I was looking for: VLC media player (http://www.videolan.org/vlc/). VLC media player is a highly portable multimedia player for various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, mp3, ogg, …) as well as DVDs, VCDs, and various streaming protocols. But it can also be used as a server to stream audio and video, in many different formats. One of these formats is MJPEG over HTTP, just what I needed for my video application. This means that I can stream any video that VLC can play, as an MJPEG stream, and play it in my video application in Wonderland.

I use the following command to serve a movie xyz.mpg as a stream on port 8050 as (accessible as http://localhost:8050/abc.mjpg):

vlc xyz.mpg –no-sout-audio –sout=’#transcode{vcodec=mjpg,vb=512,fps=5,width=640,height=480}:standard{access=http,mux=mpjpeg,dst=:8050/abc.mjpg}’ –input-repeat=10000

Parameters such as width, height and framerate can be varied. I repeat the movie 10000 times, so that it runs in a virtually endless loop.

A disadvantage of this approach is that the video is streamed using a quite inefficient codec. I have considered running VLC on the same host as the Wonderland client, and controlling VLC from the Wonderland client. However, the current solution is good enough for my purposes.

Here’s a nice mixed reality example which shows the video player in action:

Video player

Erik Reitsma, Ericsson, The Netherlands


Follow

Get every new post delivered to your Inbox.

Join 54 other followers

%d bloggers like this: