dot Wonderland: Redux

Way back in v0.4, I wrote a blog, dot Wonderland, that described the contents of the "dot" directory that held meta-information, configuration information, and cached assets for Project Wonderland. It was meant to be an under-the-covers peak into some operational details for those who were interested and users, who like me, just love to twiddle with files every once in a while. I’ve been meaning to update that blog post for v0.5, so here we go.

In that earlier blog, I posted an important disclaimer that is worth repeating: modify the .wonderland and .wonderland-server directories at your own risk! Since the information contained in these directories are implementation details and not part of any public API, anything posted here is subject to change without notice.

The .wonderland-server/ directory

In Version 0.5, the files associated with the Project Wonderland server and client are now kept in two separate directories: .wonderland-server/ and .wonderland/, respectively. The structure of the .wonderland/ directory has not changed much since v0.4, so the information described by the dot Wonderland blog is still mostly correct. There are some changes, and I’ll talk about those in another blog.

In this blog, I’d like to focus on the files pertaining to the server, in the .wonderland-server/ directory. First off, Project Wonderland keeps files for each revision of v0.5 in separate directories. If you are working with the main line source code (aka the ‘trunk‘), then files are stored beneath the 0.5-dev/ directory. If you are working with 0.5 User Preview 2, then files are stored beneath the 0.5-preview2/ directory. Once v0.5 is finalized, I’d imagine the files would be stored under the 0.5/ directory.

Keeping the files for each of these sub-versions separate really helps development: we can run and debug each of these different subversions on our development machines at the same time. Users who upgrade from, say 0.5 User Preview 1 to 0.5 User Preview 2, need to manually copy some files if they wish to keep state across these sub-versions. Hopefully the details contained in this blog will make this process a bit easier.

Log files, runtime state, worlds

If you take a look at the ~/.wonderland-server/0.5-dev directory (here, ‘~’ denotes the user’s home directory), you’ll see three subdirectories: log/, run/, and wfs/.

Log Files

The log/ directory is perhaps the easiest to explain, so let’s start there. The directory is only one level deep and contains a bunch of text files (with the .log file extension) — they are the log files for various Project Wonderland server components. The primary one is the web server log, the latest of which is web_server.log.0 (older versions from previous times the Project Wonderland server was run can be found in web_server.log.1, web_server.log.2, etc).

There are three other log files that always appear in this directory: one for the Darkstar server (darkstar_server.log), one for the Voice Bridge (voice_bridge.log), and one for the Shared Application Server (shared_application_server.log). You may also view these web files via the Web-based Administration UI.

A fourth log file (darkstar_server_snapshot.log) is only created if you take snapshots of your world (see WFS: The Wonderland File System in v0.5 for more details).

Worlds and Snapshots

The wfs/ directory contains all of the initial worlds and world snapshots. Each is stored as a collection of XML-formatted files. The structure of the way these XML files are stored on disk is defined by the Wonderland File System (WFS). WFS has evolved in v0.5 since earlier releases, and the WFS: Wonderland File System in v0.5 blog succinctly describes the contents of the wfs/ directory.

Not mentioned in the blog, however, is the recordings/ directory that you may see underneath the wfs/ directory. The recordings/ directory contains the state stored by the Event Recorder, an experimental module that performs 3D recordings inside the virtual world.

The run/ directory

I left the run/ directory as last to describe for a reason: it’s fairly complicated. This directory contains all of the installation information for the Project Wonderland server (including its various server parts, including the Darkstar server, Voice Bridge, and Shared Application Server). It also contains the "live" state of the virtual world (in the form of BerkeleyDB, an embedded database), an up-to-the-minute persistence mechanism that is implemented by the Darkstar middleware layer. The run/ directory also contains the content area for the embedded WebDav server that is bundled with Project Wonderland.

Here’s the directory listing that I current see in ~/.wonderland-server/0.5-dev/run:

content/ docRoot/ webserver/ darkstar_server/
shared_application_server/ wonderland.version deploy/ voice_bridge/
wonderlanddb/ derby.log webdeploy/

Perhaps it’s best to start off by describing which directories you can ignore for now: docRoot/, webserver/, darkstar_server/, shared_application_server/, deploy/, voice_bridge/, wonderlanddb/, and webdeploy/. These directories either contain information regarding the implementation of Project Wonderland’s modular server architecture (i.e. deploy/ and webdeploy/), it’s embedded web and application server (i.e. doctRoot/ and webserver/), or one of the server components that help implement the virtual world (i.e. darkstar_server/, shared_application_server/, voice_bridge/, and wonderlanddb/).

This leaves the following directory: content/. This directory serves as the content root for the WebDav repository. (WebDav, if you are not familiar, is simply an extension to HTTP, and let’s us manage content on the web server remotely by clients). Some of the content stored in this directory is visible to ordinary users via the Project Wonderland client: the Content Browser (Tools -> Content Browser) displays all of the content found beneath the
system/ and users/ directories here. For example, if a user drags-and-drops a model or an image or a PDF file into the Project Wonderland client, it gets uploaded to his/her space beneath the users/ directory.

The content/ directory also contains other information that is not visible to ordinary users, but is nonetheless important. It is information that is needed by the client software, so placing it in the WebDav content area was natural and eased its distribution. In fact, all modules installed into Project Wonderland are stored beneath the modules/ directory, and more specifically, the modules/installed/ directory. Feel free to browse the modules/installed/ directory for a bit: it’s layout is relatively simple. You’ll see a directory per module and the contents of the module JAR file expanded beneath that directory, along with some XML-formatted text files that describe the module. There is no other hidden registry of installed modules: its existence in the modules/installed/ directory is all that is necessary. (Incidentally, if you are ever in a position where you’d like to manually remove all of the installed modules in the system, simple clear out the modules/ directory and restart the server. On the flip side, to install a module, it’s best to go through the normal mechanisms, e.g. the Web Administration UI).

The final directory beneath content/, the checksums/ directory, contains checksums computed for various assets in the system. These checksums are computed when a module is installed and is used by the caching mechanism in the Project Wonderland client to determine when it needs to download a new asset, for example.

Migrating Important Data from One Version to the Next

If you’ve built a world, in say 0.5 User Preview 2 and want to migrate the state of that world to, say the latest trunk, then the information above should provide you with the means. By and large, it’s a simple matter of copying the proper files from one directory to another. I’ve tried this out myself by building a simple world in 0.5 User Preview 2 and then migrating the files so that I have the same world if I want to use the most recent Project Wonderland codebase (the trunk).

Using 0.5 Preview 2, I first installed the Clickable Link module (from the unstable/ section in the wonderland-modules workspace) [1] and created a simple world in 0.5 User Preview 2. I dragged-and-dropped a PDF file, which created a PDF Viewer in-world. I added a Sticky Note. I imported a 3D (.kmz) model using the Insert -> Model menu item and uploaded it to the server, placing it in a module named SatelliteModel. I then took a snapshot of the world (to preserve its state) and exited the client and killed the server.

Next, I ran the server using the trunk codebase. This populated the ~/.wonderland-server/0.5-dev/ directory. I then installed the Clickable Link module [1]. (I’m just using Clickable Link as an example; you will need to re-install any modules you had installed manually when using 0.5 User Preview 2). I then killed the server for the time being.

As a next step, I manually copied over key directories from the 0.5-preview2/ directory into the 0.5-dev/ directory. Here’s what I did:

  1. Copy the SatelliteModel/ directory in the ~/.wonderland-server/0.5-preview2/run/content/modules/installed/ directory to the ~/.wonderland-server/0.5-dev/run/content/modules/installed/ directory. Since I imported the model via the Insert -> Model menu item, it packaged the model into a module and installed it on my server. This first step copied that module from my old installation to my new installation. (You should do this for any model you installed this way). Related to this is Step 2. Note that models that have been dragged-and-dropped into the world are uploaded to WebDav, and migrated in Step 3 below.
  2. Copy the SatelliteModel/ directory in the ~/.wonderland-server/0.5-preview2/run/content/checksums/modules/ directory to the ~/.wonderland-server/0.5-dev/run/content/checksums/modules/ directory. When modules are installed, the system computes checksums for certain of their assets. Since I manually copied the module from the previous installation (Step 1 above), the checksum-generating step did not occur for my new installation. Manually copying the old checksum computation should be sufficient.
  3. Copy the users/ directory in the ~/.wonderland-server/0.5-preview2/run/content/ directory to the ~/.wonderland-server/0.5-dev/run/content/ directory. This copies over all of the content that was uploaded to the server when I dragged-and-dropped content into the world. (If you placed content in the system/ directory, you may wish to copy that over too). Update: Actually, if you’ve created system-wide Placemarks via the Web Admin UI, you will need to copy the system/ directory for sure.
  4. Copy the snapshots/ directory in the ~/.wonderland-server/0.5-preview2/wfs/ directory to the ~/.wonderland-server/0.5-dev/wfs/ directory. This copies over the snapshot I took after I created my world in 0.5 User Preview 2.

After these four steps, I restarted the server running the latest Project Wonderland codebase and restored to the snapshot I took in 0.5 User Preview 2 that I copied over to my new installation. After I restarted the client, I saw the world as I left it in the previous version of the software.

While these manual steps should not take too long to do, a migration tool to perform the migration automatically would be a big help. If you are interested in writing such a tool, I think it would make an excellent contribution from the community.

[1] When I wrote this blog, the Clickable Link module was kept in the unstable/ section of the wonderland-modules workspace. It has since been promoted to the stable/ section and is now bundled with Project Wonderland.  


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: