In the real world, we tend to take persistence for granted. If I move a book into my office, I expect it will still be on my desk the next time I go looking for it. In Wonderland 0.4, however, this type of world persistence doesn’t exist: whenever you restart the Wonderland server, it resets the world to its original state. This can be quite frustrating for spaces like the team rooms shown below. We would really like the team room to persist as users add documents, request comments, and generally live in the space. The room itself should evolve with and reflect the current state of the team that works there, as happens in real-world team rooms.
In Wonderland 0.5, we have added several new features to help maintain world persistence. From the user’s perspective, this means that Wonderland worlds will act more like the real world, which is usually a good thing. But under the covers, there are some complexities that Wonderland developers and administrators will need to pay attention to. The rest of this post will talk about the mechanics of world persistence in Wonderland 0.5.
Mechanisms for persistence
In Wonderland, we support two general mechanisms for persistence:
- For short term persistence, Wonderland uses the Darkstar datastore. As you change the world, the state is continuously written to the Darkstar data store as serialized objects. This happens as part of Darkstar’s built-in mechanisms for data integrity, so there is no special work on Wonderland’s part for this type of persistence. Since Darkstar is event driven, whenever an event occurs, Darkstar retrieves the relevant objects from its datastore, and then stores the updated version when the event completes. The catch is that, due to all this object serialization, data in the Darkstar datastore becomes difficult to manage if the server code changes. Thus the Darkstar datastore is appropriate for saving the world state through server restarts, but not across different versions of the server or as code changes. The Darkstar datastore is also an opaque storage mechanism — only live Darkstar applications can access it, so it cannot be used to share worlds.
- For long term persistence, Wonderland uses an XML serialization mechanism we call WFS (which originally stood for "Wonderland File System", an acronym that may be a bit out of date at this point). WFS describes a Wonderland world as a set of XML files, arranged in a tree structure just like the Wonderland cell hierarchy. Each file describes the properties of a particular cell in the Wonderland world. WFS is managed via a web service in the Wonderland web server. Currently, writing WFS requires the world to be in a quiesced state with no users logged in, and typically requires a server restart. Once it is written, however, WFS is independent of any Java code, so can be used to persist worlds when the server code changes. It also doubles as a interchange format, so worlds can be modified and shared as a tree of WFS text files.
Given these two overlapping persistence mechanisms, we had to determine the rules for when each is used. Typically, our goal is to use the short-term Darkstar persistence whenever possible. So when the world is running, we use Darkstar persistence exclusively. The good news for us is that this mostly happens automatically: Darkstar persists state by default, so unless we explicitly remove it, the world will remain up-to-date. The decision point comes during a server restart. We need to decide what type of persistence to use based on whether the code or the desired world has changed:
- A warm restart happens when there is an existing Darkstar database, and this database is up-to-date with both the world that is selected and the latest Wonderland code. In the case of a warm start, the server reads the current state from the Darkstar datastore on startup. For most Wonderland server code, this is completely transparent: objects in Darkstar persistence come back as if the server had never been stopped.
- A cold restart happens when the Darkstar database is out-of-date with the Wonderland code, there is no existing database, or the administrator has chosen to load a fresh world. In this case, the state of the world is read at startup from the web server via WFS. This is a fairly heavy-weight mechanism, and involves each cell getting set up individually based on the state in WFS. The world that is read in might reset the server back to some well-known previous state, or it may restore a snapshot of the state from just before the cold start.
The administrator’s window into this restart behavior is the Wonderland snapshot manager, pictured below. When you open the snapshot manager, you will see a list of initial world images and snapshots of existing worlds. You can choose which image to load by selecting "make current" or you can restore a world back to the state stored on disk by clicking "restore." Both of these operations will force a cold restart of the server, so the Darkstar datastore will be replaced with state stored in the selected XML files.
In addition to managing the current world image, you can also create a new snapshot. Snapshots store the current state of the world in a WFS directory, but do not change the Darkstar datastore. You can select "restore" to reload the world from one of these snapshots.
WFS files, both from initial worlds and snapshots, are stored in the Wonderland run directory in the "wfs" subdirectory. On my system, that is in /Users/jkaplan/.wonderland-server/0.5-dev/wfs.
Finally, it is important to note that when you install a new module that contains server code, the system will also force a cold restart. The desired behavior is that the system will automatically create a snapshot of the world, install the new server code, and the restart from the snapshot. This should bring the world back to a state that is very similar to what it was prior to the restart. As of Wonderland 0.5 dev3, this behavior is not fully implemented, so you must manually create a snapshot before installing modules if you want
to preserve your world state.
Developing for persistence
For Wonderland developers, there should be no additional work needed to support the short-term Darkstar persistence. The normal Darkstar rules about using ManagedObject and Serializable will ensure that your server cells are persisted properly in the Darkstar datastore. Since Darkstar is event driven, a restart will be transparent to your objects: the object is not notified in any way of the restart, although you may notice that a long time has passed between receiving events.
Supporting long term storage in WFS does require some work on the developer’s part. The CellMO class specifies two methods that the developer must implement to properly store data to WFS:
- getServerState() is called by the container to read the current state of your cell as a CellServerState object. The CellServerState must be serializable using JAXB APIs. This allows the container to read the current state of your cell and convert it into an XML file.
- setServerState is called by the container when your cell is restored from WFS. During a cold start, your cell will be instantiated in its default state, but before the setLive() method is called, setServerState() will be called with a CellServerState object containing the state of the cell. The CellMO object should read the CellServerState and set its internal state accordingly.
For more information on how to implement these methods, see part 1 of the Wonderland 0.5 dev3 tutorials, about how to build the server classes for a custom cell.
While testing, it is often useful to change the behavior of persistence during a restart. The restart behavior is controlled by the wonderland.sgs.persistence variable in the my.run.properties file. This variable can be set to one of three values to control the persistence:
- NONE – this disables all persistence during a restart. Every time the Darkstar server starts up, it will read the world description from WFS. This is often useful for debugging, because the world is reset to a known state at every restart.
- FALLBACK – in this mode, the server automatically detects changes to the Darkstar server code or the selected WFS, and does a cold restart if necessary. So if there are no changes, the world will be persistent, but if there are any server changes, it will fall back to the currently selected WFS.
- FULL – in this mode, the server automatically detects changes as in the FALLBACK case. Before the Darkstar datastore is cleared, the system will automatically take a snapshot of the current world state, and then restart using that snapshot. In this case, the world should appear completely persistent, even if server code is changed, since the most recent version of the world will be saved to WFS and then restored.
Note that as of the 0.5 dev3 release, the FULL option for persistence is not implemented, and FALLBACK is the default.