US military turns to Halcyon branch of OpenSim

Editor: The team behind U.S. Army’s MOSES grid have long been cutting-edge contributors to OpenSim, building a grid management system, running scalability tests, and donating content to the OpenSim community. Now they are in the process of moving to a branch of OpenSim called Halcyon, open-sourced last year by InWorldz, OpenSim’s most popular grid.

(Image courtesy Douglas Maxwell.)

(Image courtesy Douglas Maxwell.)

The MOSES project and the OpenSimulator project have diverging goals. Specifically, we place lesser value on backwards compatibility and for us security is a high priority.  I believe the legacy code is holding OpenSim back and slowing true innovation.

Last year, we dissected and profiled the OpenSimulator code and found five major areas for improvement.  These areas include physics, database management, client management, conflicting TCP/UDP network stack, and script engine operations.

Also, there was an attempt to multi-thread pieces of the simulator, but it was done in a way that did more harm than good.  In order for us to correctly assess the behavior of the simulator, the first thing we fixed was the broken frame rate counters and network performance reporting.

We had a lot of trouble getting patches approved and integrated into the OpenSimulator code base last summer, with many of the objections and comments being that the updates might have unexpected effects on existing users.

The engineering team at InWorldz had been working on their Halcyon code base during the same time period.  They saw our posts on OpenSim’s developer channels and contacted me directly.  They had arrived at many of the same conclusions we did and identified many of the same areas of improvement.  They independently worked on scripting engine improvements and database management.

I was particularly impressed with the way they used the Cassandra database.  They also did early physics work with PhysX for Windows.  We focused our initial efforts on a Linux based PhysX integration and networking.

An actual environment, on the left, compared to its virtual representation on the right. (Image courtesy Douglas Maxwell.)

An actual environment, on the left, compared to its virtual representation on the right. (Image courtesy Douglas Maxwell.)

In a lot of ways, the Halcyon developers found and fixed significant deficiencies that persist in the mainline OpenSim code even today.  Even though the InWorldz team forked from mainline OpenSim a long time ago, their current Halcyon code is quite advanced and it is no surprise to me why their grid is stable and successful.

The MOSES team spent a considerable amount of time examining the Halcyon code base and we compared its differences to the current OpenSimulator.

As it turns out, our MOSES Grid Management tools are a bit more functional than theirs, so we integrated them into Halcyon.  We have MOSES-Halcyon hybrids running on our servers at the moment and we will migrate after we finish updating the Halcyon code to accept IAR files — inventory export files.

Over the past year, we’ve had a number of engineering exchanges with the Halcyon developers.  My team and I have been to San Antonio for meetings with them and they have come to Orlando for technical exchanges at our labs.  The two teams have integrated well and our cultures are in much better alignment than trying to deal with the melee of the OpenSimulator core.  The two teams have a scheduled on-site meeting in Orlando again during the week of August 22.

Where do we go from here?  In short, we are following a professional systems engineering approach to planning a modernized OpenSimulator code base.  OpenSimulator is woefully insecure.  The first major co-development effort is to change authentication and begin using web tokens with encryption.

We are also in the planning stages of a detailed design document for the HTML5 client, which will be a browser-based viewer for OpenSim.

Near term, we will pull the entire networking stack out of the simulator.  The simulator will become strictly a compute process.  We are currently working on a network data arbiter that is a glorified firewall that will be a stateless and lightweight web translation layer between the HTML5 clients and the simulator.  This will use Google’s protocol buffers or something similar and allow us to efficiently scale on the simulator side.

This will also be a port reduction strategy and reduce the network footprint of the larger grids.  Any plans for a hypergrid-like functionality that allows teleports between separate grids will need to be completely re-engineered.

It is true that the OpenSimulator code is around a million lines of code, but some of it is fat that should be trimmed.  Unnecessary appendages will be cut or replaced with modern equivalents.  Since we are looking to create a truly distributed simulation engine, any data that is currently transmitted using clear text will be replaced with serialized formats.

Mesh vehicle convoy on the MOSES grid. (Image courtesy Douglas Maxwell.)

Mesh vehicle convoy on the MOSES grid. (Image courtesy Douglas Maxwell.)

Lastly, OAR — region export files — and IAR functionality will be preserved for the foreseeable future.  They are convenient and well done.  We would also like to include support for static meshes in the HTML5 client, so watch out for announcements on that front.  Some adjustments will also need to be made to accommodate the new physics functionality, such as preservation of new types of linked objects.

We will be pushing regular updates to our public Github site, so anyone is able to pitch in and help with development.  We’ve already had a number of people contact me with intense interest in the HTML5 work.'

Douglas Maxwell

Douglas Maxwell is the science and technology manager for virtual world strategic applications at the U.S. Army's Simulation & Training Technology Center.

  • Nick Zwart

    Thanks for the info Douglas, very interesting, also for education!

  • Talla Adam

    This is a very welcomed update on MOSES, team’s progress with their branch of Open Simulator. We had been discussing this briefly at G+ Opensim Virtual recently asking if anyone had heard news on the project. Some of the user community have been talking about how we might help and have our views considered. New Media Arts (NMA) are also interested in stimulating the development of OpenSimulator software and in an article at there is more discussion.

    This article has also been linked to

  • Selby Evans

    New Media Arts is including this information in its discussion of ways to stimulate OpenSim Development.

  • Guest

    This is simply awesome and long overdue. Finally somebody is prepared to move this project past the working prototype level. Just in time to catch all the soon to be homeless second lifers.

  • WhiteStar Magic

    Very interesting and a positive move in the right direction finally for OpenSim which has been hamstrung and held back. Virtual World tech + VR have always been headed towards convergence and in a manner that the average Jane & Joe Public would be able to use it with ease. The continued regressive holding onto what “was” rather than looking at what can be and what is to come has been the greatest failing with OpenSim.
    Sadly it appears that while Halcyon is stepping up, it may now be too late for the general OpenSim platform to breach the future market. It is ironic that so little has changed over the past couple of years given that neon writing on the walls went largely ignored by the clique’s in charge.