Back when 3rd Rock Grid first launched in the spring of 2008, it was one of the earliest OpenSim grids and the owners had to figure out everything on their own.
The grid’s managers had to learn how to install the software and maintain it, manage upgrades, backups, modules, land, security, assets and users, and handle a million other technical tasks that go into running a grid.
It doesn’t help that most grid functions have to be handled manually, via a text-only command-driven interface that’s a throwback to the bad old DOS days. Automating anything – such as having a region restart automatically if it crashes – means that someone has to write new code.
“It was a real nightmare,” grid owner Jim St. Clair told Hypergrid Business.
These days, the grid’s management team is sleeping a lot more peacefully, because the grid recently switched over to a grid management tool from UK-based PioneerX Estates, called PXE RCI.
Every five minutes, RCI checks whether every single region on the grid is working properly, and restarts it if it isn’t. RCI also handles user registrations, automates estate and land management, takes care of security, runs backups, manages the currency, and takes care of all the technical management jobs that go into running a grid.
It also replaces the old text-based command-driven interface with an easy point-and-click one that requires no programming skills or training to use. According to PioneerX owner Simon Gutteridge, grid owners can turn hypergrid connectivity on or off with a click, for example. They can also quickly add new regions or move regions around.
“Between the newer versions of OpenSim software and PioneerX products, we have reduced the amount of grid support necessary to keep our grids up and running,” St. Clair said. “Both improvements have been highly beneficial to our grid’s operation.”
(Last year, as part of a move from OpenSim 0.6.9 to 0.7, the OpenSim core developers rebuilt much of the underlying architecture of OpenSim to make it more stable and scalable and, with the exception of InWorldz and ReactionGrid and their clients, most public and commercial grids have switched over to OpenSim 0.7, which also supports a more secure version of the hypergrid protocol, Second Life Viewer 2, and media-on-a-prim.)
St. Clair not only runs 3rd Rock Grid on RCI but also the new, education-focused Curiosity Grid.
“I would recommend Simon and PioneerX to anyone who is wanting to operate an OpenSim grid,” St. Clair said. “We have now worked with Simon for a number of months, on both of our grid efforts, and have found him to be one of the best providers of virtual services we have had the pleasure to work with.”
Who needs grid management?
There are basically three kinds of OpenSim grids.
First, there’s the one-region standalone, floating out in hyperspace all by itself. A standalone can be run on a single computer, USB stick, or with a hosting provider (prices start at just $9.95 a region – full list of hosting providers here).
There’s not a lot of management that’s involved in running a standalone, though you may want the ability to make a backup of the entire region – in the form of an OAR file – or make a backup of an entire user inventory, also known as an IAR file. Many hosting providers will allow you to do this, and some have self-serve management panels to make the process easier.
A step up in complexity is the mini-grid. This is a grid typically composed of four to sixteen regions, often organized into a single no-border-crossings megaregion, which runs on a single computer, server, or USB stick. The most popular way to do this is with the Diva Distro, which is not too difficult to install on your own if you’re capable of figuring out how to download and configure MySQL, and know how to set up port forwarding on your router. If you’d rather not mess with the MySQL and the port forwarding, many OpenSim hosting providers will set up a mini-grid for you at their standard per-region cost. Some will even throw in a discount for ordering more than one region at once.
In general, standalones and mini-grids tend to be used by just one person or a small group of people for their own creative building, to hold their virtual homes or offices, to showcase virtual portfolios, or to hold educational programs. These are typically private grids, where strangers aren’t going to want to get their own avatars or buy regions. If standalones or mini-grids want outsiders to come in, they’ll typically turn on hypergrid and let people teleport in, without registering and creating new avatars.
But if you want to have more regions than can easily fit on one computer, or expect your grid to quickly grow to that point, then you want to have a full OpenSim grid, with a separate server for asset management and user inventories and other grid-wide management tasks.
This is typically what happens when folks want to run a big commercial grid, where they rent regions and estates to their own customers. But some educational institutions and businesses might also need grids this big, especially if they are used by multiple classes or business units.
This is where OpenSim gets really complicated (if the MySQL and the port forwarding wasn’t complicated enough for you already).
With more regions comes more administrative headaches, as folks want their regions moved around or restarted. You want regular automatic backups – it’s too much work to back up every region by hand. You might also want voice, groups, search, or a local currency. You might want to turn hypergrid on or off for particular regions, or grid-wide. You might need to delete assets from your grid because they infringe on copyright, and ban users for violating the grid’s terms of service. And you want new users to be able to choose from a selection of starting avatars, instead of having them all start out as Ruth, with her ugly orange tights and bad hair. Your users will also want to be able to rent land, or to rent land to others, to see their transactions, and reset their passwords.
Handling all these jobs manually is a herculean task.
PioneerX takes the lead
So far, of all the grid management suites we’ve seen, the RCI product from PioneerX is the most advanced, particularly for customers that want to run public, commercial grids with an in-world economy and private land businesses.
It is currently used by five grids, including the 180-region 3rd Rock Grid, the 55-region Sim-World grid, and the seven-region Curiosity Grid.
Grid owners who use the RCI grid management can also choose to use PioneerX to host the individual regions, or they can host the regions themselves or even with another hosting provider, Gutteridge told Hypergrid Business.
“Some of the grids, we’re holding 100 percent of the grid – hardware and management,” he said. “Some of the grids, we’re just holding part of the grid – the management, and some grid services.”
For example, PioneerX provides the management suite for the Sim-World grid, but hosts just half of the regions, he said.
With 3rd Rock Grid and Curiosity Grid, PioneerX provides just the management – 3rd Rock has its own servers for its regions.
Whether the individual regions are actually hosted by PioneerX or not, however, its grid management services will monitor the regions’ performance, automatically restart crashed regions, and take care of all the other management tasks as if those regions were hosted by PioneerX itself.
PioneerX also offers individual region hosting, for those not interested in running entire grids, either as standalones or connected to open grids like OSGrid.
Born on OSGrid
PioneerX originally started out, like many other hosting companies, by renting out regions attached to the non-profit OSGrid, which allows anyone to connect their regions.
Gutteridge wrote software to help himself manage all these regions.
“What we discovered was that the management software that we wrote in order to manage the business we had in OSGrid was desirable,” he said. “People wanted it. So we continued to expand that software suite into a full management package.”
Hosting starts at US $41 per month (25 UK pounds) for up to 15,000 prims, and $65 (40 pounds) for up to 45,000 prims. Discounts are available for customers who rent multiple regions at once. Each region can normally support 40 simultaneous avatars.
Management services for individual regions are included at no extra cost when the regions are hosted by PioneerX, Gutteridge added.
These regions can exist as standalones, or mini-grids, or can be attached to public grids like OSGrid without any central grid services required.
However, if the customer wants to create a full grid for themselves, they will need separate grid management servers to handle the grid’s assets, inventories, and other services.
PioneerX charges $41 per month (25 pounds) for basic grid services, with no currency support, or with outside currency such as the OMC currency from Virwox. For grids that want to have their own in-grid currencies, the price for the grid management is $74 per month (45 pounds).
The price includes unlimited user accounts and user inventories, Gutteridge said, though users with particularly large grids may want to add a separate asset server for $41 per month (25 pounds) extra, as well as the RCI management suite.
“We run a very large enterprise SAN [storage area network] that we store everything on,” said Gutteridge. “We don’t have storage issues.”
The benefit to having a separate server just for assets is that it speeds up operations if a grid gets a lot of traffic, he said. “As it loads up, it has no effect on anything else.”
Customers who get grid services from PioneerX will also get a free website where their users can create new accounts, reset their passwords, and manage their own land holdings. They also get one free domain name to go with the site.
Would-be grid operators that already have their website up and running will need to get PioneerX to integrate their web front end with the RCI Web system, which could take a little longer. For example, the Sim-World website is created by Sim-World itself, but once users log in they see embedded functionality provided by PioneerX. This includes a rental manager that allows tenants to quickly see if their rent is due – and to pay it if it is, and a wallet manager that shows their in-world transactions.
In the clouds
For the region and asset hosting, PioneerX runs a private cloud that allows the company to quickly add virtual servers and shift work from one machine to another when needed.
One benefit is that the company can allocate additional resources when needed. For example, regions typically support up to 40 avatars at a time. But if a company is, say, holding a big event then – with advance notice – PioneerX can add power to support up to 100 simultaneous visitors.
“We invested significant money in the private cloud,” he said. In addition, PioneerX can also interface with public clouds like the Amazon EC2 cloud, or with outside server farms and hosting providers.
“The management system will interface with any server technology,” he said.
Gutteridge recommends that customers not try to host commercial regions on their home computers, however.
“The bandwidth available to most homes is not good enough,” he said. “It’s okay for standalones and mini-grids, but not generally good enough for commercial grids.”
Down with the console
The big advantage of the RCI management panel is that it completely eliminates the need to access the OpenSim server console directly.
Say, for example, you were running OpenSim at home on your own computer and needed to create a new region. You would type the “create region” command in the OpenSim console and a new region instantly appears.
With the RCI panel, you simply click on the “create region” button – a green square in the server control panel and the new region appears when the server is next restarted.
Why wait for a server restart instead of turning the region on instantly?
“The reason we do it that way is because the commands ‘create region’ and ‘delete region’ are unreliable,” Gutteridge said. “They have a habit of causing server crashes. Our entire system is designed to be as resilient as possible with the current OpenSim technology.”
Now for something harder. Say, you wanted to enabled hypergrid teleports in and out of your grid. You would need to go into your console, and shut down your regions. Then you would need to open up the .ini file for your OpenSim and manually edit the settings to allow hypergrid, and then restart your instance of OpenSim. With the RCI console, turning on hypergrid is a simple check box, and the hypergrid change goes into effect automatically when the server is next restarted.
Setting up megaregions is normally harder still. Megaregions are useful for eliminating border crossings on a square composed of four or nine regions – nice for events or shopping areas or residential regions, though OAR backups do become tricky. With the OpenSim console, you would need to shut down your OpenSim regions, edit the .ini configuration file, edit your Regions.ini configuration file, then restart the server again. With RCI, it’s just a single checkbox. (Warning: to avoid loss of data, only empty regions should be combined into a megaregion.)
Setting up a brand new OpenSim server is normally an extremely lengthy process. You would need to download and install the OpenSim software. Install Mono, if you’re running Linux. Install and configure a MySQL database. Route the appropriate ports through your router. Then create individual regions and load your terrain files.
With the RCI panel, you click on a button, select your server options – Do you want hypergrid? Do you want megaregions? How many regions do you want to have on this server? – then sit back and let RCI do the rest. It normally takes about 20 minutes to set up a new server and have it up and running, said Gutteridge. In RCI a “server” is actually a virtual server, not a physical machine, running a single instance of OpenSim.
Connecting a server that’s running with another provider requires a little more work, however, since PioneerX will need access to the remote machine to install and manage the OpenSim region software that will run on it. But once the outside server is connected, all the management functions will be automatically in place just as if the server was located with PioneerX itself.
Upgrading OpenSim is normally even more labor intensive, as the existing database needs to be backed up, the new software downloaded, and the database migrated to the new version of the software.
“That’s one of the big advantages that the grids that run this do enjoy,” Gutteridge said. “If they wish to upgrade, they don’t have to go and visit every single instance and change every server. They can choose what revision they want and push it to all the servers — and the RCI will go ahead and do that.”
Some hosting providers handle all the upgrades manually. If they have hundreds of clients, the upgrades can take months. Literally, months.
RCI removes the manual labor from the entire process.
“It’s built to be point-and-click for what would normally be complex OpenSim operations,” Gutteridge said. “The control system is designed to be 100 percent hands-off from the console.” The end result is that grid administrators no longer need the technical know-how to edit configuration files, or to access control consoles for regions hosted on remote servers.
Other functions that can be handled by the RCI console include uploading terrain files and creating link regions for easier hypergrid teleports.
Meanwhile, some services – such as automatically restarting crashed regions, rentals management or currency management – aren’t even part of the standard OpenSim distribution.
Rents and payments
When it comes to rental management in particular, RCI allows a would-be grid owner to create a virtual clone of Second Life. Except that the new grid owners would be, say, more considerate of their users, or focused on a particular niche, or simply less expensive than Second Life.
“You can define what you sell or rent and the system will manage it for you,” Gutteridge explained. Grid owners can rent land directly to users, or to developers who create large estates composed of multiple regions that they subdivide and resell.
There are built-in payment systems where end users can pay their rent with in-world currencies. Grid owners can also set up direct PayPal subscriptions for those who rent entire regions.
Rentals can be set up by either the grid owners themselves, or by individual users of the grid.
“They can rent anything they like,” Gutteridge said. Users can subdivide regions and rent them to other users, or rent shops in a mall, or apartments in a building – even club memberships. The system will also interface with in-world tier boxes, used to pay rent in-world.
Users who rent parcels or shops can see the rentals when they log into the website, can cancel their subscriptions, or can make payments without going in-world.
Meanwhile, the in-world tier boxes are designed with resilience in mind, Gutteridge said.
“The tier boxes don’t contain any data,” he said. “If the sim crashes, your rental information isn’t lost. The rental information is only held in the RCI, and the tier boxes call into the RCI on a 30-minute basis to get the latest rental data.”
Once land is rented, RCI can automatically invite the new tenants to a group that would allow them permission to access their land.
Grid owners that want to have an in-world economy have two options with the PioneerX system.
They can create their own currency, which allows them to have an extra revenue stream on top of region rentals – they can sell virtual money. The downside is that they become responsible for the virtual money and for managing the grid’s money supply. There may also be legal liabilities associated with managing a virtual currency, depending on the jurisdiction where they operate.
There is an additional charge to have PioneerX set up an in-world currency system. The price of grid management services is normally $41 per month. Adding currency increases the price to $65 a month.
Grids that opt for an in-world currency will see additional management options on their RCI panel, and their users will see a “Wallet” option when they log in. Users will be able to buy the grid currency via PayPal or redeem it, and see their transaction history. They can also download their statements — a useful option for merchants who do in-world business and want to keep track of transactions.
The other option is to use a third-party currency such as the OMC. There is no additional cost to use OMC — customers pay just the basic $41 per month for grid management.
The OMC currency from Virwox is currently used by more than two dozen different grids. The OMC is hypergrid-enabled, allowing people to buy goods in other grids and bring them home with them, and allows on-grid merchants to sell to visitors from other grids. The OMC currency also works with closed grids, where hypergrid is disabled.
Gutteridge helped with the development testing of the OMC currency module and knows it well, he said. “We can get it up instantly.”
Virwox allows users to see their own transaction histories, and handles the currency exchange functions by logging into the Virwox website. This is an extra step for grid users, and some may find it too inconvenient. However, it also reduces the management load on the grid itself, as well as potential legal liability. VirWoX is one of the leading European exchanges for the Linden dollar. Virwox traded the equivalent of over $13 million in Linden dollars last year, an increase of almost 82 percent from 2009.
For voice, PioneerX currently offers the open source Freeswitch voice. Unfortunately, Freeswitch is significantly inferior to Second Life’s Vivox, which is a proprietary platform. A better open source alternative is Whisper, but that one isn’t yet supported by Second Life-compatible viewers.
“Vivox comes with a license requirement, which is why we didn’t go down that road,” Gutteridge said. “I was given a cost in the thousands for grid support.”
Currently, the only hosting provider offering Vivox for OpenSim grids is Avination, which negotiated a special deal for the service.
“It might be that they have changed their policy or pricing,” Gutteridge said.
Meanwhile, Whisper – while a very high quality solution – is not currently integrated with existing Second Life-compatible viewers and users have to download additional software. Freeswitch doesn’t need this integration since it piggy-backs on the connections already in place for Vivox.
“We aim to make things as simple as possible, so we chose to go with Freeswitch,” Gutteridge said. However, if a customer is already looking to use a custom viewer – or plans to install the Whisper software for its users – then PioneerX can set up Whisper for them, and can also set up Vivox if the customer has their own Vivox license.
Another feature not available in standard OpenSim is the choice of starting avatars.
Some hosting providers offer this service, at an additional fee. For example, ReactionGrid’s Custom Avatar Creation System costs $500.
For the do-it-yourself crowd who run the Diva Distro mini-grid, default user avatars can be set up through the free Diva Distro’s Wifi Web front end.
There is no additional charge for this feature with RCI.
“A grid owner will set up an avatar, give it all the assets they want their new people to have, tailor it, clothe it, and RCI will clone it at the moment of creation,” Gutteridge said.
This means that new avatars can be pre-loaded with a selection of shapes, skins, clothing, walking animations, landmarks and other items useful to new residents.
For example, 3rd Rock Grid offers a choice of a male or a female default avatar. Sim-World gives its new users a choice of six different starting avatars.
One variation is a module that requires pre-approval for all new avatars. This is the option that’s used at Curiosity Grid, Gutteridge said, since it is an educational grid and requires all teen members to have prior approval from a supervisor.
The latest feature added to the RCI management panel is the grid dashboard, which shows a number of grid metrics such as the number of people who were logged into the grid on an hourly basis, the average length of time users spent on the grid for the last seven days, which avatars have the biggest inventories, number of rezzed prims on the entire grid, number of scripts running on the entire grid, how many visitors arrived over the hypergrid, and many other metrics.
“The dashboard is designed to give the grid owners an overview of the grid,” Gutteridge said. And the number of graphs is growing steadily, he added. “The grid dashboard is new to this version, we’re still adding graphs to it.”
Next on the agenda is adding reports for currency and transactional data, he added.
Once the features are developed, they will be automatically rolled out to all RCI users, he said.
There is a module for a Dd, in-world grid control center as well, for those grid owners who would like to have one in their in-world management facilities.
“When we were inside OSGrid, we actually had that,” Gutteridge said. “If it’s something a particular grid wanted, we could turn the module back on.”
PioneerX customers can get live support between 9 a.m. and 5 p.m., London time.
“But unofficially, you can get hold of me pretty much any time of day,” Gutteridge added. In addition to Gutteridge himself, there is one additional support tech, as well.
Some clients extend this support option to the users of their grid. For example, users who log into Sim-World can see when the two PioneerX support staff – James Beejay (Gutteridge himself) and Erica Beejay are online. They can also file their own support tickets.
This frees up grid owners to focus their time on value-added activities such as creating content, holding events, recruiting merchants, and welcoming new users.
PioneerX takes daily backups of all the regions and grid assets, with any region able to be restored separately.
Exports of OAR files, which are full region backups, and IAR files, which are fully inventory backups, are not supported by the RCI system but can be requested at any time.
“Doing an OAR export or an IAR export is incredibly system-heavy,” he said. “If we allow people to be able to do it whenever they feel like it, it causes a great amount of lag in the region when the export is being produced. Importing OARs has even bigger problems – it causes a large amount of duplicates in the databases, which cause the databases to grow and load slowly.”
Instead, PioneerX will do OARs and IARs exports and imports when customers request them through the support ticket system.
“There’s no fee,” Gutteridge added. “If you want an OAR backup, we’ll do an OAR backup.”
OAR and IAR backups are useful when customers want to move individual regions or user accounts from one grid or hosting provider to another.
But they can get cumbersome when moving entire grids.
Customers who run a whole grid on PioneerX and find a better hosting provider can ask for a single export of the entire grid.
It’s not a simple process, Gutteridge said.
“To back up a reasonable-size grid takes about two hours,” he said. “Exporting those files will depend on the connection you’re sending it to.”
In fact, he said, it’s usually faster to simply copy the entire grid onto a memory stick and mail it to the new hosting company.
“We will do that for free,” he said. “There’s no lock in, no risk, for our customers.”
The only potential downside to moving away from PioneerX is losing access to the grid management system. But even that doesn’t have to be the case.
“The management panel will work with any provider,” he said. If a customer wants the RCI management panel by itself, without having PioneerX host any of the regions or the grid services like asset databases, then the price is $41 per month (25 pounds) without in-world currency support (or with just OMC) and $65 (40 pounds) with in-world currency.
“The only thing we ask is that if you get the full grid services from us, then we have a three-month minimum commitment because there’s a significant amount of manpower to set up those grid services,” Gutteridge said.
PioneerX builds new features for RCI based on what its customers ask for, Gutteridge said. For example, grids wanted to send out announcements and newsletters, and to have pre-approved avatars, and both of these features were included in the last upgrade.
“As grids come to us and say, ‘This is great, but can it do … that?’ within five days, it will do it,” Gutteridge said.
PioneerX is currently working on a data API, which would allow grid owners to automate the process of getting information from the system.
“For example, if they don’t want to use our newsletter module, they want their own, the API allows them to call in and get a list of email addresses for all of their users,” Gutteridge said.
Another use for the API is for in-world objects to be able to get profile pictures of users.
OpenSim hosting is still in its infancy, with most providers offering only the most rudimentary services through Web-based, self-serve management panels.
ReactionGrid, popular with educators and businesses, offers the R.O.A.R. Advanced Grid Management for OpenSim for $450, a Web-based tool for managing OpenSim regions. ReactionGrid allows self-serve OAR backups and restores — a feature much valued by customers like Begonia Island. The company says it is running more than 100 private grids for clients. However, these tend to be, for the most part, private education and business grids, not public grids like 3rd Rock Grid. With its focus on business and education, ReactionGrid is not known for providing currency support, rental or estate management, or other services of interest to operators of public commercial grids.
SimHost currently has one of the best reputations for managing custom grids for clients, and say that they have offer grid management software to their grid hosting clients. However, they have not yet responded to our requests for more information about this and we cannot say what specific features are available in their management system. We don’t know of any large public commercial grids with grid services provided by SimHost.
Dreamland Metaverse can operate private grids but currently only offers restarts and individual region statistics through a Web-based interface — other functions are handled manually, through support tickets, but a more complete management system is currently under development.
Other vendors are also working on building systems similar to that of PioneerX.
TalentRaspel virtual worlds Ltd. is currently running two private grids for clients — the public commercial grid Reseau Oceax and an internal grid for Baden-Wuerttemberg Cooperative State University, company president Kai Ludwig told Hypergrid Business. The company supports standard grid features like search, profiles, groups, megaregions, voice, OMC currency, and hypergrid. Private grid clients can get OAR and IAR exports, as well as full database exports, if they want to move the entire grid to another hosting provider.
There are no self-serve options for creating or restarting regions, managing users, or getting backups, however.
“Currently, we manage the above by responding to requests within our ticketing system,” said Ludwig. “A Web interface is under construction.”