RAID brings down second grid: What is it, and how to protect yourself

(Image courtesy David Goehring via Flickr.)

(Image courtesy David Goehring via Flickr.)

OSgrid, the oldest OpenSim grid, was brought down by RAID last Monday. Today, The Next Reality Grid was brought down by RAID, as well.

“Like OSgrid, our RAID has also failed completely,” said Next Reality owner Mike Hart in an announcement today. “Luckily we do have backup OARs of all the regions. But not sure about our registered avatars details in the database. Its going to be a few days before the RAID can be replaced and everything restarted again.”

Does your grid have RAIDs?

It probably does!

Stephen Colbert realizes his grid has RAIDs. (No, just kidding. Totally unrelated panic attack.)

Stephen Colbert realizes his grid has RAIDs. (No, just kidding. He’s having a totally unrelated panic attack.)

What is RAID?

Although it sounds like a fatal grid disease — or a toxic spray — RAID is actually a kind of storage. The acronym stands for “redundant array of independent disks.”

A typical RAID storage array. (Image courtesy Newegg.)

A typical RAID storage array. (Image courtesy Newegg.)

The idea is that RAID storage fails less frequently than traditional hard drives, because of the “redundant” part.

Even grids that use other types of infrastructure may be using RAID somewhere.

Melanie Thielker squareAvination, for example, a large commercial grid, bases its central infrastructure on cluster technology, grid owner Melanie Thielker told Hypergrid Business.

Avination uses a cluster to provide a group of identical machines, each able to take over for any of the others,” she said. “However, these machines still do use RAID internally because that is the easiest and most effective way to recover from hard disk failure.”

Not all RAIDs are created equal

The two most popular types of RAID are RAID 0 and RAID 1.

RAID 0 uses multiple drives to store data, in such a way that a single file might be chopped up into pieces and each piece stored on a different drive. The advantage of doing it this way is that you can save the file faster — while the first drive is saving the first half of the file, say, the other drive is saving the second half of the file. It also means you can get the file back faster, as well. But it doesn’t provide a built-in backup.

RAID 1 also uses multiple drives to store data, but it stores a full copy of file to each disk. It’s still faster to save and load data, because if one drive is busy you can use the other one. As a bonus, if one drive goes down, you still have all your data on the other drive. The downside is that you only get half as much storage capacity from the same hardware.

According to Zetamex CEO Timothy Rogers, RAID 0 offers no real advantage for OpenSim.

(Image courtesy SERT Data Recovery Services.)

(Image courtesy SERT Data Recovery Services.)

In fact, if even one drive goes down, it means the whole system goes down. Take the example of a file where half of it is stored on one drive and half stored on a second drive. If the first drive goes down, you’re left with just half a file — a very difficult situation to recover from. In this sense, having RAID 0 is worse than having no RAID at all.

Zetamex itself uses RAID 1 on all its servers, he said. If one drive is down, all the data is still there, fully copied on the other drive.

But Zetamex doesn’t just rely on the RAID mirroring for backups, he added.

“We have fail-safes for our fail-safes,” Rogers said. “We backup to one on-site server at every data center — this means we have a spare server that sits in every datacenter we have that is just used for file backups. Then we have three off-site backup locations: Amazon S3 Glacier, Google Cloud Storage, and CrashPlan.”

Zetamex backs up to all three of these providers daily, he said. “I believe that if X goes out of business or something, I have others to fall back on.”

Rogers urges other grid owners to make regular backups as well. CrashPlan Pro, for example, costs just $10 a month to back up a server.

“You just point it at your OpenSim directory and your MySQL directory and let it run in the background and never have to touch it again,” he said. “It works on Windows, Macs, and Linux.”

For those looking to do it themselves, he recommends the following three steps:

  1. Make a daily export of all OpenSim databases, including Robust
  2. Save OARs — region backup files — regularly
  3. Keep all these files in a separate, secure location

Not all RAIDs are created equal, Part 2

Keeping track of which files are saved to which disks in the RAID system is tricky. One way to do this is with software, but a faster option is to use dedicated hardware in the form of special RAID controller cards, which have slightly better performance.

Dierk Brunner

Dierk Brunner

Dreamland Metaverse CEO Dierk Brunner told Hypergrid Business that it’s likely that the problem that OSgrid and Next Reality grid have is that these RAID controller cards failed.

If that happens you need a new, compatible RAID controller as replacement,” he said. “For compatibility, the RAID controller must not only be a product of the same hardware vendor, it must be the same kind of RAID controller card, often even with a similar firmware revision number. And that is where most problems start and I guess these grids currently experience such a problem.”

In the beginning, he said, Dreamland Metaverse used software-based RAID controllers, and the impact on performance was minimal. “The advantage is that replacing failed disks is easy and there is no risk of failing RAID hardware that cannot be replaced by a compatible new card.”

 

Brunner said that his servers currently have hardware-based RAID controllers, but the data centers that host the servers guarantee that they have compatible RAID controller replacements in stock.

“Once, we had such a RAID failure and within less than a day we could replace the failed card with another new one,” he said.

Brunner said that the recent incidents underscore the need for backups, instead of relying on RAID alone.

“We do backups once a day,” he said. “And once a week we store backup data on a server at another geographic location. This way even bigger catastrophic events such as fires or earthquakes only results in a data loss of maximum seven days. Usually the lost changes are for less than 24 hours.”

Related Posts

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • hack13

    I didn’t state that RAID 0 is a bad fit for opensim, I simply stated
    that RAID 0 proposes no real advantage to opensim. However say you are
    running the database server for opensim, it might be an advantage to
    RAID 0 for faster database reads/writes. But as far as for all simulator
    servers and and such, it is much more practical for RAID 1.

  • Geir Nøklebye

    The I in RAID stands for “inexpensive” as the RAID acronym was created at a time when 1 TB of disk cost $1.5 million 🙂

  • Geir Nøklebye

    RAID 0 + 1, which is a mirrored stripe gives you the best protection in addition to speed for a database. The downside is it use a lot of disk.

    • hack13

      I completely agree, but I honestly don’t see the need to stripe much anymore because SSD’s are pretty fast by themselves if you use SSD’s.

      • Geir Nøklebye

        True for small girds and standalones, but for a bigger grid the asset database fast grows beyond the practicality of SSD.

        • hack13

          That is very true, the cost of SSD’s are still pretty high, and their capacities are not that large. But honestly when it gets to that point, I would just consider 4x 4tb SATA’s in the configuration you specified, to get the most out of it. But honestly if your a big grid, you really should have moved on to a better asset server than the default in opensimulator, as the biggest issue is most grids keep using the default and their asset server gets huge. We use our own De-Duplicating asset server at Zetamex, and many of the clients who have moved over end up shrinking their database of assets down 1/5th the original size.

          We had one grid switch to us that has about 900gigs of assets, we converted them and in shrank all the way down to around just 200 gigs. OpenSim tends to get A LOT of duplicates uploaded, so it is really a big deal to de-dupe assets.

      • Geir Nøklebye

        The biggest issue for a busy database is the number of write transactions the disk system can sustain per second. Since databases typically write 2 kb blocks or even less, we are in reality down to the mechanical limitations in how fast the disk can move the write head, and even modern disks cannot do much more than 50-90 transactions per second when writing that small blocks. I am not sure where the limit for SSD lies but it is by no means stellar.

        For bigger databases we often ended up with over 100 disks in a configurations to ensure enough writes per second, and the disk were distributed on multiple controllers as the controller bus often could not have more than 1 transaction going (some technology can do a few concurrently.)

        From what I see opensim writes a lot and the database fast becomes bloated with old records that should have been removed.

        • hack13

          Again I agree with this statement, we do our own clean up of data tables. We also run many scripts which clean up useless data left behind. Then as far as how much gets written and read from the disk we do our best to move that out of the database into file based services. Such as our de-duplication asset server is based off of SRAS which stores assets as files instead of storing them in the database(which in my opinion opensim should not store assets in the database to begin with its just not meant for that at for opensim’s scale usage of blobs).

          Part of our self-healing inventory service is how it uses compression and runs mainly out of cache to the RAM instead of consuming I/O(tech speak for disk usages, for those that don’t know). There are many techniques that we have applied to improve performance for our clients and future clients. Many of the bases of our stuff I am talking about here can be found already in OpenSim code. A good start to get away from bad SQL is to jump on the PostgreSQL train form the git-go.

          • Geir Nøklebye

            I need to move my opensim bases to PostgreSQL – first prio. 🙂

          • Geir Nøklebye

            Got the multi sim standalone converted over from MySQL to PostgreSQL, which was a semi-frustrating experience.

            On the positive side it is faster. It did however show how inconsistent the database had grown with a number of prims and sims missing texturing + missing sculpt maps after the iar-s and oar-s had been loaded back in. Had to top it up with xml export and reload of many of the support tables to get it back working. Outfits are basically broken for all avatars with invalid links, so there you go.

            I also tried raw table xml export and import, and that worked to some extent, but the inconsistencies seemed to be bigger so I settled for the IAR+OAR route.

            Next step is to take the standalone to grid mode with the PostgreSQL backend.

  • Frank Corsi

    We have always used Raid 5 with hot spare and hardware controller card. We have had issues over the years but never a total failure to cause service outages. Our DB server is costly but does the job correctly.

  • Darren Williams

    Something I think should certainly be discussed and considered for the future is using ZFS for storage, anyone who values their data should take a look at ZFS and best of all, no RAID controllers required. The ZFS equivalent of RAID 10 is striped mirrors and access can be greatly accelerated by the use of SSDs or multiple SSDs for caching something I imagine would be perfect for opensim based datasets.