Hacker destroys OpenSim regions

An unknown hacker took down more than a hundred OpenSim regions this weekend, as well as over a hundred thousand websites.

This is bad news for some OpenSim hosting companies who relied on low-cost virtual shared servers to host their regions.

Chris Greenwell, for example, lost 41 regions total, including the Business Center region on OSGrid, and the Trombly Ltd office park. His backups were stored on other servers which were also hit, and he is currently investigating how much he will be able to salvage.

Greenwell was using the FsckVPS server company, which was running the hacked HyperVM virtual server management software.

Nebadon Izumi, president of the OSGrid — the largest grid running OpenSim — estimates that at least 100 regions went down as a result of the HyperVM hack.

Simon Gutteridge, director of Pioneerx Estates, said that he lost 30 regions to the hack — but that he had off-line backups, so he’ll be able to bring them back up again.

“We have off-site backups of all critical data so regions can be restored once we get the hardware back,” he said. “But some recent additions to our unique web interface will have been lost and need to be re-done.”

For example, Gutteridge said he spent three days working on fixing an asset-related issue with the OpenSim regions, and completed it just three hours before the attack.

“It has forced me to start looking harder at getting fully dedicated hardware and our own grid services,” he said.

Losses were limited to OpenSim regions hosted on Linux shared virtual servers running the HyperVM management software from LxLabs, whose president committed suicide in Bangalore on Monday.

Server companies affected include FsckVMS and VAServ.

As a result, ReactionGrid, which runs the Microsoft stack on servers from Cari.net, was not hit by the hackers.

“We have not been affected in the slightest,” said Chris Hart, ReactionGrid’s chief technology officer.

The problems were limited to servers running Linux-based virtualization technology, she said.

“We have a machine that runs Windows Server 2008 Datacenter Edition with Microsoft HyperV virtualization software,” she said. “This gives us native hardware-level access to the underlying technology and resources on that parent machine for each virtual machine we run, and hence provides great performance. The parent machine that we host at our Cari data center is a full hardware Windows server on which we run Microsoft HyperV to split the resources across different virtual servers.”

The Linux HyperVM problem is not related to the Microsoft HyperV product, she said.

This vulnerability is not a HyperV issue, but a major security hole in one Linux-based virtualization technology that thankfully has nothing to do with the Windows platform.

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • Len W. Brown

    "Losses were limited to OpenSim regions hosted on Linux shared virtual servers running the HyperVM management software from LxLabs, whose president committed suicide in Bangalore on Monday."

    Why did he commit suicide? You referenced his death yet mention nothing of the cause. I'm curious as all hell now.

    Thanks!

  • Devastating.

  • cross examination

    I am confused. Is this an article about a security breach or an advert for Reaction Grid?

  • For more about the suicide, read here:

    Computerworld: http://blogs.computerworld.com/death_of_software_

    IT Wire: http://www.itwire.com/content/view/25559/53/

    Is this an ad for ReactionGrid? No, but it is an ad for taking a business-focused approach to OpenSim hosting, with off-site backups made on a regular schedule. As one of the sponsors of the Business Center, its loss has cast a serious damper on our business outreach efforts. Quite a few people worked on that build, and we're sorry to see it go. Will we rebuild? I don't know yet.

    And the Trombly Ltd. facility, where our offices are, was a project worked on by a large number of people, Jeffrey Scheets in particular, who put in quite a bit of work on the bookstore project.

    I recommend that all business owners ask their service providers to supply them with the OAR files from their regions on a regular basis. This is a step that we didn't take, and now seriously regret.

    — Maria

  • Ever Ready

    Seems the hack was the last straw of many …

    LxLabs boss found hanged after vuln wipes websites
    http://www.theregister.co.uk/2009/06/09/lxlabs_fu

    The article was about both. Reaction Grid just wanted to eloborate in extensive detail why they weren't infested by this bug. Doesn't mean the situation can't be reversed …

  • cross examination

    Ah, sorry to hear about the builds. 🙁 Honestly, anyone who uses OSGrid would have to decide that security is non existent. I can host a region from my house and if you rezzed your build on my land I can snag a copy with godmode as if I am a Linden on SL. Closed garden approaches may not be popular with the open source kids but to protect your content (a little better) and to keep security one must look to these providers. Another question one should ask is for real life name and business address of these providers. SL avatar Joe Blow with their 4-5 friends is just not a wise bet. Neither is OSGrid where no security exists. I hand it to places like Reaction and open life with real names behind the people who expect you to give them money.

  • Any operating system or application has security holes that can be exploited especially OpenSim in its alpha stage. To combat this ReactionGrid has developed a "save to oar" prim created by Chris Hart our CTO which our "Gridizens" can use.

    We also do daily automated backups which are taken offsite. We appreciate the positive comments and also appreciate the very hard work that OSGrid does in moving OpenSim forward and their generous way of providing no cost connections to their grid. They are an inspiration for ReactionGrid & a source of help for us when we get stuck.

    There would be much less progress and visibility without the hard work of OSGrid on OpenSim. All of us hosting OpenSim should try and work together to make OpenSim more secure but as Maria stated the key to this for now and the future is good offsite backups on a regular basis.

    Thank you Cross and Maria for your feedback and positive comments. By years end we will see a much more stable and secure platform due to the efforts of OSGrid, ReactionGrid and many others including the end users who regularly use our worlds.

    See you all in 3D!

    Kyle G-RG CEO & Gridizen

    .

  • This is a perfect example of what happens when you approach IT security with a "Fan Boy" mentality. I'm sure many of the people affected thought… "I'm running Linux, so I'm "Safe"" NOT! This is a painful lesson that hopefully reverberates through both the Virtual World, Linux and Open Source communities. Hopefully, it will cause people, regardless of the OS Platform they choose, to validate the security of their OS and all of their software infrastructure and make sure they have a backup and disaster recovery plan that is sufficient to prevent catastrophic data loss and/or downtime!

  • Pingback: links for 2009-06-10 | Metaverse3d.com()

  • Pingback: Over 100 Opensim regions wiped in weekend virtualization exploit | The gadgets()

  • Pingback: Dusan Writer’s Metaverse » OpenSim Servers Hacked, Metanomics Fail()

  • “Hopefully, it will cause people, regardless of the OS Platform they choose, to validate the security of their OS and all of their software infrastructure and make sure they have a backup and disaster recovery plan that is sufficient to prevent catastrophic data loss and/or downtime!”

    This wasn’t a flaw with linux itself, it was a flaw in an application running on top of it – an application designed for remote management of the virtualisation technology. This would not have happened with a UML, VServer or Xen stack doing all administration via the traditional “ssh in, do your stuff, logout” method.

  • Pingback: Over 100 OpenSim regions destroyed by hacker « Multiverse Newswire()

  • Pingback: Quickest OpenSim instructions ever - Hypergrid Business()

  • While I'm not acquainted with the underlying structure of the hacked site, from what I understand this was a matter of someone using a Microsoft Windows-intended application on a Linux platform. Anytime someone crosses platforms like that, it is their responsibility to make sure security is tight, and to prepare for the worst. As one user noted, this is what happens when non-professional entities try to pass themselves off as pros. It's also a harsh warning to the VR community that it's time to clearn up their act.

    It is true that the entire OpenSim system is a security nightmare. Also as someone above put it, any build that is put on anyone's region can be copied, modified, transferred to others. The only way to insure against that is for the service itself to turn off "God-Mode". But every self-owned mini-grid is going to have full God ability. People need to be aware of that. The World Wide Web today is chaotic and anarchistic in nature. We should not expect a VR web to be any different.

    Effective solution: only do business with reliable, proven companies. I have personally found Reaction Grid people to be trustworthy and customer-focused. I do not have the same feelings about OpenLife, a company that is about as unprofessional an entity as I have come across. It's the old caveat: let the buyer beware.

    I must admit that I have zero respect for the hacker who hit this time. With all the people who are disgruntled with Second LIfe, with all the people Linden Lab (allegedly) ripped off with a "bait and switch" scheme that forced the shutdown of thousands of sims… for a hacker to hit the very folks who offers that company some competition strikes me as nothing more than someone not skilled enough to hit the big company, so he decides to pick on the small fry instead. He's not a "hacker"… he's a hack. He doesn't care who he picks on, he just picks on the easy targets, and doesn't care who gets hurt in the process.

    At one time the term "hacker" was respected, defining a person who could make a computer do whatever they wanted it to do. Companies hired hackers as the most brilliant minds in the industry. Now, "hacker" is associated with antisocial back-room boys who are getting thier jollies causing whatever damage they can, to whomever they can, doing so anonymously. If they had any real skill or sensibility, they'd use their talents to help those who are fighting against the big, self-serving companies by improving the product and shoring up security. Instead, they just harm others indiscriminately. Such people have no honor, no conscience, no true skill. True skill is used to build… not destroy. True courage is evidenced face to face, not cowardly hiding behind a keyboard.

    Them hitting an OpenSim establishment is nothing more than an example of some lunatic who doesn't care who he kills as he pulls the trigger.

  • Wayfinder —

    The hacker also took out over 100,000 websites — I don’t think the hack deliberately targeted OpenSim, and OpenSim might be just an innocent bystander type of victim.

    — Maria

  • Pingback: How to choose an OpenSim hosting company – Hypergrid Business()