Zetamex recovers from breach

Las Vegas-based OpenSim hosting company Zetamex is recovering from a security breach that hit its servers late Sunday night.

Hackers attempted to use the Zetamex servers to attack random websites.

All data is safe, and upon further investigation nothing was lost, no data was compromised,” acting CEO Timothy Rogers said in an update.

Rogers has recently sold the company to a new owner, but is continuing to act as CEO until December 1.

The affected hardware was immediately pulled off the racks, he said. These servers did not contain any personal information or passwords for users, but were running OpenSim regions.

However, for security reasons, the servers needed to be fully reinstalled, which took time.

We have been informed that the datacenter will not be able to do this for five hours as we are not the only server that was effected by this, and we are in que for restore,” Rogers said in another update. “We will be offering service credits to any user that is affected by this outage….  rest assured we are not sleeping until we get everything online.”

By Monday morning, the servers had been reinstalled, and Zetamex began moving data back onto them, and they began coming back online late Monday morning.

Excellent communication

During the breach, Rogers continued to keep customers updated both via the company’s Status Blog, its Twitter account. and its Google Plus page.

He also provided information about how much longer the outage would last, and, about two hours before the servers began coming up again, he posted that they would be back up in two hours.

Zetamex’ smooth handling of the problem — from the initial detection, to the ongoing communication, to the ready availability of backups, and the fast resolution — is in stark contrast to OSgrid, which has now been down for over a month with exactly four updates during that entire time.

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • Zandramas Grid

    Tim has all the indicators of a great business man. Trustworthiness, Great communications, fair, and the most important thing has a process in place to recover from events like this that proves it works and shows that he knows what he is doing.

  • hack13

    When the repair of the breach is over we plan on launching a public statement on how and what was compromised with our full report, as part as Zetamex’s transparency.

    • sometimes i think we are to transparent but meh thats just me.

      • Too transparent:

        “Making coffee for the office now.”

        “Oh, no, the coffee was spilled!”

        “Finding mop. ETA back at support desk: 15 minutes.”

        “Update: Will need to borrow mop from office next door. ETA delayed by another 15 minutes.”

        “We’re back! But wait, we now need to make more coffee…”

        “Support staff offline for next 20 minutes. Taking restroom break. Will post pics on Flickr.”

        • As long as you’re not THAT transparent, you’re okay! 🙂

          • hack13

            lol too funny, but do expect a full report of what occured, so that other grids can patch themselves before they get attacked as well.

        • Zandramas Grid

          Kinda of on like Facebook or Twitter when people update their status message, I am now in the bathroom…. Flushing the toilet 😛

    • Geir Nøklebye

      Most likely related to the bashshock bug that would make any *nix server running Apache and CGI vulnerable for exploits. It went viral last week.

      • hack13

        We don’t like to run Apache and when we do, we NEVER enable CGI. But the exploit was related to shellshock. It was a management tool we use that is Perl CGI, it was supposed to be fully local with no exposed access. Well the application doesn’t accept external logins, a part of it was exposed, and they breached the Perl CGI, and was able to get to bash. This has been corrected on all other servers, and the mangement utilitity now only listens on localhost as it should.

        • Geir Nøklebye

          Glad you got it sorted out!

  • Trying to understand this a bit. “The affected hardware was immediately pulled off the racks, he said.”

    Was this a hardware problem? It seems inconsistent that the provider would remove the actual server hardware for a hacker (software) breach. Or was this just poorly worded and it means the VMs were shut down for a fresh (re)install or VM restore?

    • I don’t really know the full details but Tim will release more info as soon as he can. All i know is that our provider pulled the server to prevent any more attempts. Which i say is the smart and quick thing to do for something like this.

    • hack13

      To explain, when we have hardware that has been compromised we pull it off the rack, disconnect it from the internet. This is to prevent any further compromisation. If we pull it off the rack, then the attack no longer has access only we do. It gives us a chance to go in and repair the problem, and bring it back where it is no longer compromised.

      And no, it is not a virtual machine, it is actualy dedicated hardware. The server was physically powered off and then NetBooted into a Anti-Hack Restore system which generates a secure VPN connection only for us to get in with it not exposed to the internet. Once finished we reboot it from the actual hard drives.

    • hack13

      I honestly wished it was a VM, then it would have been much easier to restore. We snapshot all our cloud VM’s daily, so it would been easier to roll that back but a physical machine not so much.

  • Sammy Greenway

    Keep in mind that OSGrid has an all volunteer staff that has real world jobs and commitments. Zetamax has full time paid staff who can send out updates every hour.

    • We have staff? We have paid staff? Wait, am I getting paid for this? Tim i think you’re sending my pay checks to the wrong address.
      Actually Sammy, Tim is currently the only one who gets paid from sim orders. I do work for free as a volunteer and the new CEO is still in training but I don’t know the money arrangement between the new CEO and the current CEO. We get updates out while we work. Multitasking to the max with us.

      • Sammy Greenway

        Zetamax a company doing it for money, OSGrid is a non-profit. Im not hear to argue, just saying, Zetamax has a monetary requirement to perform. OSGrid does not.

        • hack13

          I will not disagree, we do make money. But we are not doing it for the money, if we were we would be charging the prices the other guys are. The reason I cannot really pay no one, is because we charge our costs.

          • Sammy Greenway

            Yes, but you cannot do that forever. This is a normal tactic for building a base, but then you have to raise your rates eventually. No way around it. I work for a insurance company that was recently bought out by a larger one and started doing a lot of advertising. They sold policies at a loss to gain customers, but have had to start raising rates recently to help stop the red ink from flowing. Common practice. I will give credit to Tim, he has come a long way, I remember investing some time and helping out with a grid that he owned, FreeOpenSim I believe was the name, just to have him shut it down without notice. That kinda soured me on him for a while.

          • hack13

            Perhaps not, but that’s why I no longer own the company. Let other people worry about that stuff, let me get back to focusing on my passion, which is developing for virtual worlds. I am not in it for the money, I just want to create tools to make life easier for everyone, not just Zetamex customers. Zetamex just offers me the ability to afford to make these free and inexpensive services and tools for the community.

            PS: As for the part you removed from this comment, FreeOpenSim was not a grid that I Zetamex owned, it was ran by a customer whom left without notice, and didn’t respond to emails and their account ended up getting closed for failure of payment. People tend to blame me for that, but it was not our fault. There were a lot of other things that happened, such as a few users lost inventory, and they demanded copies and we were not allowed to give them away as they were not on the owner of the grid’s access list for us to give them legal power for us to respond to requests like that. It was a bit of a mess from a long time ago, but that’s why we have implemented and working on new policies before Nov 1 with the new CEO for people who own grids, they will have to fill out a form stating whom can contact Zetamex and edit their account.

          • Sammy Greenway

            Thank you for your clarification. I removed it because I wasnt here to complain about anyone. I will say that you have done a good job with what you have done with Zetamax.

          • same passion as me which is why i am just a volunteer.

  • Sammy Greenway

    My comment correcting the error that there were exactly four OSGrid updates was removed, so im reposting. I count 7 twitter updates, 3 news posts on the OSGrid website, Several updates in the OSGrid forums, and daily updates by admin on their freenode irc channel. Most of the time they just say, still working on it. Its like watching paint dry, not much to do but wait, but no real updates sometimes.

    • lmpierce

      Hi Sammy,

      Your comment wasn’t removed. Comments with links are all automatically withheld pending moderation. I’ve approved the original comment with the link. (There is usually a delay like this.)

      • Sammy Greenway

        Thank you 🙂 Im not here to argue, BTW, just want to give my 2 cents… In the end thats about all my thoughts are worth some times, heh.

        • lmpierce

          Sure, I understand. Providing information, even contrary information, is a big part of why we have these forums.

    • You’re right — there were three posts on http://news.osgrid.org/ — and Lani and Key posted a couple of announcements in the forums.

      However, OSgrid is a grid with thousands of users, and — I hope! — more than one volunteer. I don’t expect them to quit their day jobs and post hour-by-hour updates of what’s going on with the outage. But I do expect to see regular updates of what’s going on, and some community outreach.

      The IRC channel is not an adequate method of communication — if you don’t happen to be on when someone says something, you miss it, and how many people use IRC these days, anyway?

      I think most of us were willing to give OSgrid our trust that they were working things out, at least for the first couple of weeks. And not second-guess the decisions that were being made since nobody wants to have a backseat driver, especially in a time of crisis.

      • Sammy Greenway

        As far as admins, My understanding ( and its just my understanding ) there are 3 admins at this time, Hiro, Dan Banner, and Key Gruin, two of which, Hiro and Dan, can actually do any kind of maintenance to the services. And I believe all three have real lives and jobs. Neb no longer messes with it, he has access, but has other projects that trump OSGrid. Problem with OSGrid, is that, and the admins repeat over and over, its a Test Grid. They are not obligated to give daily updates. With that being said, OSGrid is kinda in a weird position that other grids dont have. They are a Test Grid, yet they are a very large community. And that is why they are making an effort to recover the database. I heard of many people who did not want to recover the database, and just start fresh, since it is only considered a Test Grid, (as a side note, being a Test Grid may be why it doenst have a backup solution), but due to the fact there are a lot of people there, they are going out of the way to try and recover the data. OSGrid may have an identity issue.

        • hack13

          Thank you for saying what I been trying to say to people for years. It is and always has been first and foremost a TEST GRID! I been wanting to write an article about it, but I fear I would get a lot of grief. But I honestly have felt like, OSgrid should just ditch the trying to recover and just restart with what they have.

          Why?
          Because it is a test grid, it always has been. They run dev code on purpose, they try and make OSgrid break weekly, its the point of OSgrid, to break shit. That is because it’s a Test grid. I think people who want a stable environment should run their own grid or connect to something like Metropolis that runs stable releases.

          I mean some people might think I am crazy saying this, but its honestly how I feel. Its a test grid, if you cared about your stuff you should have been backing it up. Because its the grid where things break on purpose.

      • Sammy Greenway

        Of course this article isnt about OSGrid, its about Zetamax.

  • Sammy Greenway

    Two good topics can come out of Zetamax and OSGrid, backups and security.