6 disaster management tips for grid owners

1024px-USAF_EOD_explosion -- wikimedia

So your grid has been hit by a denial-of-service attack, or a server has crashed, or a hacker got in, or things just mysteriously stopped working. Or maybe you’ve had a customer service disaster, or your owner said something horrible on social media, or you’ve got management or partner problems that spilled out into the public arena. Or maybe you’ve changed your business model or your service offerings, and your customers didn’t react the way you expected them to.

Or maybe all of that happened all at once.

Some grids and hosting companies are able to recover from problems and go on even stronger than before while others are forced to close down, or suffer irrevocable damage to their reputations.

Here are some tips for managing a crisis well.

1. Prepare

If you’ve prepared well for a crisis, you’ll have solved most of your problems before they start.

Think about the kinds of things that could go wrong. Key personnel may be out of touch because they’re in a hospital with a broken leg. Servers can crash, hosting companies can go out of business with no warning, a senior executive might have an embarrassing and public sex scandal.

Since there are an infinite number of things that could go wrong, focus on the ones that you feel are mostly likely or that have happened to other companies in your industry.

For example, if your servers go down and take with them your website and your grid, you could pay for temporary cloud-based servers until the problem with your regular servers is fixed. Or you might opt for just pointing your domain name temporarily to a page that keeps your customers updated about the problem.

Or say you’re worried about key staffers losing Internet connectivity due to service outages in your area. You might invest in a pay-as-you go portable Wifi hotspot, or find out which neighborhood cafes offer free broadband.

And, of course, have backups of all your key data, documents and applications and store those backups in a different location.

The U.S. Small Business Administration offers a variety of disaster recovery resources, including a guide to crisis communications.

2. Communication

Speaking of communications, every grid and hosting company has its preferred channels. Facebook, Twitter, LinkedIn, their own website, a customer email list, an Internet relay chat service, an online forum, or another channel specific to their particular industry or customer segment.

Create a secure shared document with all required access information, keep it updated, and make sure to remove people from the access list when they no longer have the authority to communicate on behalf of your company — and update all passwords when you do.

Google Docs and Dropbox are two free options, but there are also more secure, business-friendly alternatives as well, such as Box.com.

Enable two-factor authentication on all accounts that offer it. Here’s how to set it up for Google, Facebook, and Twitter.

When a disaster happens:

  • Acknowledge that there is a problem and that you understand why people are upset.
  • Explain the background of the issue, but only if people need to know. For example, if you had a misconfiguration error that other people may be likely to make, let them know. Or if your hosting company had a power outage and it took down your servers. Do not bring up details of customer support or personnel issues.
  • Address key customer concerns, such as whether their data has been lost.
  • Explain what you are doing to fix the problem, when the problem is likely to be fixed, and what steps you are taking to make sure it doesn’t happen again.
  • Explain how you will make your customers whole. Will you give them a refund for the period of the outage, for example?
  • Say when your next update will come out.

3. Keep it civil

You customers have a right to be upset. They have the right to call you names, and to tell everyone they know what an awful company you are.

But you don’t have the right to be upset at your customers in return. Acknowledge their anger. “We messed up. We deserve that. Here’s what we’re doing to make sure this problem never happens again.”

This also applies to employees and business partners. Even if the problem is not your fault — if your servers were swallowed up by an earthquake — you still need to take responsibility because it happened on your watch.

If you feel angry and stressed out, this is not a good time to engage in any public communications. Step back, take a deep breath, relax.

If you know that you have a problem with your temper, and don’t have a polite person you can delegate communications to, one thing you can try is to create your announcement ahead of time, before anything goes wrong, then just fill in the blanks.

For example:

We care deeply about our customers/employees/partners/members of the public.

We sincerely regret the recent unscheduled downtime/login problems/database issues/way we handled a customer support issue/sexual harassment incident/the fact that you had to see naked pictures of our management team.

This is not what our company is about. We are working hard with our hosting company/with our development team/with our support staff/with our lawyers to resolve this problem.

We expect the issue to be resolved by such-and-such date.

We will update you again this evening/tomorrow/daily/by the end of next week.

We sincerely apologize for the incident and will be issuing refunds/crediting accounts/updating our backup systems/training our employees/switching hosting providers.

Then, if people complain, just say:

We understand your frustration. We are working hard to get things back to normal and will keep you updated.

Keep it brief, keep it polite, and make sure that everyone who is communicating on behalf of your company is working from the same playbook.

4. Do you really need a postmortem?

You may be tempted to let everyone know all the details about what went wrong.

First, if the crisis isn’t over yet, don’t say anything. Wait until everything is resolved and all the facts are in. Reacting impulsively to events as they occur is likely to do much more harm than good. Plus, waiting until everything is fixed will also allow tempers to settle down.

Then, consider not doing a postmortem at all.

If the reason you want to do one is to prove to everyone that the problem was  not your fault — stop! The problem was your fault. Even if there was an earthquake — you picked that earthquake-prone area to put your servers in. If the problem was caused by your hosting company, you picked that hosting company. If the problem was with your employees, you hired those employees and were supposed to be managing them. If the problem was caused by a griefer, at the end of the day you and you alone are responsibility for the activity on your grid and for setting and enforcing conduct rules. Do not point fingers. Do not make excuses. And for heaven’s sake, do not call anybody names or accuse them of illegal activity. It makes you look bad, and opens you up to potential lawsuits.

And most importantly, no matter how you feel about it, do not blame the customer. Ever. Ever. Ever.

If the reason you want to do a postmortem is to demonstrate to your customers that the problem will not happen again, then go ahead:

  • Explain what caused the problem in a non-judgmental way, or in a way where you take the responsibility. For example: “We did not adequately plan for database growth,” or “There was a fire in the server room.”
  • Explain what you did that contributed to the problem. Maybe you could have picked a different hosting company, or kept better backups, or tested new configurations first before rolling them out to everyone. If you didn’t do anything that contributed to the problem and don’t plan to change any behavior, then that means that you will have learned nothing from the experience and the problem will happen again. Nobody wants that.
  • Explain what you have learned, what changes you’ve already made, what you will do differently in the future, and what recommendations you have for other people with a similar problem.

5. Don’t worry about being right

If you feel the need to “set the record straight” or correct misunderstandings that customers have, or mistakes that the media is making, stop and think first.

Do your customers really need to know more details than what you’ve already said in your statement?

Say people are complaining that the outage was seven days long when it was only four days long. Don’t accuse them of lying — they might have experienced unrelated problems that made the outage seem longer to them.

In fact, you can simply ignore most such statements. People will say all sorts of weird things. You might be accused of, say, fleeing the country with all of your customers’ money. Ignore it, and just go about doing your job, fixing the problem, and restoring services to your customers.

If the rumor comes up again and again, you might want to add it to an FAQ on your website. “Does your company launder money for terrorists? Answer: No, we just provide grid hosting services.”

Or you can address concerns raised by an Internet search for your company name: “How does your company guard against service outages? Answer: We use a premium hosting service, with both real-time and off-side backups and protection against distributed denial of service attacks.”

Check out the Great Canadian Grid’s notice for how they’ve reacted to a recent space of such attacks.

Most people will give you the benefit of the doubt over some random person on the Internet, especially if, in the future, you do as you promised.

6. Overcome bad news with good news

It’s true that the Internet has a long memory. But it’s even truer that the more recent the news, the higher it will rank in the search results.

So ensure that there’s steady communications coming from our company about the good stuff you’re doing.

Don’t go overboard to try to get the old bad stuff taken down. If a polite request doesn’t result in the post’s removal or correction, further attempts are just likely to backfire by drawing more attention to the original problem, or sparking a new round of negative discussions.

Over time, old posts sink in the search results, websites get taken down, and people forget things.

For example, a company recently wrote me to thank me for a good article after all the trouble they had with me in the past. I didn’t remember what trouble they were referring to — but now I’m curious!

Don’t keep bringing up bad stuff. Keep your communications positive.

For example, the Great Canadian Grid talks about its DDOS protection measures without mentioning the problems they’ve had in the past.

7. Consider rebranding

In 99.9 percent of cases, fixing a problem and learning from it will be better for your company than shutting everything down and starting over from scratch.

You can even have multiple bankruptcies and thousands of lawsuits against you and still run for president.

But sometimes things happen that you just can’t recover from. For example, your company might be named “Isis” or “Santorum.”

It’s never easy to change your name. Not only do you have to go around and fix all the links, change all your social media accounts and get new business cards printed up, but you also have to start over from scratch with brand awareness.

Plus, if you do it often, you’ll get a reputation as someone who leaves a trail of failed businesses in their wake and the people who’ll most want to work with will start avoiding you.

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • Carlos Loff

    Very useful, as always

  • Gunnar Schwede

    Maria, great article and great suggestions/guidelines!

  • Freda Frostbite

    Yes! Every word of it– yes!

    • Freda Frostbite

      My comment above is referring to Maria’s article, not the mess posted above me in this forum. I would also like to say that there are some grids I will not touch with a ten-foot virtual poll because of the way their owners behave when under fire or in crisis mode. It’s not that I don’t get it. I can be a hot-head myself, but never when I am being paid for my time, efforts, and good attitude. (Well, there was the one boss who called me “girly” one too many times, but I digress.) The way some grid owners behave right out in public in these forums and in social media appalls me. I will never be associated with them. Never.

  • skylifegrid

    SkylifeGrid will not be resurrected but instead we will be Starting up a very new grid name Digitaleisure It will be a spin of what we had accomplished with Alex and AviLabs – the drama and lack of communication.. Since the grid AviWorlds and AviBrasil is down Anyone currently with AviWorlds or Avi-Labs that are PAID customers can choose to move their region over to our grid with no hassle As for the Inventory’s of all your avatars Their is not much we can do about this as Alex retains the database and wont allow us to get a copy. Your land and regions will have all content all stores will have all items for sale.. you could easily take copy’s of the inworld content to recuperate some of your inventory… anyone wishing to do so please email me at [email protected] and I can see what can be done in recovering your data Thank You

  • Alex Ferraris

    Amazing…
    I did lock the servers up but Josh and Quill hacked it and changed the passwords behind my back without my knowledge.
    I fired them both and as retaliation now AviWorlds and AviBrasil are down.
    Anyway I was the one keeping it all alive I was the who restarted the grids and was always in touch with the players.
    Unfortunately Josh and Quill are not up for all that.
    I have emails from both Josh and Quill confessing they hacked the server.
    Simple just hack the server again right??
    Anyway peoole want to trust these two its there own fall.
    Josh did help alot and even paid for two servers while I was away and I give him that.
    Give credit to whats due
    But if he was alert when hell broke loose perhaps we could avoided all the problems.
    God Speed to Josh and Quill on their new venture

  • skylifegrid

    we hacked nothing Alex and for your information we have been up untill the day you accused us of this was helping you stay afloat .. I jumped ship before Everything crashes and burns because of your lack of communication and letting servers expire without notice leaving residents with out regions they paid for … I am going to try something on my own without your financial backing and that’s all you were is Financial backing of the business but after all the funds dried up I had to pay for the servers in the end… This is better for both of us, as you never communicated properly and only ever demanded things instead of asking or suggesting. You changed your business model without talking to your staff, not thinking the effects it could or would have on us… I wish you well but please stop making up this junk just because we decided to jump ship leaving you stranded. If it was not for me and Quill you would never have made it this far.. That said Everyone currently with ALEX can make the switch and if Alex is Nice enough he will release the assets from aviworlds so as you can all have your inventory’s back we have a 270 GB asset server and If permitted I would make a copy Alex are you going to allow the users to have the data ?

    • Alex Ferraris

      Hack into server again Josh and get it yourself

      • skylifegrid

        Turn on the server damn it Alex stop being a Child and let the data download !!!!

        • Alex Ferraris

          Did u hack it? What is the new password Josh? That is AviWorlds data. It is not yours! Stay off the server!

          • skylifegrid

            ALEX your not reading what i wrote There is no server The ip Resolves nothing and we have not HACKED anything up until you started accusing us of this we were 100% on your side helping you you really must have hit your head hard to think we the people that have been saving you in media and in financial have been the ones to harm you.,,, Maybe you been inhaling too much diesel fuel on your new job ? Turn the damn server back online Alex and let the download finish!

        • skylifegrid

          All the regions of AviWorlds are still safe its the Inventorys the users lack and from what im gathering Most of the Current users are going to be switching over to Digitaleisure …. We will have it so they will not even need to re register only to login… The sad part is we do not have the inventory to go with it….. and this is something you can resolve Alex Stop being so nasty we developed your SRAS system we developed all the key features of AviWorlds we have rights to use our creations and You have no reason to stop us from keeping the residents happy by getting the data back for them

          • skylifegrid

            You turned the server off The IP pings NOTHING soo no one can access or do anything untill its back on ALEX

    • Alex Ferraris

      And you should not use any of the content from AviWorlds. No oars and no data. Thats not yours to keep or use.

      • skylifegrid

        So that’s a no. I hope you bring back everything because your customers and residents are the one being hurt not to mention the Avi dollars that have been requested paid out but never have. Your hurting the ones that built Aviworlds

  • Alex Ferraris

    When I fired Quill for incompetence and basically not doing anything I asked him to do; Him and Josh both hacked into AviWorlds server and kicked me out by changing the password which I had changed to keep Quill off the server.
    Josh and Quill together conspired against me so I fired Josh too.
    The next day the server was hacked and AviBrasil website also hacked.AviBrasil was offline and the home page had sexual content all over. Something Josh is notorious of doing.
    Im sure Quill and Josh can hack the server again no problem. Retaliation and hacking is what they do best.
    I wish them both LUCK in their new venture.

    • skylifegrid

      That’s the think you turned off the server we can’t ping it let alone make connection. You need to stop this childish act and allow the users to recover from this. Just because your angry does not mean your customers and grid users should suffer

      • skylifegrid

        Turn on the Server and allow the users to get the data they spent so long on! This is not fair to anyone and your shutting down out of the blue is NOT fair to your customers and people that have been building AviWorlds from nothing to what it was/is You need to allow the data to be recovered !!!

    • skylifegrid

      Accusing us of Hacking is slander and we NEVER hacked you we had all the passwords up untill you locked us out …. you need to turn back on the server and allow the users to get the IARS and or Files back this is Theft on your part if you want to speak legality.

  • Thelma Marks

    Well I see no one is following the posting of your article HGB and they are doing what not to do again in your comments. How unprofessional and rude this article is not even about AviWorlds or AviLabs or whatever.

    • Alex Ferraris

      Well… They are talking about me here. I only am explaining what happened. The truth will set you free…

  • skylifegrid

    Digitaleisure Is now being installed 🙂 Like I posted before anyone that wishes to move their region over can do so without any hastle Your inventory WILL NOT be there anymore but your login credentials and regions with all the content will.. For the creators that have stores you can simply take copys of your store items to have them back in your inventory .. This goes for anyone that has things Rezzed in world. We will try to recover the inventorys but there is a little hump in the way. Happy Virtual Reality to everyone …. Contact can be made at [email protected]

    • Thelma Marks

      It sounds like you have good intentions on what your doing but you do realize there are legalities involved in taking people’s user data without consent. It also is there to protect users so that they may opt out. You may want to consult a lawyer. When companies change hands both companies are supposed to do press releases etc this stops a lot of hostile takeovers and from what it sounds like if the owner of the other grid is not in agreement with what your doing this sounds almost illegal. I am by all means not a lawyer but from what I’ve heard and read what your doing could be seen as a problem.

      • skylifegrid

        My friend Thelma the data that is in question belongs to the users that created it Yes not to any one business or Domain name … The users that do not wish to be apart of it do not have too BUT i doubt that anyone that has invested time into the regions or all the content they built will just say Screw it I don’t need it…. The current provider is not providing and has stopped paying bills.. when it comes down to it Who the Hell cares ? Alex sure didn’t when he flipped the switch and blaming us !

    • skylifegrid

      Correction ! The Database retaining all your user accounts is Lost have no fear If you wish to still merge You can do so by recreating your users account and we can still do our best to get your region items back for you Send requests to email above

      • Thelma Marks

        You may also want to think about region oar files those may have content from people that have not opted out and by uploading these won’t they go in to the region owners name. This is why I really like kitely and how they have their stuff set up Llan has done an amazing job on preserving creators names On Kitley and it’s why I make it my primary home. Again you may want to consult a lawyer on this.

        • I actually wrote about the right issue during one of AviWorlds’ (many!) previous management meltdowns:

          http://www.hypergridbusiness.com/2015/04/aviworlds-vs-aviworlds-battle-rages/

          By default, copyright belongs to the original creator and, without a written agreement, the buyer only gets first-use rights to that content… This should be a wake-up call for anyone considering buying or selling a grid. Make sure all content is correctly licensed. In addition, if users and their regions and inventories are being sold, then those users should have signed agreements to the effect that give the grid owners the right to transfer all their content as part of the sale of the grid.

          In particular, grid owners should include in their terms of service that their users give them permission to move their content over if they switch hosting companies, and if they hire builders, have a clause in the contracts that lets them continue to use the content if they sell the grid, relaunch it under a new name, switching hosting providers, etc..

          • Thelma Marks

            Yep that’s a good write up you did! Thanks for sharing!

        • skylifegrid

          All the backups are done hard by Databases and the main files all primitives should retain the correct permissions and user creation dates and names…. No region will come online untill the owner has contact me requesting it.. As i have to pull out all the files from storage and sort though them all to do each one.. I have emailed Alex about helping him try to sort his current issues but not a peep back.. My 2 cents is he turned all the servers off and gave us the boot because he was tired of the grind .. but who am I to say .. I will continue my work and if I am taken to court well so be it because we have our rights and so does the community AviWorlds TOS may over though that but I was the one to give Alex the TOS in the first place

          • skylifegrid

            Everything that was done in AviWorlds from Features and additions Up to the marketplace and custom Asset server was done by ME and Quill and we took nothing in return but grief.. anything we did was never fast enough and or good enough for Alex even tho we were not paid to develop anything for him.. He expected the world from us in return for peanuts… I am glad to be done with this drama.. But I fear it has just begun

    • Da Hayward

      Josh I understand your ims are capped at moment I sent you a notecard in sl as my email is still down.
      We wish you the very best with your and Quills new grid.

      • skylifegrid

        Sounds good Hayward 🙂 I hope you are feeling better!

        • skylifegrid

          Great conversations in secondlife looking forward to the future

      • Alex Ferraris

        Im surprised to see that u have internet connection after all Da Admiral. Just remember what u thought about josh and Quill. And remember I was the one who always took care of your regions.
        And I was the one who was always available to resolve any issues u had.
        Its just sad to see how people can turn 360 degrees on you for 500 dollars they owe.

  • lmpierce

    A reminder about discussions and moderation… User comments can be flagged, but are only removed if they violate the discussion guidelines. If you feel a comment violates the guidelines and I have not already removed the comment after it was flagged, please send me an email and make reference to which guideline you feel is violated ([email protected]). Please keep in mind that comments are not removed because the commenter says something a reader disagrees with or does not like.

    Flame wars are a violation of the discussion guidelines. They are also difficult to moderate. In the current discussion, the tone has drifted from melodrama, to flame war, back to melodrama, to reasonable discussion and back to melodrama…. and so on. That is difficult to moderate with precision, so I have been allowing most of the heated discussion. Outright accusations of fraud are removed when I catch them, but innuendos have been flying around at a breakneck pace. Therefore, to keep this particular discussion open, tone it down. By ‘tone it down’, I mean stop yelling at each other and start talking. Do not make accusations, but do put forth explanations of problems and consequences. If the other person won’t listen to you talking, they certainly aren’t going to listen to you yelling.

    Please note that moderation is not instantaneous 24/7. Sometimes moderation is rather immediate, other times some hours will have passed.

    • Thelma Marks

      Thanks for your moderation I can only imagine how much work it is.

      • lmpierce

        Thank you Thelma.

        • TribeGadgets

          Interesting and appreciate the task you have especially in a 24 hour universe.. However it does seem accusations and innuendos are still flying – I wonder if those involved realise that by doing so in a public access forum response arena that they are – publicly accessible. And searchable. And archived.

  • skylifegrid

    With all this drama I need a break.. Time to go camping! Packing and leaving in a few ! Need a break!

    • I’m back everybody. Any posts you see between last week and today are because my old account got hacked by Genesis Grid! I had to sign up fro a new account.

      • Digitaleisure Grid

        This is not me and is a person posing as me beware

  • Alex Ferraris

    At this point Im keeping AviWorlds main server turned off. Josh and Quill want to download all content from AviWorlds to place into their servers.
    I say this. The content belongs to AviWorlds. Any attempt to try to get it against my permission.
    Josh tried last night again.

    • Thelma Marks

      Why not change passwords so they can not do so? If your the owner and your not allowing a transfer of ownership of customer data to take place then you have every right to stop them.

      • Alex Ferraris

        I did change the password and the next morning they had hacked into the server and changed the password without my consent. I have two emails from them where both confess they changed the passwords. So no matter how many times I change the passwords they can hack into the server and change it. The minute I place the server back online they will get into the server again and try to download all the content without my permission and then they will say they did not do it. I also have an email where JOSH is complaining that the server is down so he couldnt complete the download.

  • Da Hayward

    this is just going on too much.
    why even bother if its going to be such a screaming match?

  • Alex Ferraris

    A lesson learned.
    Since 2010 I have been trying to build this opensim business.
    It is not easy.
    I have tried everything I can and cant on this business. There are tremendous amount of competition involved combined with the need to acquire TRUST from other businesses like the creators and other types of business that can be incorporated into it.
    There is a need of money and knowledge because once a virtual world is up and running there will be a need to develop AN ECONOMY and that will be the building block that will sustain the virtual world.
    If it is an opensim hosting company there is tremendous need to find people you can trust ALWAYS.
    People that will be responsible and not incompetent and lazy and never available. A virtual world hosting company or a virtual world need to have 24/7 dependency on their customer support and technical support.
    I have tried many business modules and believe me I did try hard with passion but PASSION alone is not enough. The type of business needs to be the one that has a chance of producing a PROFIT.
    I think that if the virtual world is not going to produce a profit then just make one in your computer at home for leisure. For this reason I am going to say here with a heavy heart that I had my run and I think it is time to give up and let the horse finally die.
    6 years and I have learned a lot.
    I thank you all and I do thank the opensim developers who without them we would not have OPENSIM. I thank MARIA here for the amazing articles and the constant reminders that I shut down 8 times.
    I may open a little play grid in my own server at home one day who knows… But for the time being I had enough. Enough with the dishonest people that I got involved with and for that I am sorry.

    When a business does not produce a profit it needs to be out of existence immediately. I think I let the passion I have for virtual worlds get in the middle of all this.
    What happened here in the last 3 weeks will be a lesson learned.

    Good bye all.
    Alex Ferraris.