6 disaster management tips for grid owners

1024px-USAF_EOD_explosion -- wikimedia

So your grid has been hit by a denial-of-service attack, or a server has crashed, or a hacker got in, or things just mysteriously stopped working. Or maybe you’ve had a customer service disaster, or your owner said something horrible on social media, or you’ve got management or partner problems that spilled out into the public arena. Or maybe you’ve changed your business model or your service offerings, and your customers didn’t react the way you expected them to.

Or maybe all of that happened all at once.

Some grids and hosting companies are able to recover from problems and go on even stronger than before while others are forced to close down, or suffer irrevocable damage to their reputations.

Here are some tips for managing a crisis well.

1. Prepare

If you’ve prepared well for a crisis, you’ll have solved most of your problems before they start.

Think about the kinds of things that could go wrong. Key personnel may be out of touch because they’re in a hospital with a broken leg. Servers can crash, hosting companies can go out of business with no warning, a senior executive might have an embarrassing and public sex scandal.

Since there are an infinite number of things that could go wrong, focus on the ones that you feel are mostly likely or that have happened to other companies in your industry.

For example, if your servers go down and take with them your website and your grid, you could pay for temporary cloud-based servers until the problem with your regular servers is fixed. Or you might opt for just pointing your domain name temporarily to a page that keeps your customers updated about the problem.

Or say you’re worried about key staffers losing Internet connectivity due to service outages in your area. You might invest in a pay-as-you go portable Wifi hotspot, or find out which neighborhood cafes offer free broadband.

And, of course, have backups of all your key data, documents and applications and store those backups in a different location.

The U.S. Small Business Administration offers a variety of disaster recovery resources, including a guide to crisis communications.

2. Communication

Speaking of communications, every grid and hosting company has its preferred channels. Facebook, Twitter, LinkedIn, their own website, a customer email list, an Internet relay chat service, an online forum, or another channel specific to their particular industry or customer segment.

Create a secure shared document with all required access information, keep it updated, and make sure to remove people from the access list when they no longer have the authority to communicate on behalf of your company — and update all passwords when you do.

Google Docs and Dropbox are two free options, but there are also more secure, business-friendly alternatives as well, such as Box.com.

Enable two-factor authentication on all accounts that offer it. Here’s how to set it up for Google, Facebook, and Twitter.

When a disaster happens:

  • Acknowledge that there is a problem and that you understand why people are upset.
  • Explain the background of the issue, but only if people need to know. For example, if you had a misconfiguration error that other people may be likely to make, let them know. Or if your hosting company had a power outage and it took down your servers. Do not bring up details of customer support or personnel issues.
  • Address key customer concerns, such as whether their data has been lost.
  • Explain what you are doing to fix the problem, when the problem is likely to be fixed, and what steps you are taking to make sure it doesn’t happen again.
  • Explain how you will make your customers whole. Will you give them a refund for the period of the outage, for example?
  • Say when your next update will come out.

3. Keep it civil

You customers have a right to be upset. They have the right to call you names, and to tell everyone they know what an awful company you are.

But you don’t have the right to be upset at your customers in return. Acknowledge their anger. “We messed up. We deserve that. Here’s what we’re doing to make sure this problem never happens again.”

This also applies to employees and business partners. Even if the problem is not your fault — if your servers were swallowed up by an earthquake — you still need to take responsibility because it happened on your watch.

If you feel angry and stressed out, this is not a good time to engage in any public communications. Step back, take a deep breath, relax.

If you know that you have a problem with your temper, and don’t have a polite person you can delegate communications to, one thing you can try is to create your announcement ahead of time, before anything goes wrong, then just fill in the blanks.

For example:

We care deeply about our customers/employees/partners/members of the public.

We sincerely regret the recent unscheduled downtime/login problems/database issues/way we handled a customer support issue/sexual harassment incident/the fact that you had to see naked pictures of our management team.

This is not what our company is about. We are working hard with our hosting company/with our development team/with our support staff/with our lawyers to resolve this problem.

We expect the issue to be resolved by such-and-such date.

We will update you again this evening/tomorrow/daily/by the end of next week.

We sincerely apologize for the incident and will be issuing refunds/crediting accounts/updating our backup systems/training our employees/switching hosting providers.

Then, if people complain, just say:

We understand your frustration. We are working hard to get things back to normal and will keep you updated.

Keep it brief, keep it polite, and make sure that everyone who is communicating on behalf of your company is working from the same playbook.

4. Do you really need a postmortem?

You may be tempted to let everyone know all the details about what went wrong.

First, if the crisis isn’t over yet, don’t say anything. Wait until everything is resolved and all the facts are in. Reacting impulsively to events as they occur is likely to do much more harm than good. Plus, waiting until everything is fixed will also allow tempers to settle down.

Then, consider not doing a postmortem at all.

If the reason you want to do one is to prove to everyone that the problem was  not your fault — stop! The problem was your fault. Even if there was an earthquake — you picked that earthquake-prone area to put your servers in. If the problem was caused by your hosting company, you picked that hosting company. If the problem was with your employees, you hired those employees and were supposed to be managing them. If the problem was caused by a griefer, at the end of the day you and you alone are responsibility for the activity on your grid and for setting and enforcing conduct rules. Do not point fingers. Do not make excuses. And for heaven’s sake, do not call anybody names or accuse them of illegal activity. It makes you look bad, and opens you up to potential lawsuits.

And most importantly, no matter how you feel about it, do not blame the customer. Ever. Ever. Ever.

If the reason you want to do a postmortem is to demonstrate to your customers that the problem will not happen again, then go ahead:

  • Explain what caused the problem in a non-judgmental way, or in a way where you take the responsibility. For example: “We did not adequately plan for database growth,” or “There was a fire in the server room.”
  • Explain what you did that contributed to the problem. Maybe you could have picked a different hosting company, or kept better backups, or tested new configurations first before rolling them out to everyone. If you didn’t do anything that contributed to the problem and don’t plan to change any behavior, then that means that you will have learned nothing from the experience and the problem will happen again. Nobody wants that.
  • Explain what you have learned, what changes you’ve already made, what you will do differently in the future, and what recommendations you have for other people with a similar problem.

5. Don’t worry about being right

If you feel the need to “set the record straight” or correct misunderstandings that customers have, or mistakes that the media is making, stop and think first.

Do your customers really need to know more details than what you’ve already said in your statement?

Say people are complaining that the outage was seven days long when it was only four days long. Don’t accuse them of lying — they might have experienced unrelated problems that made the outage seem longer to them.

In fact, you can simply ignore most such statements. People will say all sorts of weird things. You might be accused of, say, fleeing the country with all of your customers’ money. Ignore it, and just go about doing your job, fixing the problem, and restoring services to your customers.

If the rumor comes up again and again, you might want to add it to an FAQ on your website. “Does your company launder money for terrorists? Answer: No, we just provide grid hosting services.”

Or you can address concerns raised by an Internet search for your company name: “How does your company guard against service outages? Answer: We use a premium hosting service, with both real-time and off-side backups and protection against distributed denial of service attacks.”

Check out the Great Canadian Grid’s notice for how they’ve reacted to a recent space of such attacks.

Most people will give you the benefit of the doubt over some random person on the Internet, especially if, in the future, you do as you promised.

6. Overcome bad news with good news

It’s true that the Internet has a long memory. But it’s even truer that the more recent the news, the higher it will rank in the search results.

So ensure that there’s steady communications coming from our company about the good stuff you’re doing.

Don’t go overboard to try to get the old bad stuff taken down. If a polite request doesn’t result in the post’s removal or correction, further attempts are just likely to backfire by drawing more attention to the original problem, or sparking a new round of negative discussions.

Over time, old posts sink in the search results, websites get taken down, and people forget things.

For example, a company recently wrote me to thank me for a good article after all the trouble they had with me in the past. I didn’t remember what trouble they were referring to — but now I’m curious!

Don’t keep bringing up bad stuff. Keep your communications positive.

For example, the Great Canadian Grid talks about its DDOS protection measures without mentioning the problems they’ve had in the past.

7. Consider rebranding

In 99.9 percent of cases, fixing a problem and learning from it will be better for your company than shutting everything down and starting over from scratch.

You can even have multiple bankruptcies and thousands of lawsuits against you and still run for president.

But sometimes things happen that you just can’t recover from. For example, your company might be named “Isis” or “Santorum.”

It’s never easy to change your name. Not only do you have to go around and fix all the links, change all your social media accounts and get new business cards printed up, but you also have to start over from scratch with brand awareness.

Plus, if you do it often, you’ll get a reputation as someone who leaves a trail of failed businesses in their wake and the people who’ll most want to work with will start avoiding you.

Maria Korolov

Maria Korolov is a science fiction writer who covers cybersecurity, AI and extended reality as a tech journalist at her day job.
Check out her author page on Amazon or follow her on Twitter, Facebook, or LinkedIn. Her first virtual world novella, Krim Times, made the Amazon best-seller list in its category. Her second novella, The Lost King of Krim, is out now.