OSgrid has begun the rebuilding process, grid administrators announced today, even as the latest attempt to recover data starts to approach the finish line.
Around 3 terabytes of data has been recovered as of this morning, out of a total of 3.5 terabytes — or about 21 million files. On November 10, the recovery effort stood at 1.9 terabytes. That’s a rate of .12 terabytes a day, meaning that it will probably take at least four days to finish the recovery process.
But rebuilding has begun anyway, so that the grid will be up soon — with, or without, its assets.
“Today we’ll be reprovisioning the old server with new disks and getting a second, identical server, to prepare as a ‘hot and ready standby’,” said James Stallings in a Facebook post yesterday. “This will leave us with two servers always up to date and ready to handle service requests.”
The team had contacted the data recovery services company on Monday to find out the status of the third — and last — recovery effort. There was no response.
“Not precisely encouraging,” said Stallings, who is also known as Hiro Protagonist in-world.
But they aren’t writing off the recovery efforts just yet, he said. It will take a little while to set up the servers and get the grid back up.
“We’ll give those guys a little more time while we get ready,” he said.
Then, this morning, he reported via OSgrid’s Internet Relay Chat channel that the recovery team was approaching the 3 terabyte mark.
“I got new disks in the raid array yesterday and set up the server operating system,” said Stallings, also known as LeTwitch on IRC, according to a repost of the chat on the OSgrid forums.”Doing some configuration today.”
OSgrid secretary Lawrence Roberts, who is also known as Albertlr Landar in-world. confirmed that the recovery efforts will continue until its clear that they will not succeed in an OSgrid forum post yesterday.
“But in the meantime, instead of just waiting for the results, it was decided to proceed with getting replacement servers and set up a new better backup system for either case, so that when a final determination is made there will be no further delay,” he said. “Regardless of which way things go the new equipment will be needed for us to go forward.”
Grid won’t be up for at least another week
Stallings did not specify how long it would take for OSgrid to be back up, except to say, “it won’t be tomorrow or even next week.”
“But we are no longer dead in the water,” he added. “Once we have this server live and workable via remote access, we’ll order the hot and ready standby server.”
According to Stallings, the new equipment will improve the grid’s technology infrastructure significantly — he called it “OSgrid 2.0.”
“We are going to get a little progressive, and try to prevent this sort of thing from happening again in 18 months,” he said.