The BIG EEVblog Server Fire

[madires]

I thought datacenters typically used Halon fire suppression systems? The last place I worked that had an onsite datacenter had one, there were warning strobes to indicate the system had discharged and asphyxiation warning signs.

Halon was banned, but old fire suppression systems may still use it. And there are a few alternatives, i.e. FM-200. Anyhow, you'll also find classic water based fire suppression systems in many data centers. Some of them create a misty spray.

[YurkshireLad]

My question is, are you staying with them or moving to another service provider? 

[Gyro]

My question is, are you staying with them or moving to another service provider? 

That's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fire yet! 

[coppice]

Halon was banned, but old fire suppression systems may still use it. And there are a few alternatives, i.e. FM-200. Anyhow, you'll also find classic water based fire suppression systems in many data centers. Some of them create a misty spray.

In the early 80s the company I worked for kept changing its insurers each year. These alternated between pro-halon insurers and pro-water insurers. We had halon and sprinkler systems alternately installed and ripped out of the computer room I oversaw for several years.

[coppice]

My question is, are you staying with them or moving to another service provider? 

Why abandon one of the hottest operations in the business? Are you concerned their service may be cooling off now?

[YurkshireLad]

My question is, are you staying with them or moving to another service provider? 

That's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fire yet! 

True, or do you stick with a provider that had a fire or move to one that hasn't. 

It is a business after all.

[Ian.M]

My question is, are you staying with them or moving to another service provider? 

Better the devil you know.   At least you can be fairly certain they'll put measures in place to prevent or reduce the impact of similar future incidents, and have some idea of their actual disaster recovery capabilities.

It would be nice if WebNX added a large NAS with its own power management in a shipping container at both their locations, with high bandwidth connectivity into their data centers.   Its only a bit over 600 miles as the crow flies between their locations, (729 miles by road, est. 10.5H driving time), so if they had another extended outage, the whole NAS could be shipped to their other location to facilitate getting fall-back servers online without the bandwidth implications of either uploading from off-site backups, or live syncing over a significant geographical distance.  They'd also need a priority standby contract with a local shipping company at each end to provide a self-loading container truck and two drivers with N hours notice 24/7.

[wilfred]

For prosperity....including the bad spelling:

Also for posterity.

[wilfred]

My question is, are you staying with them or moving to another service provider? 

That's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fire yet! 

True, or do you stick with a provider that had a fire or move to one that hasn't. 

It is a business after all.

The real question is were they prepared as they claimed they would be and did they recover as they promised they would. They can only control how they plan and respond. No-one knows which disaster scenario will play out. It could have been a Texas like power outage and the trucks delivering diesel for the generators may not have been able to get through to resupply.

I do think Dave could (should?) make preparations to restart from offsite backups. From what I read there was only 10% of servers water affected. How long would it have taken to recover from a total loss of all the site. I can't imagine sufficient servers would be readily available.

[cdev]

They put their backup server in LA?

The city thats been cinematically annihilated more times than any other!

 Why one only needs to look at maps from not so long ago to realize that California is likely to tear itself off of the mainand and strike out on its own at it's earliest opportunity..

I figured this event needed it's own thread, so moved it from the servere reports thread.
HUGE thanks to gnif for handling this:
https://hostfission.com/

Fission? Degraded state?

The server was down from 2021-04-04 21:13 UTC to 2021-04-08 03:36 UTC

It's currently still operating in a degraded state, and performance is surrently impacted until the caches catch up.
Gorillaservers upgraded the server box (maybe the old box was water damaged?) from Dual Xeon 2620V2 from the older dual L5630

Presumably they'll upgrade the other redundant box too to match, but the 2nd box is not currently online yet.

The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!
We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.

The city of LA is not a secure location. It's as insecure a location as there is in this country.

They suppress all news to the contrary, as a favor to the real estate industry.
Ive been told by a relative who was in a position to know.

They have lots of tornadoes (and waterspouts, of salty water, which Ive seen between the city and Santa Cataline Island. )
And substantial numbers of earthquakes.

I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.
[/quote]

Its truly horrible how much we depend on easily broken computers and unreliable networks.

What if (insert unspeakable tragedy here) ? Huh?

what if.. We need more resilience and redundancy..

[madires]

It would be nice if WebNX added a large NAS with its own power management in a shipping container at both their locations, with high bandwidth connectivity into their data centers.   Its only a bit over 600 miles as the crow flies between their locations, (729 miles by road, est. 10.5H driving time), so if they had another extended outage, the whole NAS could be shipped to their other location to facilitate getting fall-back servers online without the bandwidth implications of either uploading from off-site backups, or live syncing over a significant geographical distance.  They'd also need a priority standby contract with a local shipping company at each end to provide a self-loading container truck and two drivers with N hours notice 24/7.

First off, SAN, not NAS. Secondly, such an off-site backup requires fat pipes anyway for read and write, e.g. 100 Gbps Ethernet. So moving the SAN around won't speed up things. Actually it would cause a delay by the transport and also would add the risk of being damaged during transport, by a traffic accident for example.

[Ultrapurple]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

[Microdoser]

This has made me consider my own data safety and the importance of having an off-site backup of at leat the most critical things.

[Ultrapurple]

I have in front of me a 1TB microSD card, 15 x 11 x 1mm thick (fascinating hi-res X-Rays here). That works out at roughly 6GB per cubic millimetre.

Assuming we have a VW Passat (which claims 1780 litres of cargo space with the rear seats down), our modern equivalent of the station-wagon-full of tapes is potentially something like 1780 x (100 x 100 x 100) x 6GB, or around 10 exabytes (10x10^9 GB).

And yes I know we wouldn't get that capacity, they're slow to write to, and so on and so on, but it does make one think.

[Ian.M]

FedEx Bandwidth: https://what-if.xkcd.com/31/

[madires]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

Or the old fashioned sneakernet (https://en.wikipedia.org/wiki/Sneakernet)

Anyhow, do your risk assessment!

[Ultrapurple]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

Or the old fashioned sneakernet (https://en.wikipedia.org/wiki/Sneakernet)

Yup, that was the source of my Tanenbaum quote.

[xmo]

Day three without the forum....

TEA user:

"I have discovered something interesting."

I seem to be living with a woman!

Apparently, she is my wife.

She seems nice."

[tooki]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

Yes, or of actual messenger pigeons!

With that said, the critical downside of the station wagon or pigeon is the latency!

[Microdoser]

I have noticed the site, as we were told, seems a bit clunky and slow right now.

Still, gives me more time to fix up the I7 6700K 24GB Ram 256GB SSD that I found in the street fully working (except for the graphics card). It has a legitimate Windows 7 Ultimate OEM on it which I am currently getting Windows updates up to date in preparation to upgrade to Win 10 Pro

[Microdoser]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

Yes, or of actual messenger pigeons!

With that said, the critical downside of the station wagon or pigeon is the latency!

Or the time someone put a CD on the back of a snail and sent it across a table and got better data transmission rates than using the LAN (at the time)

[Ultrapurple]

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

— Andrew S Tanenbaum, 1989

Yes, or of actual messenger pigeons!

With that said, the critical downside of the station wagon or pigeon is the latency!

I'm sorry I'm dragging this so far off-topic; I promise to sit on the Naughty Step for a while and Think About What I've Done. But meanwhile, have a ponder about the effective latency in even, say, a thousand parallel, error-free, get-the-label-speed 10Gbps direct links for that amount of data. How many miles apart do your data centres have to be before it's quicker to send the data via wires than wheels?

And don't get me started on the effective data rate of a cargo plane worth of 6GB/mm³...

OK, I've arrived at the Naughty Step. Sitting in 3... 2... 1...

[madires]

Yes, or of actual messenger pigeons!

With that said, the critical downside of the station wagon or pigeon is the latency!

... or a Tesla with autopilot turned on, or Mrs. Eagle looking for a nice meal to feed her children. If that was your only backup you'll have problem.

[Hamster]

New Article : https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/

Webhosting talk forum: https://www.webhostingtalk.com/showthread.php?t=1842301

---

I would abort continuing to host anything at this data center, here in Florida, most all data centers do NOT use water.. they use an FM200 Fire Suppression System, also, Generators are not co-located with the data center, they are usually detached just in case something like this happens, Generators due tend to go, and when the go, they go violently..  the worst part is, this data center probably had multiple gensets, covering the load, when one went down, it caused a cascade effect going down the line..  Fuel should of also been shut off and isolated, but it appears not?  There should of been 0 water damage inside the facilities, this clearly is some kind of failed engineering design of a building that probably should of not been designed to be a data center.

---

Another popular site ( pinside.com ) is still down.. they have recovered the data and restored it to a lower tiered server, but still down.. they have to "order' new hardware for them.

[madires]

And don't get me started on the effective data rate of a cargo plane worth of 6GB/mm³...

I'll take the bait. So your data center #1 has burned down, nothing left, no network connectivity, no servers, no power. But you have a backup in data center #2. To which servers do you want to restore the backup?