General > General Technical Chat
The BIG EEVblog Server Fire
YurkshireLad:
--- Quote from: Gyro on April 08, 2021, 11:50:50 am ---
--- Quote from: YurkshireLad on April 08, 2021, 11:38:03 am ---My question is, are you staying with them or moving to another service provider? :)
--- End quote ---
That's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fire yet! ;)
--- End quote ---
True, or do you stick with a provider that had a fire or move to one that hasn't. ;D
It is a business after all.
Ian.M:
--- Quote from: YurkshireLad on April 08, 2021, 11:38:03 am ---My question is, are you staying with them or moving to another service provider? :)
--- End quote ---
Better the devil you know. At least you can be fairly certain they'll put measures in place to prevent or reduce the impact of similar future incidents, and have some idea of their actual disaster recovery capabilities.
It would be nice if WebNX added a large NAS with its own power management in a shipping container at both their locations, with high bandwidth connectivity into their data centers. Its only a bit over 600 miles as the crow flies between their locations, (729 miles by road, est. 10.5H driving time), so if they had another extended outage, the whole NAS could be shipped to their other location to facilitate getting fall-back servers online without the bandwidth implications of either uploading from off-site backups, or live syncing over a significant geographical distance. They'd also need a priority standby contract with a local shipping company at each end to provide a self-loading container truck and two drivers with N hours notice 24/7.
wilfred:
--- Quote from: tautech on April 08, 2021, 04:56:41 am ---For prosperity....including the bad spelling:
--- End quote ---
Also for posterity.
wilfred:
--- Quote from: YurkshireLad on April 08, 2021, 11:56:47 am ---
--- Quote from: Gyro on April 08, 2021, 11:50:50 am ---
--- Quote from: YurkshireLad on April 08, 2021, 11:38:03 am ---My question is, are you staying with them or moving to another service provider? :)
--- End quote ---
That's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fire yet! ;)
--- End quote ---
True, or do you stick with a provider that had a fire or move to one that hasn't. ;D
It is a business after all.
--- End quote ---
The real question is were they prepared as they claimed they would be and did they recover as they promised they would. They can only control how they plan and respond. No-one knows which disaster scenario will play out. It could have been a Texas like power outage and the trucks delivering diesel for the generators may not have been able to get through to resupply.
I do think Dave could (should?) make preparations to restart from offsite backups. From what I read there was only 10% of servers water affected. How long would it have taken to recover from a total loss of all the site. I can't imagine sufficient servers would be readily available.
cdev:
They put their backup server in LA?
The city thats been cinematically annihilated more times than any other!
Why one only needs to look at maps from not so long ago to realize that California is likely to tear itself off of the mainand and strike out on its own at it's earliest opportunity..
--- Quote from: EEVblog on April 08, 2021, 03:41:53 am ---I figured this event needed it's own thread, so moved it from the servere reports thread.
HUGE thanks to gnif for handling this:
https://hostfission.com/
--- End quote ---
Fission? Degraded state?
--- Quote from: EEVblog on April 08, 2021, 03:41:53 am ---The server was down from 2021-04-04 21:13 UTC to 2021-04-08 03:36 UTC
It's currently still operating in a degraded state, and performance is surrently impacted until the caches catch up.
Gorillaservers upgraded the server box (maybe the old box was water damaged?) from Dual Xeon 2620V2 from the older dual L5630
Presumably they'll upgrade the other redundant box too to match, but the 2nd box is not currently online yet.
The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!
We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.
--- End quote ---
The city of LA is not a secure location. It's as insecure a location as there is in this country.
They suppress all news to the contrary, as a favor to the real estate industry.
Ive been told by a relative who was in a position to know.
They have lots of tornadoes (and waterspouts, of salty water, which Ive seen between the city and Santa Cataline Island. )
And substantial numbers of earthquakes.
I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.
[/quote]
Its truly horrible how much we depend on easily broken computers and unreliable networks.
What if (insert unspeakable tragedy here) ? Huh?
what if.. We need more resilience and redundancy..
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version