General > General Technical Chat
The BIG EEVblog Server Fire
gnif:
After many days of stress, hair loss, and sleepless nights, we're back baby!
Please note that there may still be some disruptions over the next few days as we are still running in a degraded state.
xrunner:
Thank you. :)
CatalinaWOW:
I am sure all of us would like an after action report when you finally get back to some semblance of normalcy.
I am really curious about how use of backup generators caused a fire. Everything after that is just the dominoes falling, with an extra helping of bad luck for the EEVBlog servers.
EEVblog:
I figured this event needed it's own thread, so moved it from the servere reports thread.
HUGE thanks to gnif for handling this:
https://hostfission.com/
The server was down from 2021-04-04 21:13 UTC to 2021-04-08 03:36 UTC
It's currently still operating in a degraded state, and performance is surrently impacted until the caches catch up.
Gorillaservers upgraded the server box (maybe the old box was water damaged?) from Dual Xeon 2620V2 from the older dual L5630
Presumably they'll upgrade the other redundant box too to match, but the 2nd box is not currently online yet.
The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!
We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.
I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.
edpalmer42:
The standby power system caused a fire that shut down the data center with some servers expected to be offline for several weeks!! :palm: :palm:
https://www.gorillaservers.com/outage.html
You just can't make this stuff up!!
Navigation
[0] Message Index
[#] Next page
Go to full version