Author Topic: The BIG EEVblog Server Fire  (Read 19577 times)

0 Members and 1 Guest are viewing this topic.

Online tautech

  • Super Contributor
  • ***
  • Posts: 28136
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: The BIG EEVblog Server Fire
« Reply #25 on: April 08, 2021, 05:39:33 am »
Better drop this here too:
Avid Rabid Hobbyist
Siglent Youtube channel: https://www.youtube.com/@SiglentVideo/videos
 

Offline hendorog

  • Super Contributor
  • ***
  • Posts: 1617
  • Country: nz
Re: The BIG EEVblog Server Fire
« Reply #26 on: April 08, 2021, 05:42:41 am »
I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.

I'm actually surprised to learn that you weren't using Google Workspace or Office 365 Dave. For the sake of $8-9/month per user, you can have all of the Google services, redundancy, spam filtering and 30-something email aliases. I haven't run my own mail server for decades and it's a bit of a thing of the past.

We used the Google suite for a while at my job, absolutely hated it since it was all crippled browser based stuff, they don't even have a proper desktop email client. When we were acquired we went back to Microsoft, which while I'm not the biggest fan of Microsoft, their Outlook for email and calendar blows the doors off of Google's clunky offerings. Google Docs is nice for shared documents but the lack of a desktop version really kills it for most other uses.

For email, yeah, I wouldn't bother hosting my own server, but for the client side, no way, I don't rent software, and I absolutely hate browser based productivity applications. They are a total pain in the ass and never offer the same functionality of a desktop application.

Pretty much 100% the opposite from me. Goes to show no one solution will please everyone.

Been whoring myself to google full time now for 10 years. Became permanent after I found out that the cat was sneakily pissing on my home servers. That shortened the life of all concerned.
 
The following users thanked this post: mathsquid

Offline EEVblog

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: The BIG EEVblog Server Fire
« Reply #27 on: April 08, 2021, 05:45:46 am »
I'm actually surprised to learn that you weren't using Google Workspace or Office 365 Dave. For the sake of $8-9/month per user, you can have all of the Google services, redundancy, spam filtering and 30-something email aliases. I haven't run my own mail server for decades and it's a bit of a thing of the past.

I have Office 365, but only use it for Word/Excel etc. No idea it had other email server related stuff.
 

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: The BIG EEVblog Server Fire
« Reply #28 on: April 08, 2021, 05:47:50 am »
Pretty much 100% the opposite from me. Goes to show no one solution will please everyone.

Been whoring myself to google full time now for 10 years. Became permanent after I found out that the cat was sneakily pissing on my home servers. That shortened the life of all concerned.

Server is one thing, but desktop productivity software? The browser based stuff is a joke, it doesn't even work at all if your internet goes down, and in my case I was constantly logged out every time I cleared my browser cache which is something I do often at work. Since the calendar was browser based it couldn't integrate into the OS and provide meeting reminders so I kept missing meetings, total pain in the ass. Not to mention Google is an advertising company, their products are almost all trojan horses in a sense, their function is to entice you into giving them your valuable personal information. I use them for email and nothing else, if I log into a browser I ALWAYS make sure I log out immediately when I'm done.
 
The following users thanked this post: peter-h, Jacon

Offline Halcyon

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: The BIG EEVblog Server Fire
« Reply #29 on: April 08, 2021, 06:30:29 am »
I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.

I'm actually surprised to learn that you weren't using Google Workspace or Office 365 Dave. For the sake of $8-9/month per user, you can have all of the Google services, redundancy, spam filtering and 30-something email aliases. I haven't run my own mail server for decades and it's a bit of a thing of the past.

That still seems like a lot of money.

I guess it depends on the individual but I use Google Workspace for my own private email on my own domain. It costs me AUD$8.32 per month and I have various email addresses for things, e.g,:
payments@ - Links to my bank account (in Australia you can transfer money instantly between bank accounts using email addresses/phone numbers), also used for Paypal
forums@ - Forum subscriptions
uber@ - Uber
sonos@ - Sonos

Email aliases are essentially "disposable" and I can create/delete them on a whim without having my private mailbox spammed because someone/a service has my personal email address.
« Last Edit: April 08, 2021, 06:32:10 am by Halcyon »
 

Offline TomS_

  • Frequent Contributor
  • **
  • Posts: 834
  • Country: gb
Re: The BIG EEVblog Server Fire
« Reply #30 on: April 08, 2021, 06:35:03 am »
I thought datacenters typically used Halon fire suppression systems?
I don't think anyone uses Halon any more, it would be something like FM200 these days. And yes, this would usually be installed in the server rooms.

You would usually get something like 30 seconds from alarms and strobes activating to evacuate before it is dumped into the room because yes, it's whole purpose is to displace oxygen to starve a fire.

But if the fire was caused by one of the generators, it's anyone's guess whether they installed the same fire suppression in that area. Presumably not if it got this bad!?

But there seems to be a missing piece of the puzzle (or at least I've missed it)... They say the fire was caused by a generator. The generators would not normally be installed in the same room as servers. So how does a generator fire end up leading to water damage to servers?? Unless it burned through a wall??

OVH also suffered a major fire recently. Interesting trend of events.
 

Offline Ranayna

  • Frequent Contributor
  • **
  • Posts: 856
  • Country: de
Re: The BIG EEVblog Server Fire
« Reply #31 on: April 08, 2021, 06:56:28 am »
Yay, EEVBlog is back :)

Interestingly the second datacenter fire that affected services that i use in only a couple of months. According to statistics the services i use should be save now for some time :p

Anyway, always interesting to see what can happen. Makes me glad that my employer has definitly not skimped of the equipment and setup of our internal datacenter.
Our internal datacenter is quite small, we only have 26 Racks. But it is set up very well: redundant cooling, redundant power (one leg UPS and generator protected). All the critical components are in separated rooms with at least 2 firewalls (the brick kind ;) ) separating them.
The room with the racks and the UPS room are equipped with a Novec fire suppression system. The generator room, on the other hand, is not. It was explained that it was, first, not really needed, since the generator has its own room, the fuel has its own room, and the datacenter itself is two rooms away. And secondly, misfires of the required early detection system might trigger on the slightest leaks of the generator. And refilling the Novec would supposedly cost around 20.000 Euro :D

This kind of setup is expensive, but at least for us, it seems to be worth it: We had not a single outage beyond faults of individual servers. Yet...  :-BROKE
 

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: The BIG EEVblog Server Fire
« Reply #32 on: April 08, 2021, 07:18:29 am »
I don't think anyone uses Halon any more, it would be something like FM200 these days. And yes, this would usually be installed in the server rooms.

You would usually get something like 30 seconds from alarms and strobes activating to evacuate before it is dumped into the room because yes, it's whole purpose is to displace oxygen to starve a fire.

But if the fire was caused by one of the generators, it's anyone's guess whether they installed the same fire suppression in that area. Presumably not if it got this bad!?

But there seems to be a missing piece of the puzzle (or at least I've missed it)... They say the fire was caused by a generator. The generators would not normally be installed in the same room as servers. So how does a generator fire end up leading to water damage to servers?? Unless it burned through a wall??

OVH also suffered a major fire recently. Interesting trend of events.

Ah, yeah FM200 was what we had at that place, I just assumed it was a brand name for a Halon system, I never really looked into it.

Yes I wondered about that too, I would have thought they'd have the generators in a separate structure, but who knows, maybe the datacenter is in a highrise or something. I actually spent a night in Ogden UT once but it was close to 15 years ago and I don't think I ever went into town.
 

Online peter-h

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: The BIG EEVblog Server Fire
« Reply #33 on: April 08, 2021, 08:03:43 am »
Well done on EEVBLOG for having kept good backups.

Not many people do that.

Also a lot of "hits" on forums are malicious, perhaps organised by people who got banned for behaving badly.
« Last Edit: April 08, 2021, 08:07:16 am by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline tszaboo

  • Super Contributor
  • ***
  • Posts: 7307
  • Country: nl
  • Current job: ATEX product design
Re: The BIG EEVblog Server Fire
« Reply #34 on: April 08, 2021, 08:08:41 am »
The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!
We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.
I was taught by the back-end people that your second server shall be on another continent, with a different government in charge of the company running it.
 

Offline Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: The BIG EEVblog Server Fire
« Reply #35 on: April 08, 2021, 08:51:07 am »
The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!
We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.
I was taught by the back-end people that your second server shall be on another continent, with a different government in charge of the company running it.

It might have sounded prudent some while ago but these days, with numerous jurisdictions in each other's pockets, I don't think it matters anymore.
iratus parum formica
 

Online peter-h

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: The BIG EEVblog Server Fire
« Reply #36 on: April 08, 2021, 09:06:11 am »
Also a "mirror" server won't protect you from a clever attack, where only older data is deleted, perhaps over time.

One has to do several things at the same time, including a database snapshot every day or so and keep the snapshots for many months.

Hacking is a big thing these days - much bigger than servers blowing up.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline Towger

  • Super Contributor
  • ***
  • Posts: 1645
  • Country: ie
Re: The BIG EEVblog Server Fire
« Reply #37 on: April 08, 2021, 09:23:33 am »
But there seems to be a missing piece of the puzzle (or at least I've missed it)... They say the fire was caused by a generator. The generators would not normally be installed in the same room as servers. So how does a generator fire end up leading to water damage to servers?? Unless it burned through a wall??

From the sounds of it, I would not be surprised if the generators were on the roof.  Normally I would have expected the generators to be in a separate building.  But I have heard of similar issues with mission critical UPS systems failing.  If you have a fire and call you call any local fire department, they will want all power turned off.  They don't care about your business model etc, minimising risk to their firefighters is more important.  The same goes if you call out the lifeboat, their job is to save life, saving the vessel is of secondary importance.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: The BIG EEVblog Server Fire
« Reply #38 on: April 08, 2021, 09:30:08 am »
Yes I wondered about that too, I would have thought they'd have the generators in a separate structure, but who knows, maybe the datacenter is in a highrise or something. I actually spent a night in Ogden UT once but it was close to 15 years ago and I don't think I ever went into town.

it's here:
https://www.google.com/maps/place/119+600+W+Bldg+3B,+Ogden,+UT+84404,+USA/@41.2631939,-111.994884,17z/data=!3m1!4b1!4m5!3m4!1s0x87530c31eeede3af:0xe0843e2258f1e68c!8m2!3d41.2631939!4d-111.9926953

 
The following users thanked this post: james_s

Offline EEVblog

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: The BIG EEVblog Server Fire
« Reply #39 on: April 08, 2021, 09:31:39 am »
Well done on EEVBLOG for having kept good backups.

Didn't need the backups in the end though.

Quote
Also a lot of "hits" on forums are malicious, perhaps organised by people who got banned for behaving badly.

It's happened.
 

Offline Towger

  • Super Contributor
  • ***
  • Posts: 1645
  • Country: ie
Re: The BIG EEVblog Server Fire
« Reply #40 on: April 08, 2021, 09:33:07 am »
Also a "mirror" server won't protect you from a clever attack, where only older data is deleted, perhaps over time.

Royal Bank of Scotland learned that the hard way, it took years to properly sort out: https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system_problems
In their case it was incompetence. They could have pulled the plug on the main system before it was automatically replicated on the backup system, a couple of hours later.

 

Offline MadScientist

  • Frequent Contributor
  • **
  • Posts: 439
  • Country: 00
Re: The BIG EEVblog Server Fire
« Reply #41 on: April 08, 2021, 09:49:25 am »
I’d be really interested to know what happened and see a layout of the building

Genset fires are not uncommon , ( IC engines etc) and typically are positioned well away from the devices they powered, airgapped by 100s of feet , walls , separate buildings etc. Gensets should be able to burn to the ground and not damage their associated systems. Not to mention comprehensive fire suppression

Someone skimped here at gorilla methinks
« Last Edit: April 08, 2021, 09:52:28 am by MadScientist »
EE's: We use silicon to make things  smaller!
 

Offline Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: The BIG EEVblog Server Fire
« Reply #42 on: April 08, 2021, 09:53:59 am »
I’d be really interested to know what happened and see a layout of the building

Genset fires are not uncommon , ( IC engines etc) and typically are positioned well away from the devices they powered, airgapped by 100s of feet , walls , separate buildings etc. Gensets should be able to burn to the ground and not damage their associated systems. Not to mention comprehensive fire suppression

Someone skimped here at gorilla methinks

My money's on switchboxes. They really can and do indeed release magic black smoke.
iratus parum formica
 

Offline Ian.M

  • Super Contributor
  • ***
  • Posts: 12807
Re: The BIG EEVblog Server Fire
« Reply #43 on: April 08, 2021, 09:59:10 am »
EEVblog got an honorable mention in the article about the incident at 'The Register': https://www.theregister.com/2021/04/06/webnx_data_fire/

The best unofficial live discussion thread about the incident is at: https://www.webhostingtalk.com/showthread.php?t=1842301
It has a lot of stuff that WebNX and GorillaServers were either slow to, or did not, publicly release.
 
The following users thanked this post: duckduck

Offline MadScientist

  • Frequent Contributor
  • **
  • Posts: 439
  • Country: 00
Re: The BIG EEVblog Server Fire
« Reply #44 on: April 08, 2021, 10:01:18 am »
I’d be really interested to know what happened and see a layout of the building

Genset fires are not uncommon , ( IC engines etc) and typically are positioned well away from the devices they powered, airgapped by 100s of feet , walls , separate buildings etc. Gensets should be able to burn to the ground and not damage their associated systems. Not to mention comprehensive fire suppression

Someone skimped here at gorilla methinks

My money's on switchboxes. They really can and do indeed release magic black smoke.

Hmm, normally the changeover systems are in the genset space not the supported devices space, the switchgear there would be in use daily using the grid supplies etc and wouldn’t or shouldn’t fail like it did. ( if it did ) , sometimes the changeover system burns out , but again this should be significantly airgapped 
EE's: We use silicon to make things  smaller!
 

Offline Ultrapurple

  • Super Contributor
  • ***
  • Posts: 1027
  • Country: gb
  • Just zis guy, you know?
    • Therm-App Users on Flickr
Re: The BIG EEVblog Server Fire
« Reply #45 on: April 08, 2021, 10:06:35 am »
Whatever the cause of the fire and resultant outage, congratulations to all concerned for getting EEVblog (and presumably many other systems) back up and running - even at mildly reduced capacity - so quickly.
Rubber bands bridge the gap between WD40 and duct tape.
 
The following users thanked this post: Towger

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 19280
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: The BIG EEVblog Server Fire
« Reply #46 on: April 08, 2021, 10:18:16 am »
The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!

You're not the first to notice that, and won't be the last.

The Brown's Ferry fire very nearly caused two nuke reactors to meltdown. It was caused by a candle :)

Talking about the control circuits (my emphasis):
Quote
Speaking of tape recorders, there was one really interesting phone conversation between J. R. Calhoun, the chef of TVA’s Nuclear Generation Branch at the time and Frank Long of the NRC (and reported by a Canadian website):
    Calhoun: Yah, you know everything for those two units comes through that one room. It’s common to both units, just like the control room is common to both units.

https://hackaday.com/2018/12/06/fail-of-the-week-1975-the-browns-ferry-nuclear-incident/
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 19280
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: The BIG EEVblog Server Fire
« Reply #47 on: April 08, 2021, 10:24:42 am »
One has to do several things at the same time, including a database snapshot every day or so and keep the snapshots for many months.

... and test the backups.

Old story: backups to tape drive completed successfully, but when needed the tapes were found to be blank. The tape head was electrically connected but hanging loose away from the tapes!
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 
The following users thanked this post: tooki

Offline Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: The BIG EEVblog Server Fire
« Reply #48 on: April 08, 2021, 10:34:46 am »


My money's on switchboxes. They really can and do indeed release magic black smoke.

Hmm, normally the changeover systems are in the genset space not the supported devices space, the switchgear there would be in use daily using the grid supplies etc and wouldn’t or shouldn’t fail like it did. ( if it did ) , sometimes the changeover system burns out , but again this should be significantly airgapped

Don't underestimate the power of fire protection systems and the fire department.

A plant near me recently had the control room catch fire. Such a mess. The panel where the fire started was vaporized. The adjacent panels had the wiring behind get so hot that all that is left looks like the wiring in a cheap mattress. The auto fire suppression system went nuts and sprayed yellow/orange gunk all over other panels not near the fire. And the boys that were sent to fix it all now need counseling, because of 60+? years of control mods that nobody kept track of.

Management: "But will it be all working again by this time next week?"

 :popcorn:
iratus parum formica
 

Offline MadScientist

  • Frequent Contributor
  • **
  • Posts: 439
  • Country: 00
Re: The BIG EEVblog Server Fire
« Reply #49 on: April 08, 2021, 10:35:35 am »
All their reports say they had a mechanical genset  failure resulting in a fire. So this isn’t a switch gear fault I suspect. The issue seems to be the proximity  of servers to the Gensets. It sounds like they were in the same space  :palm:
« Last Edit: April 08, 2021, 10:37:35 am by MadScientist »
EE's: We use silicon to make things  smaller!
 
The following users thanked this post: Ed.Kloonk


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf