| General > General Technical Chat |
| The BIG EEVblog Server Fire |
| << < (19/36) > >> |
| bd139:
Interesting thread. Also after looking at the photos and Google maps, an observation: Don’t go with a facility with no crash wall, no fence and parks a trailer probably with a nice propane bottle actually in the boundary of the building. If they do that then there’s probably 9000 even worse things inside the building you can’t see which are waiting to take your business out. May be better to leverage an IaaS cloud here. The whole availability zone concept makes these events into non events. If a facility burns then you lose an availability zone, not your entire deployment. I made this observation originally after mistakenly hosting half a rack of shit at a red neck provider here. Edit: also always host your email somewhere completely different as you might find it difficult to contact support if your email server is on fire (did that once) |
| CatalinaWOW:
--- Quote from: james_s on April 08, 2021, 06:15:58 pm --- --- Quote from: Bud on April 08, 2021, 06:10:25 pm ---I am questioning the details of the culprit, that is - what happened to the backup generatir, did they buy it on Alibaba? --- End quote --- Generators fail. Even well known name brand stuff like Caterpillar, Detroit, etc blow up now and then, especially older ones which it was mentioned this was. These are big diesel engines, they require maintenance and occasionally stuff breaks. If it spins a bearing, or has an injector problem, or something like a bad seal in a turbocharger can cause the engine to run away consuming its own lubricating oil until it throws a rod through the side of the block and spills oil and/or fuel all over the hot exhaust system and then you have a fire. --- End quote --- Failure is expected. Failure with catastrophic results is what is surprising. Aerospace uses something called FMECA, (Failure Modes, Effects and Criticallity Analysis) to try to prevent the latter. A spun bearing, or low oil or .... It isn't perfect. Things still slip through the cracks, but apparently we don't care that much about data loss, or respirator failure or anything else that depends on backup generators. |
| ve7xen:
--- Quote from: james_s on April 08, 2021, 08:56:23 pm --- "do we REALLY need to invest $$$ in rebuilding or replacing the genset or can we defer that to next year?" It happens, and it's much easier to predict and point fingers with the benefit of hindsight. If this place had a history of generators failing I would be more critical, but a single catastrophic failure is much too small of a sample to judge by. It could have been a total fluke where something just blew up due to a manufacturing defect that was never caught, or it could be it wasn't properly maintained, or it could be somebody monkeyed with it at some point, we don't know. --- End quote --- And like you say, sometimes you still get bitten. We're pretty good about maintaining our generators, every 6 months a guy comes from the service company and does all the scheduled maintenance items according to the manufacturers' recommendations, we do monthly full-load tests, and so on. A few years back we suffered a ~12h outage anyway. The genset fired up and took the load, no problem, and a few hours later, we got the overheat alarm and it shut itself down. Go to the bunker and find that one of the coolant hoses has burst and it's pumped all of its many litres of coolant onto the floor. We later learned that while the maintenance guy had been there two weeks before, and a 5-year replacement of the hoses was due, he happened to not have that one particular hose in his truck that day, and we allowed to be deferred to the next maintenance. Murphy gets you every time. |
| EEVblog:
I just got one months credit. So they implemented the 30 day limit clause. |
| bdunham7:
--- Quote from: EEVblog on April 09, 2021, 12:37:07 am ---I just got one months credit. So they implemented the 30 day limit clause. --- End quote --- At least they didn't invoke force majeure and weasel out entirely! |
| Navigation |
| Message Index |
| Next page |
| Previous page |