The BIG EEVblog Server Fire

[jh15]

Just put the servers in rice.

A few days before this, I heard to put something in rice on the NCIS tv show, a rerun. Now I know where most people got it from.

NO!

Best part is about 13 mins in.

[EEVblog]

@eevblog : Welcome back

The pictures Dave posted look like a random light industrial park of the sort of place my friend's machine shop is in, there's even a sports bar & grill in one of the units. Looks like the genset has to be in the same space, there isn't anywhere else to put it.

I agree. I was wondering how water ended up on the racks. And there I was thinking the backup genny was in a shipping container some 100 feet from the main building. Just in case it caught fire. Nar, that is extra rental space.

Maybe not odd in Utah, but certainly odd looking in the Uk, the power control boxes, transformer and what I assume is the deisel fuelling point, are not caged in collision resistant fencing - or even a crash carrier. All it would take is a truck driver choking on a hotdog from the grill... and back to square one.

Morale of the story from both WebNX and OVH is, backup power systems are highly flammable! Just let the power go off and rebuild the filesystems, not the entire data center.

I can confirm that the generator was only meters away from some of the racks in the same room. And that no servers were damaged due to the fire, it was all water.

[madires]

Diesel generator and racks in the same room? That's insane for many reasons!

[SilverSolder]

Diesel generator and racks in the same room? That's insane for many reasons!

It does sound a bit how'ya doin'...

[bd139]

I think that transcends that and into “what the hell were they thinking?” territory....

Suitable ORA book attached 

[tggzzz]

The main issue in North America is the "pioneering spirit" electrical system where wires are strung up among the trees in rural / suburban areas...   what could possibly go wrong? 

So, completely the opposite of this example, 4 miles from the centre of a major UK city

[Ed.Kloonk]

The main issue in North America is the "pioneering spirit" electrical system where wires are strung up among the trees in rural / suburban areas...   what could possibly go wrong? 

So, completely the opposite of this example, 4 miles from the centre of a major UK city

We are world famous here for our y-shaped gum trees on the side of the road.

https://www.smh.com.au/national/nsw/ausgrid-accused-of-street-tree-vandalism-by-sydney-councils-20150807-giugg9.html

[calzap]

I’m curious as to how many data center standby generators are powered by diesel versus propane.  In the U.S., diesel is usually more expensive per kw-hr produced than propane, especially because EPA requires such generators to use ultralow sulfur diesel fuel.  Diesel has shelf-life issues which require ongoing testing and maintenance, and replacement in some cases.  Propane has an infinite shelf-life.  Most building codes in the U.S. do not allow liquid propane inside buildings.  So, if there is no outside storage area, propane can’t be used.

I know of a biological lab near me that became so fed-up with diesel fuel issues with their generators that they switched to propane and have been happy with it.

Nice link:
https://www.csemag.com/articles/understanding-backup-power-system-fuel-choices/

At my ranch, we have a small (8 kw continuous) backup generator.  It’s just big enough to run one well plus a few refrigerators/freezers and a few low wattage items.  Forced to run it 4-6 times per year.  Longest run has been 2 days.  It’s powered by gasoline, and there have been no fuel issues (we use a preservative).  It’s in a “dog house” attached to the smallest building on the property.  Only real worry is that building is where electrical panels for incoming power are located.  In retrospect, I could have built the dog house 10 m away from any building and should have.  We have propane available, and dual-fuel installation is on my to-do list … has been for 15 years!

Mike in California

[David Hess]

I’m curious as to how many data center standby generators are powered by diesel versus propane.  In the U.S., diesel is usually more expensive per kw-hr produced than propane, especially because EPA requires such generators to use ultralow sulfur diesel fuel.  Diesel has shelf-life issues which require ongoing testing and maintenance, and replacement in some cases.  Propane has an infinite shelf-life.  Most building codes in the U.S. do not allow liquid propane inside buildings.  So, if there is no outside storage area, propane can’t be used.

Diesel also has the disadvantage of requiring special containment provisions because it makes a hell of a mess if it spills, which also brings the EPA into it.  For this reason and the others you mention, sites which are remote tend toward propane now.

For tall buildings, diesel may not be stored at the upper levels so provisions must be made to pump it from tanks near the ground.

In some cases requirements are so strict that backup generators of any type are precluded and the only solution is sufficient battery power.

[David Hess]

Also the control chassis is a mess with f*ing LM723 being abused as OPAMP, and no chip is decoupled properly.

I do not really consider that an abuse, except of course for the lack of proper decoupling.  723s do not make very good operational amplifiers but most applications do not require good operational amplifiers; consider all of the older regulator designs which only use discrete differential pairs for the voltage and current control loops.  723s also have several virtues including a built in reference and provisions for a high current output.

In short, the old design uses old parts, which is no longer made. So the maintenance had to look for alternative parts, and they found an Indian company making supposedly identical old parts for special customers. It turns out the Indians made better parts, faster and more stronger. Different process and die, same paper spec.

That is a common problem even when the same manufacturer changes processes.  Unspecified characteristics are unspecified and cannot be relied on.  Either test for them or design to handle them.

If not the NRC being dissatisfied and contracted a friend of mine to fix it who contracted me for a small portion of the project, I wouldn't believe how flaky something used as a last line of defense from a nuclear disaster can be.

...

And no, the NRC will not take a new control board or new 21st century parts. They insist on all digital control parts must be nuclear certified, and all critical power parts must too. The only leeway we have were non power analog parts, in other words we had to patch on an old stupid design with the oldest technology.

Non-power analog parts are actually more susceptible to radiation damage than power analog parts, but as long as the NRC follows the rules no matter how stupid, they are safe.

It is stupid but I am not surprised.  The "safest" option when authority is divorced from responsibility is to do nothing, and make sure the blame will fall on someone else, which is why having nothing to do with that sort of project is the best thing to do.  I have learned not to even notify them; they are not interested and doing so can create further jeopardy.

[madires]

Diesel also has the disadvantage of requiring special containment provisions because it makes a hell of a mess if it spills, which also brings the EPA into it.  For this reason and the others you mention, sites which are remote tend toward propane now.

So propane doesn't need any special containment provisions? Diesel is easy to handle, to refill and to get hold of. Anyhow, you simply use the fuel which is easily available and allowed by local regulations. If you go for a battery system only you need deep pockets, because it has to provide several MW for a few days.

[Tom45]

At a place I worked 50 years ago we had backup power for two large mainframes. The mainframes were fed by motor generator sets so switchover timing wasn't too critical. The top floor of the building was filled with lead acid batteries to hold over until the diesel generator got going.

The generator was buried in a concrete pit under the parking lot with a diesel tank above ground.

The whole thing worked well until one stormy period when power was so flaky they decided to just run continuously off the generator until the weather calmed down. No problems for a couple of days until someone forgot to order more diesel. Oops.

No, it wasn't me.

[calzap]

Diesel also has the disadvantage of requiring special containment provisions because it makes a hell of a mess if it spills, which also brings the EPA into it.  For this reason and the others you mention, sites which are remote tend toward propane now.

So propane doesn't need any special containment provisions? Diesel is easy to handle, to refill and to get hold of. Anyhow, you simply use the fuel which is easily available and allowed by local regulations. If you go for a battery system only you need deep pockets, because it has to provide several MW for a few days.

Any fuel stored onsite obviously must have a primary container.  In the U.S., propane tanks containing liquid propane must be outdoors.  There are usually building setback requirements as well.  So, if there is a leak, it wafts away as a gas.  Large diesel tanks in most jurisdictions must have secondary containment in case of a leak.  If indoors, there are usually fire sprinkler or other suppression and protection requirements as well.

Mike in California

[David Hess]

Diesel also has the disadvantage of requiring special containment provisions because it makes a hell of a mess if it spills, which also brings the EPA into it.  For this reason and the others you mention, sites which are remote tend toward propane now.

So propane doesn't need any special containment provisions? Diesel is easy to handle, to refill and to get hold of. Anyhow, you simply use the fuel which is easily available and allowed by local regulations. If you go for a battery system only you need deep pockets, because it has to provide several MW for a few days.

Propane requires a pressure tank but is actually *safer* if there is a leak or fire.  A diesel leak makes a hell of a mess which is where the EPA gets involved.  A propane leaks leaves nothing to clean up.

Propane tanks handle fire just fine.  When the pressure relief valve activates, the propane exhaust is burned and evaporation cools the tank until the propane is exhausted.  Just make sure that the exhaust is directed in a safe direction.  I know of one case at a mountaintop repeater site where the exhaust was directed at the blockhouse.  When personal showed up to find out why all of the repeaters had failed after a brush fire, they found that the blockhouse was completely destroyed (melted) by the jet of burning propane.

[james_s]

I’m curious as to how many data center standby generators are powered by diesel versus propane.  In the U.S., diesel is usually more expensive per kw-hr produced than propane, especially because EPA requires such generators to use ultralow sulfur diesel fuel.  Diesel has shelf-life issues which require ongoing testing and maintenance, and replacement in some cases.  Propane has an infinite shelf-life.  Most building codes in the U.S. do not allow liquid propane inside buildings.  So, if there is no outside storage area, propane can’t be used.

I've seen natural gas fired backup generators which have the obvious advantage of fuel being piped in rather than stored on the premises. Propane, that I have not seen other than for small portable generators and some that are used in RVs where you already have propane available. I think diesel pretty much owns the large backup generator market, the engines are the same as used for things like semi trucks, motor yachts and locomotives. I don't think anybody is making huge spark ignition engines anymore although there have been some really big ones in the past.

[SilverSolder]

The cost of diesel probably doesn't actually matter, since the generators are not used that often.

[madires]

Propane requires a pressure tank but is actually *safer* if there is a leak or fire.  A diesel leak makes a hell of a mess which is where the EPA gets involved.  A propane leaks leaves nothing to clean up.

I'd think that propane is more prone to a nice BOOM than diesel. BTW, it's also heavier than air. And don't worry too much about the storage of diesel. Storage regulations for oil and oil based fuels take care about that.

[TimFox]

In the recent Texas cold-weather catastrophe, the natural gas delivery failed due to lack of power at the wellhead compressors and frozen distribution pipes.

[Ultrapurple]

You don't seem to have been here long

I invite you to check my profile info.

[bdunham7]

I don't think anybody is making huge spark ignition engines anymore although there have been some really big ones in the past.

Waukesha Engine is still around, under different ownership.

https://www.innio.com/en/products/waukesha

[calzap]

I've seen natural gas fired backup generators which have the obvious advantage of fuel being piped in rather than stored on the premises. Propane, that I have not seen other than for small portable generators and some that are used in RVs where you already have propane available. I think diesel pretty much owns the large backup generator market, the engines are the same as used for things like semi trucks, motor yachts and locomotives. I don't think anybody is making huge spark ignition engines anymore although there have been some really big ones in the past.

I agree diesel-powered industrial generators are probably the most common now.  However, I think that will change in favor of natural gas and propane where they are readily available.  Energy costs, emissions limits and fuel stability will bring it about.

Natural gas has the advantage of not having to store it on site.  In fact, storing it on site, as CNG or LNG is an expensive proposition.  However, as recent events in Texas have shown, natural disasters can stop the flow of natural gas.  That low temperatures did it is primarily a reflection of inappropriate penny-pinching in engineering and building pumping stations.  Even with appropriate design and installation, natural disasters can interrupt flow … like in an earthquake area where I live.

Large industrial standby generators powered by propane or natural gas are already available.  For example, Generac sells 150 kW propane generators and 500 kW dual fuel (natural gas/ diesel).  And no, separate engines aren’t required for natural gas/diesel gensets.  They are diesel engines modified to aspirate an air/gas mixture, which provides most of the energy.  Small diesel injections provide ignition, but the engines can run on diesel alone if necessary.  Generac’s largest natural gas generator is 1 MW and powered by a 12-cylinder, 49 L spark-ignited engine.

Mike in California

[tautech]

Gas powered gensets are nothing special or new. Not far from me is a bank of 1MW gensets running on methane from a landfill.
https://www.terracat.co.nz/power-systems/new-power-systems/epg/gas-generator-sets

[james_s]

I'd think that propane is more prone to a nice BOOM than diesel. BTW, it's also heavier than air. And don't worry too much about the storage of diesel. Storage regulations for oil and oil based fuels take care about that.

That's definitely true. You can throw a lighted match into a bucket of diesel fuel and it will go out. If you do manage to light the stuff, it burns pretty lethargically, it's similar to kerosene or salad oil as far as flammability. Propane on the other hand can be dangerous stuff. Unlike many fuels, it doesn't need a stoichiometric mixture to burn explosively, indeed a small engine will run pretty well if you just poke the end of a non-lit propane torch into the air intake, it's not like gasoline where the mixture has to be just right, propane will still go bang in very rich or very lean conditions.

[floobydust]

I looked at google street view and only one exhaust pipe for a generator, near the electrical room.
The facility appears to be in some old warehouse (military?) district with brick exterior walls and a wooden roof? If true that's a problem.

Had to laugh, not a solar panel in sight.

[NiHaoMike]

How much can the backup power infrastructure be cut back if there's a system to force all CPUs to minimum frequency when running on backup power?