Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE

[wraper]

Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.

Except it's a huge waste of effort and money to keep it maintained for for any practical purpose https://nypost.com/2016/11/11/brooklyn-bridge-repairs-expected-to-cost-811m/ The only value it has is architectural and historical. Not to say it was actually a quite innovative and beat the records of it's time. Not conservative at all as it was an early example of a steel-wire suspension bridge. Innovative stuff may fail here and there due to not yet known issues which may become apparent later, but it's a path to rapid improvements, and better results overall. FWIW in other circumstances it would be more effective to simply replace it, as most of outdated infrastructure.

[Siwastaja]

Maybe very dumb solution to neighbouring racks catching fire:

Would some emergency spring loaded ejection system work? When overheating/fire is detecting, shove the module(s) outside of the rack? Possibly into some kind of concrete ditch in front of the system filled with sand/gravel?

I love such overengineered fantasies when the reality shows that they fail at even such trivial thing as just reporting overheating/fire!

[thinkfat]

Just launch the complete pack into orbit. I'm sure SpaceX can lend a hand.

[TimFox]

Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.

Except it's a huge waste of effort and money to keep it maintained for for any practical purpose https://nypost.com/2016/11/11/brooklyn-bridge-repairs-expected-to-cost-811m/ The only value it has is architectural and historical. Not to say it was actually a quite innovative and beat the records of it's time. Not conservative at all as it was an early example of a steel-wire suspension bridge. Innovative stuff may fail here and there due to not yet known issues which may become apparent later, but it's a path to rapid improvements, and better results overall. FWIW in other circumstances it would be more effective to simply replace it, as most of outdated infrastructure.

One example of the conservative design approach to the Brooklyn Bridge is that Roebling estimated the "fudge factor" needed for specifying the cables that would be required since he knew political operators would cheat on the procurement.  The non-conservative design approach that doomed the Quebec Bridge was that the engineer remained in New York during construction, and had calculated the stresses with "theoretical" values of the structure weight (from the original design drawings) and did not confirm the weight from the detailed fabrication drawings.  That, and the construction supervisor ignored reports that the structure was yielding and sent workers out on the bridge: they would be out on the bridge when it failed with substantial loss of life.
So, maintenance repairs to the Brooklyn Bridge are estimated at $811 M.  Did the article mention how much a replacement would cost?  The Oakland Bay Bridge recently cost $6.4 B to replace.

[wraper]

The Oakland Bay Bridge recently cost $6.4 B to replace.

Don't underestimate the ability of California to blow the money. FWIW they are building the slowest and the most expensive in the world high speed rail system. And the costs will probably triple over current $80B estimate by the time they finish it (if they ever do), considering their track record.

[TimFox]

And you think the State of New York will be any more cost-efficient?

[floobydust]

Had to laugh, the Hornsdale court case was hinging on the "right to not generate" electricity. Renewables are coming in too late, they are mission critical when they really shouldn't be if the Grid was well-managed, had sufficient generation and reserve capacity, fat cats actually forecasted demand increases and acted accordingly.


The location of the radiator, fan and coolant pump seems to be the main difference between PP and MP.
MP has the radiator and fan located at the top of the cabinet, whereas the PP has it in the door with the distinguishing door grille. The battery packs seem to need names?

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

Boeing 787 lithium batteries they added ceramic spacers to block fire?
The irony, Musk 2013 tweet "Maybe already under control, but Tesla & SpaceX are happy to help with the 787 lithium ion batteries."

[wraper]

Boeing 787 lithium batteries they added ceramic spacers to block fire?
The irony, Musk 2013 tweet "Maybe already under control, but Tesla & SpaceX are happy to help with the 787 lithium ion batteries."

Very funny, especially considering that fire did not start at batteries as it was in multiple 787, and where battery was extremely tiny in comparison.

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

You literally have no idea how they are actually built, yet act if you are an expert in the field.

[floobydust]

It's a battery pack, not a space ship. You can look at the relevant patents. Tesla didn't start from scratch, you can see the Powerpack design internals and my assumption is they just upsized to the larger cells with minor changes.
None of this matters because at the end of the day, the cabinet burned up with only guesses as to why.

From my armchair, the battery pack spacings are terrible with no fire breaks, and nothing for the firefighters to do but sit around and watch it burn out. Tell me how great this is and how it will never happen again.
 
Megapacks are being sold like hotcakes all over the globe, if the design went too far we'll know within a year.

[T3sl4co1l]

One example of the conservative design approach to the Brooklyn Bridge is that Roebling estimated the "fudge factor" needed for specifying the cables that would be required since he knew political operators would cheat on the procurement.

Don't underestimate the ability of California to blow the money. FWIW they are building the slowest and the most expensive in the world high speed rail system. And the costs will probably triple over current $80B estimate by the time they finish it (if they ever do), considering their track record.

As projects grow in size, and thus political visibility, never underestimate our* capacity to skim off a little more, here and there, for our friends/contractors/cronies.  This effect, probably even more so than the direct benefits (or arguable lack thereof, across various similar proposals), is a huge reason why such projects may never get built in this country.  It's a systemic issue, and addressing that first, would seem to be a prerequisite to actually building these things.

*US specifically, yes, but it happens everywhere, democracy or otherwise.  Tendering contracts among friends, is the democratic version of more overt corruption under dictatorships (ref: _Rules for Rulers_).  Different mechanism, and absolutely, at a different rate/amount -- but the same basic effect.  So, ultimately it would seem, by "our", I mean humans in general.  Or at least, any large-scale system humans have yet developed.  So, this isn't a political point, or at least any more so than concerns that relative amount.

Tim

[sandalcandal]

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

Modern power electronics, particularly for these high end systems goes >90% with >99% efficient AC-DC systems available. The datasheet posted at the start of this thread has 87% to 90% round trip efficiency for 0.5E to 0.25E loading so efficiency each way would be \$\sqrt{0.87}=0.933\$ so ~93.3% efficiency each way.

So far as thermal design needing to be completely redesigned between the two, both are rated for the 0.5E to 0.25E loading so I don't see how that's a requirement, even then the cooling system setup is clearly not the same. The Powerpack datasheet has round trip efficiency rated at a slightly lower 85.5% to 89.5%.

Not that any of that matters in this case since the fire had nothing to do with operational heat load and was apparently due to an uncontrolled failure with resultant heat from the failure.

The design, layout and density of the Tesla systems don't look that far off other system I've seen but I wanted double check so here are some other grid scale systems below.

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Attached below:
LG Chem Rack System R1000- UPB4860  https://greenpowerco.com.au/products/batteries/lg-chem/lg-chem-rack-system-r1000-upb4860-gen2/

Samsung SDI http://www.samsungsdi.com/upload/ess_brochure/201803_SamsungSDI%20ESS_EN.pdf

ABB Enviline https://new.abb.com/medium-voltage/switchgear/railway-switchgear/dc-traction-power-supply/energy-management/enviline-ess

Fluence (Siemens AG and The AES Corporation) Cube https://fluenceenergy.com/energy-storage-technology/gridstack-grid-energy-storage/ (There's also photos of a very large system floating around but I can't find what it actually is. Only official use is as a banner on their about page)

GE Reservoir Storage Unit https://twitter.com/generalelectric/status/1154428597895122944/photo/1

[sandalcandal]

1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.

Again as per the datasheet, the supported protocols are Modbus TCP / DNP3 / Rest API.

[trobbins]

The SCADA mapping issue could have been due to using default settings for aspects like how long to wait for a response, or a myriad other technical and comms issues that obviously hadn't been adequately identified and then ticked off the punchlist.  This could also easily have been using a different PLC/software combination than previously used, which would reset prior experiences.

There are always stressful installation and commissioning timelines and milestones, whatever the project, and whatever the key reasons, and sadly decisions are made to keep going with not enough non-Tesla high level experience to put hold's in place for certain aspects.  There may also have been a lack of awareness of past megapack issues, such as can only be gained from other site visits by non-Tesla people when those sites have their key people available and who are not constrained to be open.

[EEVblog]

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Only Megapack photo I can find is:



Which looks nothing like their wanky render:

[sandalcandal]

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Only Megapack photo I can find is:
Which looks nothing like their wanky render:

Yeah, no real "complete" photos of the innards.

[EEVblog]

Twitter comes though:
https://twitter.com/sillybaboy/status/1443823137678897164

Video:

https://globalnews.ca/video/rd/92b2b6fa-30ce-11ea-9e92-0242ac110004/?jwsource=cl

[sandalcandal]

Twitter comes though:
https://twitter.com/sillybaboy/status/1443823137678897164

Video:

https://globalnews.ca/video/rd/92b2b6fa-30ce-11ea-9e92-0242ac110004/?jwsource=cl

Great stuff. Can see coolant lines going off to some manifold on the left, RJ45 daisy chaining between modules and it looks like the control systems are powered by an external 24V bus running off to the right. If they are powered by an external supply rather than deriving power internally from each battery pack then it makes a bit more sense the monitoring systems for the whole thing could be disabled at a single point. I guess these could also potentially be inverter modules?

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

[jonovid]

from seeing Dave's video about this  Telsa Victoria Big Battery Fire REPORT

so it takes 12 to 24 hrs just to log a coolant system feature! I find that level of poor design of a remote location telemetry system just astonishing!
it just as well tesla is not building or running nuclear reactors.   

[gmb42]

from seeing Dave's video about this  Telsa Victoria Big Battery Fire REPORT

so it takes 12 to 24 hrs just to log a coolant system feature! I find that level of poor design of a remote location telemetry system just astonishing!
it just as well tesla is not building or running nuclear reactors.   

Are Tesla responsible for the SCADA?  I would assume that's someone else, the MegaPacks are the field devices.  The issue could still be those field devices being slow to respond, but from the other reports, it seems that the SCADA system isn't very performant.

Does anyone know what the SCADA software is?

[floobydust]

Look at volume of data in one SCADA scan. How many analogs and digitals should there be from a single battery module? Do a back of the napkin calculation.
V, I, temperature, inverter V, I this that, alarms, status etc. Say 20 points and avg. 12 bytes/point in packets, that's about 240 bytes of data.
Multiply by 16 battery modules per cabinet, 11 cabinets/MP row and 2 rows back-back, and the site has 40 of 150 MP's in the commissioning phase. I think 14,080 battery modules so far? Gees that's a lot.
My rough estimate is 3.2MB of data every poll of 281,600 points. If you are polling over serial at 9,600 that is an hour lol.
With my math, it will be almost 4X larger/slower when the site is finished, 1 million points...

For reliability you would not use a single network, there would be many. Large utility sites can have a high failure rate for RS-485 transceivers, one short from a ground-fault or lightning etc. and you lose comms. To limit the loss of visibility and control, there would be multiple networks.

DNP is an antique utility industry (substation) protocol from the days of dial-up. It's stuck around and I think it's a gross protocol as DNP3 due to the drama with standardization, that it's obfuscated and inefficient. IEC 61850 over Ethernet is the modern way to go. But this is the utility industry. MODBUS is bare-bones minimum and fine over TCP/IP.

How Neoen decided to go... you would never use a PLC or low-cost simple HMI system due to the volume of data. Serial is way too slow here.
Over a million points I estimate, which is the league of distributed control systems DCS. I don't know of any factory or plant with that many points to read and log and display on an HMI. It's never done on a single computer system, it would be broken up but this costs money.

I'd say they (Neoen) went too cheap and ended up with turtle bus. After the fire, they've speeded it up to a one hour scan time, seems terrible. You won't know there is a fire alarm coming in, faster to just look for smoke?

[gmb42]

For systems such as this you wouldn't get in the door with your Modbus ideas, polling can easily miss transitions which are somewhat common and important in the electrical monitoring world.  You need event buffering and reporting, i.e. what DNP3 was designed for.

Yes, DNP3 has not adapted the data frame from the old serial days, so the 2 byte CRC for every 16 bytes of data is useless overhead when run over TCP, but then Modbus TCP frames drop the Station address and the CRC and add in the transaction ID, protocol ID and length words to make a frame with more overhead than the serial equivalent.  Both protocols still retain their small data size per packet so the raw frame size is a toss up.

With DNP3 you poll for events, much more efficient than Modbus polling, and if you enable unsolicited mode on the devices you don't even need to do that, they report events of interest.

Practical real world experience has shown me that effective data update rates in large systems are way better with DNP3 than Modbus.  When using polling protocols such as Modbus, SCADA systems end up having to implement "fast scans" to increase the poll rate on devices of interest, i.e. when viewing the info for a particular substation (reducing it for other devices) to satisfy operator requirements.

DNP3, due to it's longevity, is pretty well understood and implemented, there are always exceptions (I'm looking at you, Rockwell ML-1400 p.o.s) and thus is still specified and implemented in the industries that suit it.  There now seems to be a move to use DNP3 in water\waste water systems as pump stations are becoming increasingly complex and data "rich", and Modbus and other polling protocols just don't cut it.

[floobydust]

In the pics of the Saint John Energy (NB) Megapack, I can see daisy-chained RJ45's so it's a multidrop network between the cabinet's battery modules, and who knows how far they took that, if there is a data concentrator somewhere taking Megapacks beyond sluggish serial CAN or RS-485, up to Ethernet.
Improving things from 12-24hr to 1hr poll is great but still pretty much a bandwidth disaster... and the site is 1/4 finished. The LG system advertises supporting CAN2.0 and MODBUS TCP/IP. It should be ok with Ethernet speeds.

I don't think anyone has thought it through, how much data is there. Substation controls, decisions are done locally i.e. in the relays and devices, with the SCADA/HMI "eyes only".
So no decisions or logic/controls are running at the top level. If there's an overcurrent, the breaker trips on its own. Of course there are trip/close, local/remote, auto/manual switches on the HMI for operators to actuate if needed.

If the SCADA/HMI went down, the site would keep running.

[EEVblog]

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

Given that the render doesn't match the other panels, I wouldn't put any stock in it.

[sandalcandal]

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

Given that the render doesn't match the other panels, I wouldn't put any stock in it.

Really? I think the new images/video confirm the renders are accurate.

We can clearly see that mostly empty cabinet with what appears to be the main isolator switch which looks pretty much exactly like the renders. Right next this cabinet in the video (in the same location as the renders), they also show another cabinet filled with rack modules without the "subpanel" covering that look near identical to the render. Then as per my last post we can also see the same panels covering the front of some rack modules. The only differences seem to be some of the wiring in the main switch panel is still incomplete and missing wires/coolant lines for the middle right rack cabinet shown.

[floobydust]

The site is apparently now up and running  https://www.centralwesterndaily.com.au/story/7543112/victorian-big-battery-switched-on/
Not sure when it's really needed and will actually do some work to prove itself.