Author Topic: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE  (Read 31716 times)

0 Members and 1 Guest are viewing this topic.

Online mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #125 on: September 28, 2021, 01:40:39 pm »
I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit.  I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.
Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 21609
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #126 on: September 28, 2021, 07:38:42 pm »
Between carbonization and electroplating, a substantial short circuit could be formed.  Remember this is DC, so ionic transport is a thing, and ions can be plated off one electrode onto the other, usually in a haphazard way (in a foamy deposit due to excess hydrogen production, or in dendritic crystals without an agent to encourage smooth deposits).  There shouldn't be much electrolyte available, at least not initially, but that just means it'll take longer and have higher resistance; and cell voltage is in no short supply here.

The coolant leak could've been among the battery itself, which logically should occupy the bulk of internal volume regardless of where the controls/inverter are placed, and obviously has to have lots of metallic connections, not necessarily exposed connections, but would they take the time to seal everything up when it's already supposed to be nicely contained in pipes, and the enclosure(s) already handle rainwater?

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 6927
  • Country: ca
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #127 on: September 28, 2021, 08:00:33 pm »
So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.

The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?

In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.


"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."

So we don't why the leak existed and how it could not happen again  :palm:
 

Online wraper

  • Supporter
  • ****
  • Posts: 16802
  • Country: lv
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #128 on: September 28, 2021, 08:39:34 pm »
So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.

The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?

In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.


"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."

So we don't why the leak existed and how it could not happen again  :palm:
You've seen the word "FIRMWARE", completely ignored the rest and made a huge sensation out of it  :palm:. Firmware change mentioned was just another thing among several measures to improve safety. Not to say watch Dave's video or read the actual article by NSV rather than watching CNBS.

Quote
The following actions have been put in place to prevent a recurrence of this incident.
• Each Megapack cooling system is to be fully functionally and pressure tested when installed on site and
before it is put into service
• Each Megapack cooling system in its entirety is to be physically inspected for leaks after it has been
functionally and pressure tested on site
• The SCADA system has been modified such that it now ‘maps’ in one hour and this is to be verified
before power flow is enabled to ensure real-time data is available to operators
• A new ‘battery module isolation loss’ alarm has been added to the firmware; this modification also
automatically removes the battery module from service until the alarm is investigated
• Changes have been made to the procedure for the usage of the key lock for Megapacks during
commissioning and operation to ensure the telemetry system is operational
• The high voltage controller (HVC) that operates the pyrotechnic fuse remains in service when the key lock
is isolated
Quote
Conclusion
The incident was most likely initiated by a Megapack coolant leak. The absence of a number of monitoring
and protection systems that would have been available had the initial Megapack not been subsequently
switched to off-line service mode allowed the initial fault to go undetected and resulted in the total loss of two
Megapacks
« Last Edit: September 28, 2021, 08:49:56 pm by wraper »
 

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 6927
  • Country: ca
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #129 on: September 28, 2021, 09:11:17 pm »
Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.

Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26757
  • Country: nl
    • NCT Developments
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #130 on: September 29, 2021, 08:08:07 am »
Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.

Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?
Well, that is why these battery units are designed to spit flames upwards and not to the side. In the end there is only so much you can do before a design gets very expensive. OTOH it is likely that the design goes through some further iterations but that is normal for any product.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline somlioy

  • Regular Contributor
  • *
  • Posts: 128
  • Country: no
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #131 on: September 29, 2021, 12:38:49 pm »
Hah, the same thing happened on a norwegian ferry with a 2MWh battery pack, where the battery was taken out of service as well, so the crew didnt get any warning on the control system before the thermal runaway was a fact.

A gasket in the cooling system was leaking causing a short circuit. In addition an explosion happened due to the seawater from the sprinkler system.

https://www.ctif.org/news/explosion-norwegian-battery-hybrid-ferry-may-have-been-caused-fire-extinguishing-system
 

Offline trobbins

  • Frequent Contributor
  • **
  • Posts: 755
  • Country: au
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #132 on: September 29, 2021, 12:58:30 pm »
somlioy, that link does not identify 'the same thing happened'.  Certainly there was a similarity that alarms did not annunciate, but for different reasons.
 

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 6927
  • Country: ca
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #133 on: September 29, 2021, 07:20:10 pm »
Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve

It's terrible there is so much pressure, perhaps too much on new renewables technology. The politics for a stable grid before summer hits.
Tesla is shaping things, not in a great way. I don't like the disposable/non-repairable Megapacks and the "consumer grade" design they rolled out. No firewall, too late to increase cabinet spacings and no thought of a coolant leak occurring in the design. These fires will be happening again.

1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 21609
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #134 on: September 29, 2021, 09:10:36 pm »
1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.

It wouldn't be too far out of place, I suppose -- recall what their first cars were like.  The products themselves had a number of issues, like panel alignment; everything was thrown together by hand, quality control was poor.  (They still do, with things like placement of the 12V battery being a microcosm of anything else I'm sure.)  Production was even more fraught, with frequent delays and poor utilization of automation systems.  The "throw everything at a wall and see what sticks" and "continuous integration" mentality of the software/startup crowd (and owner) was blamed for this.

Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market.  Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline trobbins

  • Frequent Contributor
  • **
  • Posts: 755
  • Country: au
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #135 on: September 29, 2021, 11:39:00 pm »
Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve

It's terrible there is so much pressure, perhaps too much on new renewables technology.

I think there are two sad aspects of the recent windfarm legal battle.  Firstly, the HV distribution tower failures that initiated the wide area blackout shouldn't have happened - but the designers of those towers don't appear to have been brought in to public scrutiny.  Secondly, the windfarm disconnect scheme had obviously been tested and approved and given the authority to connect, but that aspect also appears to not have been brought in to public scrutiny, only the aspect that the wind farm operators should have programmed the disconnect scheme to endure more than six separate trip events before actually disconnecting.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37664
  • Country: au
    • EEVblog
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #136 on: September 30, 2021, 01:42:17 am »
I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit.  I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.
Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length

I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.

 

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #137 on: September 30, 2021, 04:03:54 am »
Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve
FYI official press release here: https://www.aer.gov.au/news-release/hornsdale-in-court-for-inability-to-provide-contingency-services-as-offered

As per the above official statement, the issue is related to power not being supplied as stipulated due to a power failure of Kogan Creek Power Station (Coal fired power plant in Queensland). That's over 1 500 km (930 miles) away. I saw a comment else where saying the issue seemed more due to network capability and distance rather than Neoen.

Edit:
Similar lawsuits seem pretty common from AER checking their press releases here: https://www.aer.gov.au/news?f%5B0%5D=type%3Aaccc_aer_news Most of these are unreported in the media however. Probably just getting extra attention due to the usual Tesla/Musk perversion.

Issue might be inaccurate modelling of system capability over the Australian distribution network causing them to over bid and under deliver?
Edit2: See Trobbins' comment below.

« Last Edit: September 30, 2021, 04:56:43 am by sandalcandal »
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #138 on: September 30, 2021, 04:04:39 am »
I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.
Metal cover holds the radiator system and maybe pumps I presume.
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #139 on: September 30, 2021, 04:05:56 am »
Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market.  Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.
Yeah, automotive and aerospace markets are so lax in comparison :-DD
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline trobbins

  • Frequent Contributor
  • **
  • Posts: 755
  • Country: au
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #140 on: September 30, 2021, 04:52:29 am »
Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision.  I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC.  I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support.  For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).
 
The following users thanked this post: sandalcandal

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #141 on: September 30, 2021, 04:57:49 am »
Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision.  I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC.  I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support.  For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).
Yeah, sounds reasonable. I'm not too familiar with grid scale systems.

Given it has gone to court and Neoen isn't just paying a fine would that indicate there is some contention (or Neoen thinks it can otherwise reduce its penalty)?
« Last Edit: September 30, 2021, 05:00:12 am by sandalcandal »
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline askji432d

  • Newbie
  • Posts: 1
  • Country: cn
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #142 on: September 30, 2021, 05:54:14 am »
the radiator was inside metal cover.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37664
  • Country: au
    • EEVblog
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #143 on: September 30, 2021, 05:55:49 am »
I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.
Metal cover holds the radiator system and maybe pumps I presume.

It's moot anyway because that's the older PowerPack. The Megapack seems to be an entirely different design.
 
The following users thanked this post: sandalcandal

Offline JohnG

  • Frequent Contributor
  • **
  • Posts: 567
  • Country: us
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #144 on: September 30, 2021, 02:16:42 pm »
engineers are used to very relaxed schedules --
  :o

"Reality is that which, when you quit believing in it, doesn't go away." Philip K. Dick (RIP).
 

Offline thinkfat

  • Supporter
  • ****
  • Posts: 2150
  • Country: de
  • This is just a hobby I spend too much time on.
    • Matthias' Hackerstübchen
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #145 on: September 30, 2021, 02:33:58 pm »
Stack them.
Everybody likes gadgets. Until they try to make them.
 

Offline f4eru

  • Super Contributor
  • ***
  • Posts: 1086
  • Country: 00
    • Chargehanger
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #146 on: September 30, 2021, 02:54:57 pm »
Burn, Baby, Burn :)

Offline Marco

  • Super Contributor
  • ***
  • Posts: 6694
  • Country: nl
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #147 on: September 30, 2021, 03:00:50 pm »
Who would want to be the general contractor doing site design and install? It's too risky.

Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.

If it was Tesla's fault, they can sue Tesla in return.
« Last Edit: September 30, 2021, 03:06:10 pm by Marco »
 

Online wraper

  • Supporter
  • ****
  • Posts: 16802
  • Country: lv
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #148 on: September 30, 2021, 03:16:22 pm »
Who would want to be the general contractor doing site design and install? It's too risky.

Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.

If it was Tesla's fault, they can sue Tesla in return.
If you are consevative, you end up with stagnation and outdated infrastructure which is falling apart due to its age.
 

Offline TimFox

  • Super Contributor
  • ***
  • Posts: 7934
  • Country: us
  • Retired, now restoring antique test equipment
Re: Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE
« Reply #149 on: September 30, 2021, 04:29:24 pm »
Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf