Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE

[EEVblog]

Australia's Biggest Tesla Battery Storage System at Moorabool is on FIRE! 
The 450MWh plant wasn't operational yet, it was still being tested.
https://www.abc.net.au/news/2021-07-30/tesla-battery-fire-moorabool-geelong/100337488
https://reneweconomy.com.au/tesla-megapack-container-on-fire-at-site-of-australias-biggest-battery/

[Ian.M]

Hmm.  You'd have thought that they'd have spaced the containers out a bit more to reduce the risk of fire spreading.   Its a green field site so the cost of say 50% more land would probably be a small part of the total budget . . .

It seems Tesla Megapacks don't come with built-in fire suppression!

Its already breaking news on select tech sites worldwide e.g. https://www.theregister.com/2021/07/30/tesla_battery_on_fire/

Edit: Tesla Megapack (summary) datasheet: https://impulsora.com/wp-content/uploads/2020/09/Ficha-Tecnica-Mega-Pack.pdf

[fourtytwo42]

I wonder how they are going to deal with all the toxic waste from that one, quietly bury it on the adjacent farmland ?

[Gyro]

"If we try and cool them down it just prolongs the process," the CFA's Assistant Chief Fire Officer Ian Beswicke said.

"But we could be here anywhere from 8 to 24 hours while we wait for it to burn down."

I guess the 'don't use water unless you have lots of water' electric car fire strategy just breaks down when you have a 13 tonne battery to contend with!

I'm a bit surprised they're not trying to cool down adjacent cabinets though. I'd expect them to suffer some thermal damage too, time will tell I guess.

[Berni]

Whops that must have been pretty expensive.

Tho i don't see why they wouldn't cover the other batteries with water. Sure it doesn't sounds like a good idea to put out the fire on this battery, its probably just gonna keep burning and its all toxic scrap now anyway. But giving the surrounding batteries a light shower of water sounds like it could go a long way in making sure they are still fine. For example the white front panels on the next battery across the row already turned black so they must be getting rather toasty. I'm sure a bit of water would go a long way for bringing the temperature of those panels down. The batteries are outside so they must survive rain anyway.

Perhaps it was more of a safety concern that none of the firefighters wanted wet mud around the burning battery becoming a shocking hazard. Don't think that would ever actually happen but firefighters are trained not to fuck with giant lithium batteries... for a good reason.

[Kleinstein]

The video footage shows some water spraying going on, though at a low level.

Tesla should have enough experiance with burned batteries, so they know how to handel the disposal of the 2 brunt containers.

[sandalcandal]

Reposting from the video thread

The suggestion of full on building brick walls between packs seems a bit excessive to me. From the photos shown the fire is only present in one bank. We'll see tomorrow morning if it spread.

The thing with many small cells being more likely to fail is intuitive but incorrect. The total capacity is of course dependent on electrode mass/area. If that capacity is achieved using many small units or fewer large units, the electrode area and chance of random manufacturing faults in the jelly roll is ~equal. By having the capacity split into small units however, failures can be better isolated and rejected in the manufacturing QC and also implementation isolating failures to single smaller cells.

Also as far as a single cell fault setting off a whole pack. IEC 62619 and UL 1973 as well as other standards for battery safety which is could fall under have "propagation testing" aka "Single Cell Failure Tolerance" if one cell blows up then the rest of the pack needs to remain safe. Reason being if there does happen to be a cell with a manufacturing fault or other random fault sneak in, it can't set the whole pack off.

From the above two you can hopefully see how small cell designs are generally considered safer.

As for what happened here, the Tesla battery pack designs are liquid cooled which should make them incredible tolerant to fire, as long as coolant is present (not even pumped, just sitting in and filling the coolant lines). Given this occurred during construction, my guess is that during commissioning there was some fault with coolant not being in the system whether that be insufficient or not at all filling the coolant or a coolant leak then during a commissioning load testing the pack with faulty coolant fill could have overheated and caught fire. The propagation testing only guard against limited individual cell failure, not a whole pack overheating.

Regardless, something went very wrong with the system/installation process.

Edit: From Ian M's datasheet


Edit2: https://www.greenburghny.com/DocumentCenter/View/7245/PB-20-18-Eagle-Energy---Tesla-Safety-Data-Sheet
Info on fire safety design. "All Tesla products undergo rigorous testing to standards such as UL 1973 and IEC 62619 that ensure the battery modules are resistant to single cell thermal runaway propagation."

[sandalcandal]

I wonder how they are going to deal with all the toxic waste from that one, quietly bury it on the adjacent farmland ?

It's a lithium ion based system not a NiCd or lead acid. The most toxic substance is the electrolyte which will be well burned off. The hazard in lithium ion battery disposal is batteries starting fires.
https://en.wikipedia.org/wiki/Environmental_impacts_of_lithium-ion_batteries

Edit: Also not sure what that "toxic smoke warning issued by authorities" is that's plastered on a few news sites. If it actually happened then it was taken down quick. Lithium ion battery fires just produce "normal" smoke. Official source: "There is currently no threat to the community though residents and motorists will notice smoke in the area. An Advice message has been issued." - https://www.frv.vic.gov.au/large-battery-fire-moorabool

[EEVblog]

Hmm.  You'd have thought that they'd have spaced the containers out a bit more to reduce the risk of fire spreading.

They are letting it burn itself out overnight, so will be interesting to see how many banks were hit. Looks like at least two, maybe 3.
This is will actually provide valuable data on fire spread in these types of systems.
Maybe they already have data on this and it's a basic cost/benefit trade-off?

[EEVblog]

Edit2: https://www.greenburghny.com/DocumentCenter/View/7245/PB-20-18-Eagle-Energy---Tesla-Safety-Data-Sheet
Info on fire safety design. "All Tesla products undergo rigorous testing to standards such as UL 1973 and IEC 62619 that ensure the battery modules are resistant to single cell thermal runaway propagation."

"Resistant" is not the same as guaranteed never to happen ever.
Standards and certification to them are not perfect.
It's all good to boast about that track record, until Murphy bites you on the arse and the incredibly unlikely thing happens.

I'm not saying it's a cell, but I wouldn't rule it out. It is afterall one of the primary, if not the primary cause of possible fire in such systems, that's why the standards exist in the first place to try and prevent it happening.

[EEVblog]

An early report in the comments is that this was its first operational testing charge, and had only been charging for a matter of hours before the fire, so 

[EEVblog]

I wonder how they are going to deal with all the toxic waste from that one, quietly bury it on the adjacent farmland ?

It's a lithium ion based system not a NiCd or lead acid. The most toxic substance is the electrolyte which will be well burned off. The hazard in lithium ion battery disposal is batteries starting fires.
https://en.wikipedia.org/wiki/Environmental_impacts_of_lithium-ion_batteries

Edit: Also not sure what that "toxic smoke warning issued by authorities" is that's plastered on a few news sites. If it actually happened then it was taken down quick. Lithium ion battery fires just produce "normal" smoke. Official source: "There is currently no threat to the community though residents and motorists will notice smoke in the area. An Advice message has been issued." - https://www.frv.vic.gov.au/large-battery-fire-moorabool

Sounds pretty toxic to me:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5577247/#:~:text=Lithium%2Dion%20battery%20fires%20generate,amounts%20of%20gas%20and%20smoke.&text=Fluoride%20gas%20emission%20can%20pose,large%20Li%2Dion%20battery%20packs.

How far that spreads, though, no idea.
I guess that's why they are taking it seriously

"A FRV HAZMAT appliance is on scene conducting atmospheric monitoring with a Scientific Officer in support. FRV’s specialist RPAS (drones) unit has also been deployed.

[EEVblog]

As for what happened here, the Tesla battery pack designs are liquid cooled which should make them incredible tolerant to fire, as long as coolant is present (not even pumped, just sitting in and filling the coolant lines). Given this occurred during construction, my guess is that during commissioning there was some fault with coolant not being in the system whether that be insufficient or not at all filling the coolant or a coolant leak then during a commissioning load testing the pack with faulty coolant fill could have overheated and caught fire. The propagation testing only guard against limited individual cell failure, not a whole pack overheating.

My reply on Youtube:

I would have presumed that the system would have a safety charging/discharge cutoff for a pack and/or cabinet in the event of a lack of coolant or coolant system failure, as the odds of something going wrong with a coolant pumping system would be orders of magntiude higher than cell failure. That would be bread-and-butter safety stuff in the design of such a system. So I'd be surprised if it's that.
What has caused the various Tesla battery car fires? There must be a lot of data on that by now?
Yes, will be very interesting to find out the actual cause.

[fourtytwo42]

Yes, will be very interesting to find out the actual cause.

I suspect unfortunately many well paid lawyers will ensure that we don't 

[sandalcandal]

Thanks for taking the time to read my responses and pinning my YouTube comment.

They are letting it burn itself out overnight, so will be interesting to see how many banks were hit. Looks like at least two, maybe 3.
This is will actually provide valuable data on fire spread in these types of systems.
Maybe they already have data on this and it's a basic cost/benefit trade-off?

IMO it should be either "safe" or a no go. The only cost/benefit trade-off is pay for it to be safe or not at all.

"Resistant" is not the same as guaranteed never to happen ever.
Standards and certification to them are not perfect.
It's all good to boast about that track record, until Murphy bites you on the arse and the incredibly unlikely thing happens.

I'm not saying it's a cell, but I wouldn't rule it out. It is afterall one of the primary, if not the primary cause of possible fire in such systems, that's why the standards exist in the first place to try and prevent it happening.

From my knowledge designing and testing for failure propagation, the main thing that can cause a compliant design to go non-compliant is elevated ambient temperature. If a whole pack is sitting toasty up at >90°C and one goes off then they're all going off. Given it's winter here right now I don't think elevated environmental ambient temperatures are the cause (16.5°C top in Moorabol today). Multiple cells being in close proximity and on the edge of failure such that a failure cluster with sufficient energy to set off the whole pack occurs seems also seems statistically pretty unlikely to me.

Even with cell failures, in liquid cooled systems it is extremely difficult to initiate any significant thermal runaway to begin with because even without coolant being pumped, once temperature hits 100°C the water based coolant boils off and hard limits the temperature rise. It generally (depending on construction and exact chemistry) takes temperatures around 120°C in a short amount of time to get cells to do something violent instead of safely venting. Having a sustained fault sufficient to deplete (boil off) coolant to the point where still volatile cells can get hot enough start significant and self-propagating fires seems also unlikely. If the coolant system was intact and filled I strongly believe there would be no fire, particularly of this scale. Perhaps some venting of a few overheated cells but that's it. Perhaps a really sustained failure event where safety monitoring systems failed to pick up a deviation and shut things down??

I would have presumed that the system would have a safety charging/discharge cutoff for a pack and/or cabinet in the event of a lack of coolant or coolant system failure, as the odds of something going wrong with a coolant pumping system would be orders of magntiude higher than cell failure. That would be bread-and-butter safety stuff in the design of such a system. So I'd be surprised if it's that.
What has caused the various Tesla battery car fires? There must be a lot of data on that by now?
Yes, will be very interesting to find out the actual cause.

You make a fair point about cut-offs for detecting coolant system failures but other than a coolant failure I don't really see how it could have gone up in flames... In the case of fiery Tesla car and battery pack failures I've looked at in the past, that all also appear related to some breach or failure of the coolant system or were otherwise exposed to a burning ICE car long enough to boil off all the coolant. My best guess is still that something went shitty with the coolant system and it wasn't caught. [Edit: I don't think actually seen the coolant monitoring systems on the Tesla packs, I mostly look at the electrical stuff and air cooled systems]

I'll ask some colleagues what they think next week. There are some other experienced lithium ion battery system engineers on this forum too so hopefully they can check what I'm saying.

I wonder how they are going to deal with all the toxic waste from that one, quietly bury it on the adjacent farmland ?

It's a lithium ion based system not a NiCd or lead acid. The most toxic substance is the electrolyte which will be well burned off. The hazard in lithium ion battery disposal is batteries starting fires.
https://en.wikipedia.org/wiki/Environmental_impacts_of_lithium-ion_batteries

Edit: Also not sure what that "toxic smoke warning issued by authorities" is that's plastered on a few news sites. If it actually happened then it was taken down quick. Lithium ion battery fires just produce "normal" smoke. Official source: "There is currently no threat to the community though residents and motorists will notice smoke in the area. An Advice message has been issued." - https://www.frv.vic.gov.au/large-battery-fire-moorabool

Sounds pretty toxic to me:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5577247/#:~:text=Lithium%2Dion%20battery%20fires%20generate,amounts%20of%20gas%20and%20smoke.&text=Fluoride%20gas%20emission%20can%20pose,large%20Li%2Dion%20battery%20packs.

How far that spreads, though, no idea.
I guess that's why they are taking it seriously

"A FRV HAZMAT appliance is on scene conducting atmospheric monitoring with a Scientific Officer in support. FRV’s specialist RPAS (drones) unit has also been deployed.

UL 1973 includes a clause for toxic emissions. As I mentioned before, the most toxic substance is the electrolyte which is generally fluorine based so can produce some toxic gasses when burnt. The amount is pretty miniscule though, particularly for an outdoor system so that's why it's generally not a problem. It can be a problem for large batteries in a confined space as that paper (and others) point out but I'd also be equally worried about general electronics smoke in that scenario. Not saying there are zero hazards and right by the emergency services to act with caution and get their own data.

Also the rest of the fire safety section you screenshoted:

They make design efforts to help prevent spread of fire from pack-to-pack as you would hope. I think it's possible we'll see only the one bank burned tomorrow (though more will need replacing for safety).

What could be a worry though is if this event turns out relatively inconsequential people will get too complacent with large battery systems. Not every system is designed like the Tesla Megapacks to such high levels of compliance. I've seen some real dodgy stuff out there. I hope it doesn't hurt the renewable industry sector but I also hope people don't get lax either.

[sandalcandal]

Yes, will be very interesting to find out the actual cause.

I suspect unfortunately many well paid lawyers will ensure that we don't 

Australia is a small country with a small renewables industry. I'm not involved in this project but I'm hoping I can hear something from a "friend of a friend" most likely from the fire department's reports on the incident.

[Marco]

The suggestion of full on building brick walls between packs seems a bit excessive to me.

I'd expect sprinklers though.

[sandalcandal]

The suggestion of full on building brick walls between packs seems a bit excessive to me.

I'd expect sprinklers though.

They're literally filled with water (inside tubes). Under normal circumstances I don't see a sprinkler system being of much use, if a fire actually starts you aren't putting it out, just preventing it from spreading... to the other water filled battery packs.

[sandalcandal]

Just found out. The installation uses the Megapack which is a different product to the Powerpack used at Hornsdale.
https://cleantechnica.com/2020/10/05/tesla-megapack-powerpack-powerwall-battery-storage-prices/
https://en.wikipedia.org/wiki/Tesla_Megapack

Megapack:


Powerpack:

[Marco]

They're literally filled with water (inside tubes).

Running water has an infinite heat capacity, a watercooling loop does not.

[Gyro]

You'd have thought that it wasn't past human ingenuity to have an X-Y grid of rails running on the ground through the whole site. It would be an easy job then to just roll in a set of vertical fire resistant barriers around the failed unit.

Damn, I should have patented the idea!

[sandalcandal]

They're literally filled with water (inside tubes).

Running water has an infinite heat capacity, a watercooling loop does not.

From the perspective of a single cell failure, the volume of the watercooling loop is the same. As I said before, something must have gone horribly wrong with a large number of cells at once or the coolant system went. As bad as it can get, in the design I can see so far, there's not any significant benefit a sprinkler system would bring to reducing the damage experienced.

Maybe it could help a bit? Especially if you expect no one to be able to show up and attend to a major failure for a few hours?

[sandalcandal]

You'd have thought that it wasn't past human ingenuity to have an X-Y grid of rails running on the ground through the whole site. It would be an easy job then to just roll in a set of vertical fire resistant barriers around the failed unit.

Damn, I should have patented the idea!

Alright, good luck finding the people to roll the barriers into place or doing automatically with any reliably outdoors!

Probably cheaper and more reliable to just space them out more or or build stationary walls if they needed to, though the seems to be they don't think they need anything?

[Gyro]

You'd have thought that it wasn't past human ingenuity to have an X-Y grid of rails running on the ground through the whole site. It would be an easy job then to just roll in a set of vertical fire resistant barriers around the failed unit.

Damn, I should have patented the idea!

Alright, good luck finding the people to roll the barriers into place or doing automatically with any reliably outdoors!

Haha, I wasn't thinking about getting a bunch of guys with corks round their hats to just wander out and take a bunch probably 1 tonne rolling barriers for a walk.

Naturally you would use mechanical means, probably using pre-layed steel cables. The infrastructure for putting the barriers on the appropriate rails and locating them would be on the periphery of the array. You're talking probably a portable crane and a few winches.

[sandalcandal]

Naturally you would use mechanical means, probably using pre-layed steel cables. The infrastructure for putting the barriers on the appropriate rails and locating them would be on the periphery of the array. You're talking probably a portable crane and a few winches.

Dave was making it sound like the whole place was going up in flames but we've only seen a single "bank" on fire and I reckon and that's all there will be. Just one burnt out bank and another charred adjacent bank. They seemed to have it under control and the lack of further news reports or alarms being called would support that assumption. I could still be wrong  we'll see soon enough. I do hope Dave makes a follow up or edits corrections into his video somehow. Some of the stuff he was saying was pretty far off.

Edit: My main issue was with Dave telling the world this must be a single bad cell and small cells make a more dangerous system.

[nctnico]

They're literally filled with water (inside tubes).

Running water has an infinite heat capacity, a watercooling loop does not.

No. At some point steam will form a barrier between the walls and the water causing the cooling capacity of the water to drop to zero.

[schmitt trigger]

My concern here is that the news agencies, looking for a “hot” story to feed to a public fed up with Covid news, will blow up this event way out of proportion.

[floobydust]

I did see firefighters watering down adjacent banks to stop the fire from spreading, at the end of the 7News clip. It should fizzle out instead of growing...

Physical construction is like that of a data center, except outdoors and with no.... halon. Oops we're all side by side with no barriers.
I agree, it needs larger aisle ways for a robot fire fighter to go in there, and fire breaks.

News and stock market could care less about this.

[Gregg]

I’m sure Elon has spin experts working on official responses already.  But he could use this incident to show he really wants to be the good guy --- or not. 

[nctnico]

Hmm.  You'd have thought that they'd have spaced the containers out a bit more to reduce the risk of fire spreading.   Its a green field site so the cost of say 50% more land would probably be a small part of the total budget . . .

It is more likely farm land which is expensive. Remember having these batteries in a desert makes little sense because the consumers are far away and the temperatures are much higher. However, the closer you move to civilisation and land that is useful for farming, the more expensive the land will be. Another thing to factor in is that you'll need to pave and maintain (get rid of weeds) the area the batteries are on; a compact area is cheaper.

[EEVblog]

Just found out. The installation uses the Megapack which is a different product to the Powerpack used at Hornsdale.
https://cleantechnica.com/2020/10/05/tesla-megapack-powerpack-powerwall-battery-storage-prices/
https://en.wikipedia.org/wiki/Tesla_Megapack

Megapack:


Powerpack:

Yeah, they look like entirely different designs.
Are there any other Megapack installations?

[EEVblog]

I did see firefighters watering down adjacent banks to stop the fire from spreading, at the end of the 7News clip. It should fizzle out instead of growing...

Yeah, I wouldn't expect it to spread any further than the two banks and that thrid one that looked heat damaged.
Although you'd also have to be concerned for the state of the two banks next to it.
And this photo shows the upper half bank which sparked it looks to have burned out, and it's now the lower half bank that's on fire.
There were also strong winds, and luckily they were blowing away from the other two megapacks what looks like only 50cm away.

Have not seen an update yet though, they let it burn throughout the night.

EDIT: One story in the last hour say "Big Battery continues to burn nearly 24 hours after catching alight"

[sandalcandal]

Copying my response from the cesspit that is YouTube comments.

@EEVblog 10:50 "You see in all the news reports that is their main fear [...] it's gonna spread to the other packs. Not is that only a cost impact, don't want the whole place going up." You did say 17:22 "I'm sure they'll save most of it" It wasn't a part of my earlier more visible posts and I get you're talking about potential for spread and doing commensurate safety/site planning for fire safety but half the video was spent talking about the fire spreading and I felt, particularly after reading comments people were treating it as though the place has already got significant fire spread. Maybe not your intention but the resultant effect IMO.

Maybe a cooling system fail maybe is in some ways "worse" than a statistical cell failure but it is what it is [or turns out to be] and someone is gonna face the music. 16:20 "I think it's just one cell inside...completely come a gutsa" is not a great understanding of how these things are design, built AND tested. Spreading that misunderstanding isn't great either...

The toxic smoke thing also felt like repeating misreports too.

[sandalcandal]

Just found out. The installation uses the Megapack which is a different product to the Powerpack used at Hornsdale.
https://cleantechnica.com/2020/10/05/tesla-megapack-powerpack-powerwall-battery-storage-prices/
https://en.wikipedia.org/wiki/Tesla_Megapack

Megapack:
[img]

Powerpack:
[img]

Yeah, they look like entirely different designs.
Are there any other Megapack installations?

The Megapack picture is from this article here https://huddle.today/building-the-utility-of-the-future-with-the-tesla-megapack/ about an installation by Saint John Energy in Saint John, Canada.

Had a quick squiz because I was interested myself too.


Holes Bay in the South of England, up and running since mid-2020 apparently

Also same people doing another one in West Sussex, southern England. [ https://www.energy-storage.news/news/construction-begins-on-fotowatios-second-uk-project-using-tesla-megapack-ba ]

Also doing a first party install in Texas apparently?
https://electrek.co/2021/03/08/tesla-secretive-big-battery-project-texas/
https://www.bloomberg.com/news/features/2021-03-08/tesla-is-plugging-a-secret-mega-battery-into-the-texas-grid?sref=DWzi38c2

PG&E electric substation in Moss Landing, California. This is the biggest one apparently
https://www.tesmanian.com/blogs/tesmanian-blog/tesla-pg-e-megapack-182-5-mw-battery-storage-installation-makes-stunning-progress

Ventura County, California
https://electrek.co/2021/06/30/tesla-megapacks-power-on-battery-replacing-gas-peaker-plant-california/

Probably a few more. Electrek keeps some decent tabs on Megapack here: https://electrek.co/guides/tesla-megapack/

[EEVblog]

Naturally you would use mechanical means, probably using pre-layed steel cables. The infrastructure for putting the barriers on the appropriate rails and locating them would be on the periphery of the array. You're talking probably a portable crane and a few winches.

Dave was making it sound like the whole place was going up in flames

Bullshit I was. Anyone who thought that watching and listenting to the video is an utter moron.

[EEVblog]

Maybe a cooling system fail maybe is in some ways "worse" than a statistical cell failure but it is what it is [or turns out to be] and someone is gonna face the music. 16:20 "I think it's just one cell inside...completely come a gutsa" is not a great understanding of how these things are design, built AND tested. Spreading that misunderstanding isn't great either...

Dude get over it. You seen to have a hard on for proving every minute thing I say wrong.

The toxic smoke thing also felt like repeating misreports too.

That was the damn report at the time the video was made. And it IS potentially toxic, it's why they issued the alert which they later downgraded, and why they have expert air monitoring person on site. You are demonstrably wrong, the toxic report was an official alert issued by the fire department:
https://news.cfa.vic.gov.au/firefighters-battle-large-battery-fire-near-geelong

A Watch & Act Warning was issued at 12.53pm for toxic smoke for Batesford, Bell Post Hill, Lovely Banks, Moorabool residents, who were advised to close windows and doors, turn off heating and cooling systems, and bring pets indoors.

It is expected to be downgraded to an Advice message this evening.

FRV initially led the response to the incident, with CFA taking control of the incident at 3.30pm. The two agencies worked in support of one another throughout the incident, and were also supported by Victoria Police, Ambulance Victoria and the EPA.

A scientific officer was on scene conducting atmospheric monitoring, while FRV’s specialist RPAS (drones) unit was also deployed.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5577247/#:~:text=Lithium%2Dion%20battery%20fires%20generate,amounts%20of%20gas%20and%20smoke.&text=Fluoride%20gas%20emission%20can%20pose,large%20Li%2Dion%20battery%20packs.

[sandalcandal]

Dude get over it. You seen to have a hard on for proving every minute thing I say wrong.

I'm not trying to rile you up here. I treat other forum members the same way when they post misinformation (A, B, C, D, E, F, there are more many). Don't go thinking you're special and I've got some sort of affixation for you and are holding you in unreasonably high account; this is just the kind of person I am. Call me a nut bag for caring too much about having correct information I guess. I do contribute actual engineering discussion too: 1kW Resonant Converter - Analysis, Design, Build and Validation, Picking a DSP MCU for Power Conversion - Experiences?

IMO Your whole idea of how such systems fail is far from "minutely" off.

FRV initially led the response to the incident, with CFA taking control of the incident at 3.30pm. The two agencies worked in support of one another throughout the incident, and were also supported by Victoria Police, Ambulance Victoria and the EPA.

Point taken, I still feel like action should be taken to correct the initial erroneous reports.
https://www.meaa.org/meaa-media/code-of-ethics/
https://ethics.journalists.org/topics/corrections/
https://www.niemanlab.org/2019/03/can-our-corrections-catch-up-to-our-mistakes-as-they-spread-across-social-media/

Anyway, this is clearly getting more emotional than rational. Even if you don't want to I'm taking break from this conversation.

[EEVblog]

FRV initially led the response to the incident, with CFA taking control of the incident at 3.30pm. The two agencies worked in support of one another throughout the incident, and were also supported by Victoria Police, Ambulance Victoria and the EPA.

Point taken, I still feel like action should be taken to correct the initial erroneous reports.

Oh FFS, it was NEVER an "erroneous report", it was an OFFICAL warning issued by the fire department, and rightfully picked and disseminated by the local media.
It was later downgraded, presuably after the atmospheric testing they were doing on site.

[EEVblog]

Don't go thinking you're special and I've got some sort of affixation for you

Ditto.
And I LOL'd at your "Dave pinned my comment because he knows who I am" comment.

[sandalcandal]

Don't go thinking you're special and I've got some sort of affixation for you

Ditto.
And I LOL'd at your "Dave pinned my comment because he knows who I am" comment.

Easiest (most directly apparent) way to try cut through the crap in youtube comments. Pretty cringe in retrospect. 

[floobydust]

I also noticed the new containers arriving on site and then thought how the hell are they going to get in there to install them?
It really looks like no room for a vehicle, crane or lift to grab and move a container    that's engineering

[EEVblog]

I also noticed the new containers arriving on site and then thought how the hell are they going to get in there to install them?
It really looks like no room for a vehicle, crane or lift to grab and move a container    that's engineering

They have a large crane onsite that looks like it can reach acros the entire four columns.
So they build four columns at once and then move the crane, and then another four etc.
So the entire layout would be dictacted by the crane reach and access.
Would also be why the cables are underground, so the crane can get around later through the matrix and replace modules.

[EEVblog]

According to this, Neoen was awarded cheap finance for this from the Victorian government, so kinda-sorta some government money involved here.

https://www.reuters.com/world/asia-pacific/fire-breaks-out-tesla-australia-mega-battery-during-testing-2021-07-30/

[wilfred]

Hmm.  You'd have thought that they'd have spaced the containers out a bit more to reduce the risk of fire spreading.   Its a green field site so the cost of say 50% more land would probably be a small part of the total budget . . .

It is more likely farm land which is expensive. Remember having these batteries in a desert makes little sense because the consumers are far away and the temperatures are much higher. However, the closer you move to civilisation and land that is useful for farming, the more expensive the land will be. Another thing to factor in is that you'll need to pave and maintain (get rid of weeds) the area the batteries are on; a compact area is cheaper.

There are farms out there but not much. It is pretty close to being outer suburban Geelong but it isn't there yet. There are High Voltage transmission lines running through which is obviously why they built it there.

https://www.google.com/maps/@-38.0373953,144.2996623,1733m/data=!3m1!1e3?hl=en-US

And for the movie trivia buffs Anakie Rd is a filming location for the original Mad Max film. Only they showed a sign (1:45 in the link) showing roads to Anarchy and Bedlam.

[nctnico]

Hmm.  You'd have thought that they'd have spaced the containers out a bit more to reduce the risk of fire spreading.   Its a green field site so the cost of say 50% more land would probably be a small part of the total budget . . .

It is more likely farm land which is expensive. Remember having these batteries in a desert makes little sense because the consumers are far away and the temperatures are much higher. However, the closer you move to civilisation and land that is useful for farming, the more expensive the land will be. Another thing to factor in is that you'll need to pave and maintain (get rid of weeds) the area the batteries are on; a compact area is cheaper.

There are farms out there but not much. It is pretty close to being outer suburban Geelong but it isn't there yet. There are High Voltage transmission lines running through which is obviously why they built it there.

Looking at Google maps the rectangles in the landscape are a dead giveaway that this is farm land. The number of farms doesn't matter; a single farm can work on a huge piece of land.

[Ian.M]

Apparently the battery covers about five hectares of land.    Tesla Megapacks have a unit price in larger installations of approx. 1M USD per pack (so approx. 1.3M AUD).  The recent record price for farmland in Victoria:

In March, a 179-hectare (442-acre) block of land in western Victoria sold for $23,588 a hectare ($9,550 an acre), Ararat Elders agent Gary Todd said people were "gobsmacked".

https://www.abc.net.au/news/2021-05-02/soaring-market-for-australian-farms/100106246

That's pretty clear that the land the whole battery sits on is at least an order of magnitude cheaper than a single Megapack unit, so is negligible compared to the value of what's sitting on it

I *still* reckon making the aisles as narrow as they did and closely clustering the Megapacks in groups of four was penny wise and pound foolish.  They are lucky to have only totally lost two units with casing damage to the units across the aisle from them.

Edit: currency conversion!

[Siwastaja]

Direction of wind seems optimal / lucky, saving the too-close adjacent module (seen on the left on the latest photo on this thread). I agree a few extra meters would be insignificant regarding land cost, but oh well, it haven't spread so maybe it's good enough as it is.

Regarding toxicity of smoke, remember, "normal" smoke is rather toxic. Even when just burning wood or fuels that are supposed to be burned, even in optimal burning conditions, exposure to smoke must be still very limited. Burn plastics at temperatures too low, and it's already orders of magnitude worse. Likely the contribution of the electrolyte is small because the amount of electrolyte is small compared to everything else burning.

Any large fire causes people to be evacuated down wind.

[EEVblog]

Some technical data on the Tesla Victoria Big Battery fire. It was charging just before the incident.

https://t.co/Bfqdg5dbNh?amp=1
https://t.co/mgeRH2ep53?amp=1
https://t.co/zsnxtci9wo?amp=1

[Marco]

I *still* reckon making the aisles as narrow as they did and closely clustering the Megapacks in groups of four was penny wise and pound foolish.  They are lucky to have only totally lost two units with casing damage to the units across the aisle from them.

A lot of things will be getting re-engineered for the next installation I bet, I doubt they even considered this a possible failure mode outside a meteor/lightning strike.

[sandalcandal]

Regarding toxicity of smoke, remember, "normal" smoke is rather toxic.  [...]
Any large fire causes people to be evacuated down wind.

100% agreed, any smoke is a health hazard but there is a significant difference between a lithium ion battery fire and a PCB (polychlorinated biphenyl) storage facility fire in terms of risk to public health, that's the point I was trying to make in terms of "toxic smoke".

Some technical data on the Tesla Victoria Big Battery fire. It was charging just before the incident.

https://t.co/Bfqdg5dbNh?amp=1
https://t.co/mgeRH2ep53?amp=1
https://t.co/zsnxtci9wo?amp=1

Really nice info, thanks for sharing. Clever investigation of the incident timeline using analysis of NEM data.

[Ian.M]

Direction of wind seems optimal / lucky, saving the too-close adjacent module (seen on the left on the latest photo on this thread). I agree a few extra meters would be insignificant regarding land cost, but oh well, it haven't spread so maybe it's good enough as it is.

They lost the far too close back to back module which it *did* spread to, so turned a ~$1M USD loss into a $2M + USD loss. 

[nctnico]

I *still* reckon making the aisles as narrow as they did and closely clustering the Megapacks in groups of four was penny wise and pound foolish.  They are lucky to have only totally lost two units with casing damage to the units across the aisle from them.

A lot of things will be getting re-engineered for the next installation I bet, I doubt they even considered this a possible failure mode outside a meteor/lightning strike.

First come up with some numbers about the total number of power packs installed versus the number of fires. I'm quite sure they have considered the risk of fire versus the distance between the units.

[Marco]

I doubt they have had enough installed for long enough to make an economic estimate purely on statistics. If not they can either decide that maybe this was a fluke and wait a bit to be sure. Or assume there are unknown unknowns and be a little more conservative for the next installation.

A lot of things will be getting re-engineered for the next installation I bet.

[floobydust]

First come up with some numbers about the total number of power packs installed versus the number of fires. I'm quite sure they have considered the risk of fire versus the distance between the units.

Based on those numbers, this is "acceptable risk"? 
The site's not designed at all for fires of this magnitude. The containers packed too close, there are no fire breaks anywhere with ceramic or brick wall in the rows. Mr Fireman ride your bicycle to get in there.
It's not far from a disaster kind of situation and everybody wants to keep it quiet.

It might not be Tesla that dictates pack spacing and site layout, Neoen maybe- but it's a copycat error that's everywhere now. Nobody planned for this kind of event.

Another issue is damage to the AC bus wiring would take out that entire row and everybody grab a shovel to fix that. In the electric utility industry, above ground cable trays are used for good reason- you can easily replace or add cables as needed, they are completely serviceable, transformer fires included.
Here it's underground and cooked due to the fire, not an easy repair. I don't know the intermediate bus voltage they chose.

The crane looks feeble with tiny counterweights so I can't see it moving a loaded powerpack, and the crane likely just there for construction afterwhich it's taken down.

[Siwastaja]

Physical separation is excellent fire protection.A brick wall allows reducing this distance.

It's all about optimization of land cost vs. cost of building the brick wall. Where land is cheap, just add 10-15 meters of separation to the units and be done with it. At high land value sites, do 2 meters + brick wall + another 2 meters instead.

Instead of brick, it can be like two layers of steel, hollow inside. Not that expensive, not that difficult to make.

[EEVblog]

The installation of these megapacks seems quite standardised, this is the Moss landing California facility:

[EEVblog]

Another issue is damage to the AC bus wiring would take out that entire row and everybody grab a shovel to fix that. In the electric utility industry, above ground cable trays are used for good reason- you can easily replace or add cables as needed, they are completely serviceable, transformer fires included.
Here it's underground and cooked due to the fire, not an easy repair. I don't know the intermediate bus voltage they chose.

The cables would all be underground to allow for the crane to get in and out later to replace packs and transformers etc.

[sandalcandal]

Second pack taken by fire, probably the unit directly behind the one on fire [initially].

There was one battery pack on fire to start with, but it did spread to a second pack that was very close to it.

And fire has subsided to within the enclosure. 3:30pm AEST

Update at 3.30pm on 1 August, 2021: The fire has subsided significantly. Crews are now doing atmospheric monitoring around the cabinet and will check to see how active the fire is internally behind each of the doors.

https://news.cfa.vic.gov.au/firefighters-battle-large-battery-fire-near-geelong

[sandalcandal]

Instead of brick, it can be like two layers of steel, hollow inside. Not that expensive, not that difficult to make.

It seems like they have a double wall steel structure or at least an air gap venting path directing heat out the top of the enclosure and some front facing vents towards the wider aisles looking at previously posted images and safety brochure
https://www.greenburghny.com/DocumentCenter/View/7245/PB-20-18-Eagle-Energy---Tesla-Safety-Data-Sheet

Evidently insufficient here. Maybe we'll see more back-to-back separation in future installations.

[sandalcandal]

The installation of these megapacks seems quite standardised, this is the Moss landing California facility:

Separation between back-to-back pairs of packs is much larger than the Moorabool facility image you posted earlier it looks like, unless there's 4 units under each covering?

Can also see water being blasted between packs to the left to protect the other packs.

[Siwastaja]

Instead of brick, it can be like two layers of steel, hollow inside. Not that expensive, not that difficult to make.

It seems like they have a double wall steel structure or at least an air gap venting path directing heat out the top of the enclosure and some front facing vents towards the wider aisles looking at previously posted images and safety brochure
https://www.greenburghny.com/DocumentCenter/View/7245/PB-20-18-Eagle-Energy---Tesla-Safety-Data-Sheet

Evidently insufficient here. Maybe we'll see more back-to-back separation in future installations.

Of course it's insufficient, given very close spacing (some dozen centimeters gap only). Temperatures inside the burning battery pack are well in excess of 1000 degC, and thermal runaway onset of the adjacent pack is mere 150degC or so, painfully close to ambient temperature. Even if fire is directed upwards, the fire itself radiates enough heat to heat surroundings to 150degC for several meters. I see no other option than significant separation, or less separation coupled with another layer of steel wall, high enough to block the fire rising above the units. Of course that secondary wall could be thermally insulated (mineral wool) but it would need to be large enough to contain the fire and hot gases around the burning unit, forming a "chimney". Probably more expensive than just adding a few meters of extra separation.

Seeing these are installed as pairs, at very least, one such pair should be assumed to be consumed during fire so that it's not a surprise to anyone when that happens.

[Jan Audio]

Did it all burn down ?
*disaster tourist i am.

Why not dig them in ?

[nctnico]

Instead of brick, it can be like two layers of steel, hollow inside. Not that expensive, not that difficult to make.

It seems like they have a double wall steel structure or at least an air gap venting path directing heat out the top of the enclosure and some front facing vents towards the wider aisles looking at previously posted images and safety brochure
https://www.greenburghny.com/DocumentCenter/View/7245/PB-20-18-Eagle-Energy---Tesla-Safety-Data-Sheet

Evidently insufficient here. Maybe we'll see more back-to-back separation in future installations.

Of course it's insufficient, given very close spacing (some dozen centimeters gap only). Temperatures inside the burning battery pack are well in excess of 1000 degC, and thermal runaway onset of the adjacent pack is mere 150degC or so, painfully close to ambient temperature. Even if fire is directed upwards, the fire itself radiates enough heat to heat surroundings to 150degC for several meters. I see no other option than significant separation, or less separation coupled with another layer of steel wall, high enough to block the fire rising above the units. Of course that secondary wall could be thermally insulated (mineral wool) but it would need to be large enough to contain the fire and hot gases around the burning unit, forming a "chimney". Probably more expensive than just adding a few meters of extra separation.

Seeing these are installed as pairs, at very least, one such pair should be assumed to be consumed during fire so that it's not a surprise to anyone when that happens.

Still, looking at the pictures in the first post it looks like the unit installed right next to it didn't catch fire like the unit with the flames bursting out. To me it seems as if the electronics caught fire but not the batteries themselves. But this is purely speculated based on the pictures posted in the first post.

[NiHaoMike]

Of course it's insufficient, given very close spacing (some dozen centimeters gap only). Temperatures inside the burning battery pack are well in excess of 1000 degC, and thermal runaway onset of the adjacent pack is mere 150degC or so, painfully close to ambient temperature. Even if fire is directed upwards, the fire itself radiates enough heat to heat surroundings to 150degC for several meters. I see no other option than significant separation, or less separation coupled with another layer of steel wall, high enough to block the fire rising above the units.

Some fire sprinklers will do the trick. And now maybe they figured out that sprinklers on the outside would be an even better idea.

Or just use a safer, more economical energy storage technology? If it's mostly to run air conditioning, making ice is an order of magnitude or two cheaper than batteries (total system cost, the storage medium is even cheaper than that) and I have never heard of a block of ice catching on fire...

[floobydust]

quite the impressive site, so much for Cannery Row.

Are there any Megapack sites up and running yet?
Moss Landing at least has them spaced reasonably, 8 packs per transformer and also has a lift for moving the containers.
Moorabool looks like SF6 switchgear and step-up transformer, every 4 packs. Actually the pics show cable trays along the rows and underneath the cabinets.
Many site drawings show no allowance for fire, it's packed in as close as possible.

[sandalcandal]

Fire was declared under control 3:05pm today (Monday, 2nd August). CFA said the fire started at around 10:30am Friday 30th July (the source analysing NEM data posted earlier corroborates this). That's 76:35 hr:min from incident start to under control.

Update at 5.00pm on 2 August, 2021:

The Moorabool incident was declared under control at 3.05pm on Monday 2 August.

Firefighters have successfully completed the operation of opening all doors to the container of the battery, with no sign of fire.

A smaller number of firefighters and fire trucks from CFA will remain on scene for the next 24 hours as a precaution in case of reignition. They will continue taking thermal temperature readings two-hourly to monitor damaged units.

https://news.cfa.vic.gov.au/firefighters-bring-large-battery-fire-near-geelong-under-control

[tszaboo]

I *still* reckon making the aisles as narrow as they did and closely clustering the Megapacks in groups of four was penny wise and pound foolish.  They are lucky to have only totally lost two units with casing damage to the units across the aisle from them.

A lot of things will be getting re-engineered for the next installation I bet, I doubt they even considered this a possible failure mode outside a meteor/lightning strike.

First come up with some numbers about the total number of power packs installed versus the number of fires. I'm quite sure they have considered the risk of fire versus the distance between the units.

Yeah, I'm sure a company made by Elon made all the necessary calculations, and left enough time for all the engineers to finish their work with due diligence. And they didn't rush some half finished product that was cobbled together from leftover parts from other companies. Didn't rush, because there was no chance in hell, that the company wasn't profitable so "I dont care how it is we have to ship it, otherwise we run out of investors money". Not a chance. I mean they dont have a track record of this at all.*

*This entire post was sponsored by Sarcasm Inc.

[floobydust]

The only tool here was spraying water on adjacent cabinets. This is literally stone age.
"If we try and cool them down it just prolongs the process," the CFA's Assistant Chief Fire Officer Ian Beswicke said." 

IMHO the cabinets could not have been spaced any closer together and lithium battery fires happen as we see in the models X and S along with the F/W change (and recent class action lawsuit) over temporarily reducing maximum voltage (capacity).

Why not include site allowances for increased spacing? Because using more real estate costs money. Or it's an error between Tesla and Neoen over who dictates the Megapack spacings.
Maybe create firewall technology or something to put out the fire instead of letting it burn and spew toxic fumes for 3 days.

"more than 30 fire trucks and 150 firefighters" it's literally $1M in labour costs to let it burn down. Is everything sensationalized nowadays, or is this guys working in shifts getting overtime $$$$ 

[james_s]

Or just use a safer, more economical energy storage technology? If it's mostly to run air conditioning, making ice is an order of magnitude or two cheaper than batteries (total system cost, the storage medium is even cheaper than that) and I have never heard of a block of ice catching on fire...

But how do you distribute that ice based air conditioning among the thousands of individual buildings and houses? Drive around and deliver blocks of ice like back in the old days? What do you do with the ice then? Have a separate ice-based air conditioning unit to supplement the electrical one? If the goal is to provide air conditioning for a single large building or complex then ice may make sense.

[NiHaoMike]

But how do you distribute that ice based air conditioning among the thousands of individual buildings and houses? Drive around and deliver blocks of ice like back in the old days? What do you do with the ice then? Have a separate ice-based air conditioning unit to supplement the electrical one? If the goal is to provide air conditioning for a single large building or complex then ice may make sense.

One unit per house, it's trivial to scale as needed.

[Ed.Kloonk]

Did it all burn down ?
*disaster tourist i am.

Why not dig them in ?

The media here is either ignoring it or the ones who are running the story have to go full retard.

https://www.theguardian.com/australia-news/2021/aug/02/tesla-big-battery-fire-in-victoria-burns-into-day-three

A large blaze at Victoria’s “big battery” project has been brought under control by firefighters after burning for more than three days, allowing investigators to begin examining the site.

A Tesla battery bank caught fire while it was being set up in Moorabool on Friday morning, and then spread to a second battery.

The fire burned throughout the weekend and into a fourth day, before it was declared under control just after 3pm on Monday.

Fire crews will remain at the site for the next 24 hours “as a precaution in case of reignition” and will take temperature readings every two hours, the Country Fire Authority said.

[trobbins]

There is no technical information to support more news, and I'd anticipate no new photos from initial site inspections (unless they are leaked), so all we are left with is 60+ posts here with most trying to vent their opinions and speculate what could have been done better.

Nobody yet knows what the root cause was, or whether mitigating processes had a fault, or whether product or module or part changes could result that would avoid this in the future, or .....

[Berni]

But how do you distribute that ice based air conditioning among the thousands of individual buildings and houses? Drive around and deliver blocks of ice like back in the old days? What do you do with the ice then? Have a separate ice-based air conditioning unit to supplement the electrical one? If the goal is to provide air conditioning for a single large building or complex then ice may make sense.

Yep you use an air conditioner to cool a block of ice somewhere in your basement. Have the air conditioner run on command from the grid when there is cheep excess power available. Then when the peak of the day heat comes you pump antifreeze from coils inside the ice into a regular air conditioner wall unit.

This is effectively the same thing as if you had a battery that charged up at night and then powered the air conditioner during the day. But imagine how big of a pile of lithium batteries it would take to make that happen. Now imagine how big of a battery it would take to do this for 10 000 homes, the result is something like these giant battery storage systems. Except that blocks of ice don't catch fire even if you try to set one on fire.

Sure you might be hard pressed to sell this system in Norway. But some of the well populated hot places like Texas, California, Florida....etc that run air conditioners for most of the year could really benefit from it. But the power companies need to get on board and pass a good chunk of the savings down to the costumer so that such a system is a sensible way of saving you money to pay for the extra trouble and cost of installing this thing.

EDIT: And i know this is missing the benefit of a giant battery storage system of being able to pump huge peak powers back into the grid on the drop of a hat. So there is still reason to have these things, but much fewer and smaller installations would do the job if it was helped out by other storage.

[floobydust]

In the utility industry here, 240kV up substations and equipment it is mandated to have GPS-sync'd timestamps on all SCADA alarms. This is so you can backtrack major outages and grid collapses, across the continent if need be, at msec resolution to determine the sequence of events.

All they can do here is go through the historian/log file, assuming these battery packs are on CANBUS which is recorded by a central HMI, and see what the sequence of events was.
I would think a danger is Tesla taking an automotive battery pack doing a few mods and throwing it into a different, unfamiliar environment. You can't record events inside the  battery pack that's now ashes...

It looks like there's a gag order right now.  As long as they have a logfile to analyze.

[trobbins]

The event was during initial commissioning.  They had utility approval to connect to grid (so had passed all documentation and inspections and comms connections for both HV and LV equipment and disconnects and monitoring/commands, and energisation of grid-connect Txs).  Event appears to have occurred part way through charging, but who knows whether that was just on a subsection of battery and its associated grid-connect TX, or they had multiple sections going.  They obviously would have confirmed all emergency facilities prior to testing and confirmed scada and status monitoring.  I would be amazed if local module logging was only logged within the PLC in the module, given a site scada.

External time sync would only be a concern imho if an external event initiated the fault.  It sounds like the export/connection was dropped appropriately, including a site-wide trip.  They may have localised/grouped battery backed comms/PLC facilities which could have failed at some time and left the site scada blind.

As I understand it, the grid product battery packs are not aligned to Tesla vehicle battery use - there certainly may be initial similarities, but the manufacturing plant and what is embedded in a pack could be significantly different.  I recall seeing a Megapack patent a few years ago, which was certainly different in the way battery shelves were modularised, wired and deployed (but I can't quickly find the patent).

[ahbushnell]

Of course it's insufficient, given very close spacing (some dozen centimeters gap only). Temperatures inside the burning battery pack are well in excess of 1000 degC, and thermal runaway onset of the adjacent pack is mere 150degC or so, painfully close to ambient temperature. Even if fire is directed upwards, the fire itself radiates enough heat to heat surroundings to 150degC for several meters. I see no other option than significant separation, or less separation coupled with another layer of steel wall, high enough to block the fire rising above the units.

Some fire sprinklers will do the trick. And now maybe they figured out that sprinklers on the outside would be an even better idea.

Or just use a safer, more economical energy storage technology? If it's mostly to run air conditioning, making ice is an order of magnitude or two cheaper than batteries (total system cost, the storage medium is even cheaper than that) and I have never heard of a block of ice catching on fire...

This is for grid stabilization.  I don't think ice will work.

[Berni]

Some fire sprinklers will do the trick. And now maybe they figured out that sprinklers on the outside would be an even better idea.

Or just use a safer, more economical energy storage technology? If it's mostly to run air conditioning, making ice is an order of magnitude or two cheaper than batteries (total system cost, the storage medium is even cheaper than that) and I have never heard of a block of ice catching on fire...

This is for grid stabilization.  I don't think ice will work.

On demand load modulation also does the job of grid stabilization.

The idea is that once you have enough storage to store the heating or cooling energy for one day then you can freely chose when over the 24h that energy will be pulled from the grid. So when load on the grid drops and the standby power generation if forced to throttle back (throwing away the energy as heat rather than letting it trough the turbines) you can instead command 10 000 air conditioners to turn on suddenly gulping megawatts of power and selling it to the residents. This saves money by not throwing away power. Then when there is a sudden rise in demand on the grid you can command the 10 000 air conditioners to turn off and suddenly there are a few extra megawatts of power on the grid within seconds. Then once the plants spool up to catch up you slowly let them turn back on gradually to make up for the time they had to turn off.

This has the same effect as having a giant battery that you command to suck up  megawatts of power when there is too much of it, or tell it to release it megawatts of power when there is need. Pushing extra power to the grid or removing the same amount of load on the grid has the same effect of there being more power available.

Ice stores about 330kJ/kg so about a 100kg block stores about 9kWh and costs a few cents worth of water.
Lithium batteries store about 260 Wh/kg so a 100kg battery pack stores about 26kWh and for a cheep one could get as low as 100$ per kWh so 2600$
So yes ice latent heat is only about 1/3 as energy dense as a lithium battery pack, but it costs many orders of magnitude less. So you might as well go for the cheap option to start with and move to batteries once most air conditioners have already been used up for on site storage. Just like solar roadways only make more sense once we run out of other better places to put solar panels like roofs.

[Kleinstein]

When you compare heat stored in Ice and electricity, one has to take into account the efficiency of the air conditioner. The heat ca. be something like 5 times the electricty consumption, with a low temperature difference possibly even more than 10 fold. Going all the way to making ice needs extra energy when all one really needs is some 20 C for a compftable room.

Yes some heat pumps could be used to control the load and this way reduce the need for electric storage. It is quite effective and already used in large facilities like central cold (frozen goods) storage. Still with small units (e.g. household size may be only 200W) there is quite some control effort.
Still modulated load can not fully replace real storage - someone still needs to provide the little power in a low wind night. Real storage would also act double, as a source and controlled load.

[james_s]

It's too bad it's so difficult to store a really large amount of heat. In the winter my heat pump is pushing heat from outside into my house, in the summer it's going the other way pushing heat from inside to the outside. If I had a massive insulated water tank I could freeze it into a solid block of ice over the winter from heating and then melt it over the summer using it to air conditioning and wind up with hot water by the time winter rolls around. I have not done any calculations but I suspect the amount of water (or whatever) required to store a season worth of heat would be massive though.

A friend of mine said one of his friends once built a ground source heat pump back in the 80s by burying a couple of old condenser coils underground and it created a permafrost.

[Gyro]

Years ago I saw a building project where they were building a massive insulated water tank into the foundations as a thermal buffer - just using its thermal mass, no heat pump involved.

By the time you start involving the latent heat of freezing / melting you have a massive increase in available thermal energy storage. Ideally you'd want to stay close to freezing / melting, otherwise you're just relying on the thermal mass and the temperature change will be much steeper.

[EEVblog]

There is no technical information to support more news, and I'd anticipate no new photos from initial site inspections (unless they are leaked), so all we are left with is 60+ posts here with most trying to vent their opinions and speculate what could have been done better.

Yeah, I don't expect any more news, let alone photos. Tesla or the owner won't be dumb enough to release photos, and the Fire department would probably get in big trouble if they released on-site private property photos.
Might even be the last we ever hear of it, unless as you said, someone leaks something.

[NiHaoMike]

It's too bad it's so difficult to store a really large amount of heat. In the winter my heat pump is pushing heat from outside into my house, in the summer it's going the other way pushing heat from inside to the outside. If I had a massive insulated water tank I could freeze it into a solid block of ice over the winter from heating and then melt it over the summer using it to air conditioning and wind up with hot water by the time winter rolls around. I have not done any calculations but I suspect the amount of water (or whatever) required to store a season worth of heat would be massive though.

It has been done with geothermal heat pumps, get below the water table and you get a thermal connection to a very, very large volume of water.

[trobbins]

It's possible that background info could come out as part of a mitigation description (by Tesla, utility, Oz regulator or other/future project operator PR), or down the track via some site contractor personal who aren't under nda.

[james_s]

Yeah, I don't expect any more news, let alone photos. Tesla or the owner won't be dumb enough to release photos, and the Fire department would probably get in big trouble if they released on-site private property photos.
Might even be the last we ever hear of it, unless as you said, someone leaks something.

I would think a news station could get all the photos they want without setting foot on the property by using a helicopter carrying a stabilized camera with a telephoto lens. The installation is outdoors so unless it's covered by tarps or something it will be visible.

[EEVblog]

Yeah, I don't expect any more news, let alone photos. Tesla or the owner won't be dumb enough to release photos, and the Fire department would probably get in big trouble if they released on-site private property photos.
Might even be the last we ever hear of it, unless as you said, someone leaks something.

I would think a news station could get all the photos they want without setting foot on the property by using a helicopter carrying a stabilized camera with a telephoto lens. The installation is outdoors so unless it's covered by tarps or something it will be visible.

Possible. Also with a drone. I'm not sure anyone will bother though.

[Berni]

True i did forget to take in account that heat pumps move significantly more heat than they consume. But still a block of ice remains very cheep because its just water, all the cost is in the container around it and this is easily expandable. It's not uncommon to have water heaters that hold more than 100kg of water, so the size of such ice storage would not get unpracticaly large just to store one day worth of air conditioning cold.

As for storing heat energy for half a year to cover the summer and winter difference is not easy. This does reach unpracticaly large sizes to store enough of it. Since latent heat can't be used for this, it makes it even worse (Tho molten wax can store heat pretty efficently). If you really wanted to have an advantage in terms of heating/cooling the best solution is to have an underground house. The ground temperature only varies by a few degrees trough the year. Tho in most climates the average ground temperature is too low to be comfortable, but some insulation and the waste heat from all the electrical appliances could bring that up closer to comfortable. But building such a house is expensive and you end up living in a windowless basement so it's not that great of an idea.

Geothermal  is also in most cases too cold to be directly useful, but even the cold groundwater in winter is still a much better source for feeding a heatpump rather than the freezing air outside, can work for air conditioning tho. Unless you end up on top of a hot geothermal spring, those are very useful for heating, but are rather rare.

[ahbushnell]

It's too bad it's so difficult to store a really large amount of heat. In the winter my heat pump is pushing heat from outside into my house, in the summer it's going the other way pushing heat from inside to the outside. If I had a massive insulated water tank I could freeze it into a solid block of ice over the winter from heating and then melt it over the summer using it to air conditioning and wind up with hot water by the time winter rolls around. I have not done any calculations but I suspect the amount of water (or whatever) required to store a season worth of heat would be massive though.

A friend of mine said one of his friends once built a ground source heat pump back in the 80s by burying a couple of old condenser coils underground and it created a permafrost.

https://www.energy.gov/energysaver/heat-and-cool/heat-pump-systems/geothermal-heat-pumps

[TimFox]

Ice:
Improperly designed ice structures can be dangerous:  https://apnews.com/article/arts-and-entertainment-tennessee-905233721dbb5b7692fa99c94e8c4e3b
However, water in its various states has unique physical properties (such as heat capacity) compared against other materials.

[TimFox]

Yeah, I don't expect any more news, let alone photos. Tesla or the owner won't be dumb enough to release photos, and the Fire department would probably get in big trouble if they released on-site private property photos.
Might even be the last we ever hear of it, unless as you said, someone leaks something.

I would think a news station could get all the photos they want without setting foot on the property by using a helicopter carrying a stabilized camera with a telephoto lens. The installation is outdoors so unless it's covered by tarps or something it will be visible.

Possible. Also with a drone. I'm not sure anyone will bother though.

There have been two industrial accidents near Chicago recently that attracted media attention.  Lots of big black smoke, emergency response, and evacuation orders.   There was no shortage of aerial video on local TV stations.

[floobydust]

No news updates, surprising to see a media blackout as if this had occured in a communist country. Dave's 1411 is highly ranked on the search engines. If I lived nearby or downwind I would be upset with the news (lack of) coverage.

I found this days older pic (paywalled Australian News) which showed the fire burning, along with the crane brought in - Tesla's new hot swap technology. Doesn't matter if it's on fire, you can put in a new Megapack before anyone notices...  Actually it likely means they pulled up and ripped out adjacent containers to stop the fire from spreading? If that's the case, then the site is a mess and no pictures of that please.

CFA, Energy Safe Victoria, WorkSafe Victoria, Minister for Energy and Renewables, EPA, Labour Government and Opposition having a conniption 

I would expect the site's planned November operation date to get pushed back, unless they add manpower. TBH I have worked on large scale (AC) power projects like this and it's cookie cutter work unless they make drastic safety changes. It's already too late to increase spacing between units, that would blow up the scope of work so I'm not sure how they'll spin not needing that.

One concern I had is the smoke crossing the HV transmission lines will induce arcing because the spacings are generously for air, not carbon soot etc. Smoke will start arcs between HV lines and typically the breakers will do reclose operations then give up. So it's entirely possible for this incident to have taken out transmission on that line until the smoke moved away.

[EEVblog]

No news updates, surprising to see a media blackout as if this had occured in a communist country.

Nothing nefarious, it's just that no one cares. It's made the headline and that was it.

[chickenHeadKnob]

One concern I had is the smoke crossing the HV transmission lines will induce arcing because the spacings are generously for air, not carbon soot etc. Smoke will start arcs between HV lines and typically the breakers will do reclose operations then give up. So it's entirely possible for this incident to have taken out transmission on that line until the smoke moved away.

Discharge through the air not the biggest problem. I would be worried if the insulators downstream are accumulating soot on their surfaces. My father told of more than one occasion where a bird sitting on an insulator shit down the side and vaporized himself.

[floobydust]

Moorabool Terminal has 500kV (~73m high towers) and 220kV (~56m high towers) transmission lines running through. I know smoke between HV lines induces arcs, I've seen it happen a few times at lower heights/voltages. On second thought in pics the transmission lines are quite far away and high so I don't know what the chances are, but you don't want to take out one of those lines. The 2014 incident where they accidentally grounded the line on one phase was a fun read for the chain of events. Here we had a dumb project manager forget the ground chains were still attached to the lines, switched in 240kV to ground and 1/4 the country noticed those milliseconds.
I haven't seen a single-line for the Tesla Powerwall tie in. It looks like 33kV to 220kV in the yard.

"Every Megapack arrives pre-assembled and pre-tested from Tesla’s Gigafactory – including battery modules, bi-directional inverters, a thermal management system, and AC main breaker and controls. No assembly is required, the Megapack’s AC output is simply connected to the site wiring." 

[james_s]

I remember watching a video where they were starting up a vintage diesel locomotive at a rail museum and the smoke belching out of the stack caused a flash over from the overhead wires used for electric trains.

[sandalcandal]

I *still* reckon making the aisles as narrow as they did and closely clustering the Megapacks in groups of four was penny wise and pound foolish.  They are lucky to have only totally lost two units with casing damage to the units across the aisle from them.

A lot of things will be getting re-engineered for the next installation I bet, I doubt they even considered this a possible failure mode outside a meteor/lightning strike.

First come up with some numbers about the total number of power packs installed versus the number of fires. I'm quite sure they have considered the risk of fire versus the distance between the units.

Yeah, I'm sure a company made by Elon made all the necessary calculations, and left enough time for all the engineers to finish their work with due diligence. And they didn't rush some half finished product that was cobbled together from leftover parts from other companies. Didn't rush, because there was no chance in hell, that the company wasn't profitable so "I dont care how it is we have to ship it, otherwise we run out of investors money". Not a chance. I mean they dont have a track record of this at all.*

*This entire post was sponsored by Sarcasm Inc.

Yeah, over 7 years of development on stationary energy systems for a product launched in 2019 is definitely not enough time for all the necessary calculations, and left enough time for all the engineers to finish their work with due diligence. Plus given it was another 2 years since launch that the system in question was installed there's no way they didn't rush some half finished product that was cobbled together from leftover parts from other companies. And unquestionably they rushed because there was no chance in hell, that the company wasn't profitable because they were already profitable last year and are unquestionably even more profitable this year with over $1B in GAAP net income in the last quarter.*

*This segment was sponsored by Snark Inc.

Sorry is this too fast for you? Maybe we should take a 15 years before releasing a product because nearly a decade is too fast for you? What is this guy smoking?

[EEVblog]

"Every Megapack arrives pre-assembled and pre-tested from Tesla’s Gigafactory – including battery modules, bi-directional inverters, a thermal management system, and AC main breaker and controls. No assembly is required, the Megapack’s AC output is simply connected to the site wiring." 

In that case it seems almost certain the fire will be Tesla's responsibility, unless it's dodgy mains wiring contacts or something? As it appears the fire originated within the megapack cabinet somewhere.

[Grapsus]

It's basically confident IT guys from California building high power stuff from scratch, because disruption. Might take 10-20 years to figure out that you might need walls between the modules in order to limit damage. It will then be called smart-separator or something

[Neilm]

I remember reading in a local paper (very many moons ago) that fire investigators had managed to pinpoint why a "factory" had burnt down. The factory contained a high voltage limulator for simulating lightning strikes. Apparently someone had left a spanner lying around and during testing an upgrade, it had fallen across the capacitor bank.

I don't think there is any point speculating until some form of report is produced.

[TimFox]

Industrial accidents like that happen all the time, but sifting through the wreckage to pinpoint the cause can take a long time.
I was in California a few years ago when a wildfire forced me to evacuate the worksite.
Since the authorities could locate the starting point for the fire, after it was safe they traveled to the power pole in question and found the burned carcass of a large bird that had been electrocuted, set on fire, and fallen to the dry ground whence the fire then started to spread.

[floobydust]

Do we know what battery technology is at this site?
~May 2021 Tesla changed to using lithium-iron phosphate in Megapacks. I wonder if there is new and old out there. Lithium-ion don't have a great safety record, nevermind the lack of fire fighting ability. The shift is supposedly to avoid nickel and cobalt shortages.

The cabinet wiring looked complete, like you need to daisychain power and data connections. During commissioning you check all wiring several times. It wasn't a loose twist-nut.
The hardware is a module kind of a consumer/automotive design glue the thing together and then add more glue, it's not industrial or even utility industry designed. The charge control electronics are in the blob. No sign of disconnects or fuses though.
"An electrical fault sparked the blaze that engulfed a part of the $200m big battery northwest of Geelong last month, the CFA says." Ahhh that's what it was 

These sites will have high line voltages whenever the transformers have light load.
I don't see a voltage reg transformer (tap-changer) anywhere so playing around at the 25MW level, the voltage stress would have been high on the Megapacks during charging. But again, they're supposed to have fuses and firmware to cope.

[T3sl4co1l]

True i did forget to take in account that heat pumps move significantly more heat than they consume. But still a block of ice remains very cheep because its just water, all the cost is in the container around it and this is easily expandable. It's not uncommon to have water heaters that hold more than 100kg of water, so the size of such ice storage would not get unpracticaly large just to store one day worth of air conditioning cold.

As for storing heat energy for half a year to cover the summer and winter difference is not easy. This does reach unpracticaly large sizes to store enough of it. Since latent heat can't be used for this, it makes it even worse (Tho molten wax can store heat pretty efficently). If you really wanted to have an advantage in terms of heating/cooling the best solution is to have an underground house. The ground temperature only varies by a few degrees trough the year. Tho in most climates the average ground temperature is too low to be comfortable, but some insulation and the waste heat from all the electrical appliances could bring that up closer to comfortable. But building such a house is expensive and you end up living in a windowless basement so it's not that great of an idea.

Geothermal  is also in most cases too cold to be directly useful, but even the cold groundwater in winter is still a much better source for feeding a heatpump rather than the freezing air outside, can work for air conditioning tho. Unless you end up on top of a hot geothermal spring, those are very useful for heating, but are rather rare.

If you're still curious, the back-of-the-envelope is around 40 tonnes H2O.

That's assuming 1MWh for the season (guessing that's an overestimate for most houses? Didn't look it up), no heat loss to environment and no heat pump dissipation, and a temp swing of 20K in the reservoir.

The heat pump I don't think changes anything, as it's always dissipating its heat somewhere, and the house and tank tend to dissipate that heat to the environment in turn.  The tank will end up biased on average, some degrees above/below average outdoor temperature, depending on which need is greater overall (cooling/heating).  Interesting is the bias is nonlinear, because of course the heat pump is less efficient for greater differences.

Or maybe it doesn't matter because you'd need a thermal time constant over a year to have this make any sense, and that's actually hard to do (not sure).  And if a random tank in the yard or ground has a shorter time constant, then that means you have more heat flux available through ambient dissipation, and the reservoir is simply irrelevant -- the solution reduces to a normal in-ground system.    Certainly, it seems it's at least good enough to do it that way, and you'd only stand to gain a few percentage points by, essentially, putting a bypass capacitor on one side of the heat pump.  That is, the ground-to-buried-line impedance acts like ESR, and a capacitor can average it out over some cycles (say, a time constant of one or a few days) -- which could be provided by a more economically sized reservoir.  Maybe doing double duty as, say, the water in a water softener brine tank or something, Idunno.  Or if you have well water, an oversized surge tank could do, heh.

Tim

[Berni]

This is pretty much the idea behind the ground based heatpumps. The soil has a large amount of heat capacity just because there is so much of it, while also not being all that good of a heat conductor so it insulates itself from the varying air temperature above. So its just using a big mound of material that is already in your yard as a heat capacitor.

The groundwater approach is the same except with water. So it doesn't really make all that much sense to bury a giant water tank in the yard for the purpose of heat storage.

However brick is also good at storing heat. We have a brick house and use it to scrape by without air conditioning by opening windows during the night and closing them during the day. When a heatwave hits it actually takes a few days for all the brick to actually warm up and the temperature inside to get uncomfortably hot. Unfortunately it also holds that heat so it also takes a few cold nights to cool it back down after a heat wave. There is one drywall addition to the house that is not brick and that thing has basically no thermal inertia, it quickly gets hot in the day and cold in the night despite having more insulation.

[thinkfat]

Using the ground as thermal storage is not without side effects though. While you store heat during the summer you dry out the soil and in spring it will stay frozen for longer. This has an impact on what you are able to do with the land. I recently talked to a guy who is using this approach for heating his home during winter, it seems planting something on this patch of land is not a good idea any more.

[Siwastaja]

Using the ground as thermal storage is not without side effects though. While you store heat during the summer you dry out the soil and in spring it will stay frozen for longer. This has an impact on what you are able to do with the land. I recently talked to a guy who is using this approach for heating his home during winter, it seems planting something on this patch of land is not a good idea any more.

This needs to be dimensioned right so that it's not only "using" charge from the "capacitor", but also letting it charge via natural conduction (both geothermal from inside the planet, and from above, basically by solar energy). Make it large enough, and temperature differences will be small. But yeah, ground source heat pumping is an expensive investment so it's appealing to try with a minimized collector area to reduce that cost. Here such installations are fairly common for household heating and the differences between installations (freezing wells vs. those that Just Work) are huge. Those who ask for a few quotes then pick the one who suggested the largest one and asks for +20-30% extra dimensioning will be satisfied.

But everyone doing this in densely populated areas will have consequences. Here it works best in the northern parts of country where air-source heat pumping is out of question and population density is near zero as well.

[trobbins]

Update on the investigation in to the commissioning event.
https://mobile.abc.net.au/news/2021-09-28/fire-at-tesla-giant-battery-project-near-geelong-investigation/100496688

Commissioning has its risks!  Perhaps exacerbated by contractual pressure to get through written test procedures on individual blocks without having someone on site with an intimate awareness of underlying risks when all the safeguards are not in place (due to the act of commissioning).

[sandalcandal]

Update on the investigation in to the commissioning event.
https://mobile.abc.net.au/news/2021-09-28/fire-at-tesla-giant-battery-project-near-geelong-investigation/100496688

Fire at Tesla giant battery project near Geelong was likely caused by coolant leak, investigation finds
[...]
An investigation conducted by Energy Safe Victoria found the "most likely" cause of the fire to be a coolant leak in the Megapack cooling system, which caused a short circuit that led to a fire in an electronic component.
[...]
ESV said several changes had since been made to prevent any future fires, including each Megapack cooling system being inspected for leaks before on-site testing, and the introduction of a new "battery module isolation loss" alarm to firmware.

As for what happened here, the Tesla battery pack designs are liquid cooled which should make them incredible tolerant to fire, as long as coolant is present (not even pumped, just sitting in and filling the coolant lines). Given this occurred during construction, my guess is that during commissioning there was some fault with coolant not being in the system whether that be insufficient or not at all filling the coolant or a coolant leak then during a commissioning load testing the pack with faulty coolant fill could have overheated and caught fire. The propagation testing only guard against limited individual cell failure, not a whole pack overheating.

Regardless, something went very wrong with the system/installation process.

Vindication! Although they attribute the coolant failure more to causing the catastrophic failure via short circuit rather than allowing failure propagation.

[trobbins]

I wouldn't SHOUT about your assumed vindication as your proposed fault path was not correct.  We only see a sanitized snippet of what happened and why.  It seems to me that far more than one 'fault' led to this event, given they only owned up to about 3-4 cascading faults.

[sandalcandal]

I wouldn't SHOUT about your assumed vindication as your proposed fault path was not correct.  We only see a sanitized snippet of what happened and why.  It seems to me that far more than one 'fault' led to this event, given they only owned up to about 3-4 cascading faults.

My proposal is that a coolant system failure was the "primary" cause of failure. This would have lead to the currently suspected short circuits (though I think this may change) as well as cascading faults of thermal failure propagation. For the coolant system failure to be allowed to propagate, there must have also been a failure (or lack of) coolant system monitors.

[EEVblog]

Source:
https://esv.vic.gov.au/news/cooling-system-leak-led-to-victorian-big-battery-fire/
and
https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

[EEVblog]

https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

"The affected Megapacks failed safely despite total loss"

I'm not sure I see how they "failed safely" when one of them caught on fire due a coolant fault? 

[sandalcandal]

Source:
https://esv.vic.gov.au/news/cooling-system-leak-led-to-victorian-big-battery-fire/
and
https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

Those reports look pretty final.

Root cause
The most likely root cause of the incident was a leak within the Megapack cooling system that caused a short circuit that led to a fire in an electronic component. This resulted in heating that led to a thermal runaway and fire in an adjacent battery compartment within one Megapack, which spread to an adjacent second Megapack.

The Megapack that caught fire had been in service for 13 hours before being switched into an off-line mode when it was no longer required as part of the commissioning process. This prevented the receipt of alarms at the control facility.
• A key lock was operated correctly to switch the Megapack to off-line service mode (which was no longer required for ongoing commissioning) but this caused:
o telemetry systems for monitoring the condition of the (now out of service) Megapack to shut down and so remove visibility of the developing event
o the battery cooling system to shut down
o the battery protection system to shut down, including the high voltage controller (HVC) that could have operated a pyrotechnic fuse to disconnect the faulty battery unit.

Conclusion
The incident was most likely initiated by a Megapack coolant leak. The absence of a number of monitoring and protection systems that would have been available had the initial Megapack not been  ​subsequently switched to off-line service mode allowed the initial fault to go undetected and resulted in the total loss of two Megapacks.
The affected Megapacks failed safely despite total loss

ESV requires Tesla to provide the final results of its investigation (when available) into why the fire resulted
in the loss of a second Megapack and what it is to do to prevent that circumstance arising again.

"The affected Megapacks failed safely despite total loss"

I'm not sure I see how they "failed safely" when one of them caught on fire due a coolant fault? 

I think they mean no one (public, workers or emergency services) was hurt or exposed to unmanaged hazard as a result.

[trobbins]

The murky and unclear statement imho is "The most likely root cause of the incident was a leak within the Megapack cooling system that caused a short circuit that led to a fire in an electronic component".  Given that the megapack was being tested for 13 hours prior, and apparently passed what was likely some cycling at high rates, that could mean a few suspects for where the short circuit occurred. It would seem that the short circuit caused the battery to feed the fault, and likely at such a rate that battery cooling was required to cope with the fault current (that in itself is suspect as the fault current level may well have been beyond the capability of the cooling system). 

If the megapack was off-line at the time the short-circuit occurred then that may exclude power conversion to the line (unless there was latent heat in power conversion parts that were just surviving from say fan cooling, or from just enough coolant still being available at the end of testing).  Another possibility could be an accessory process such as balancing was automatically kicking in and it assumed cooling was available.

The description of the other subsequent fault items is fairly clear, although I'd have to say that doing commissioning of a megapack with the Scada still having safety related punchlist items open sounds like bordering on negligence.

[Ed.Kloonk]

https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

"The affected Megapacks failed safely despite total loss"

I'm not sure I see how they "failed safely" when one of them caught on fire due a coolant fault? 

Perhaps they mean that it only caught fire and didn't blow someone's head off. Dunno. Failed safely from what? It's no doubt worded so the lawyers and insurance ghouls don't go bonkers.

[floobydust]

What a bunch of bullshit.
"The supervisory control and data acquisition (SCADA) system for a Megapack took 24 hours to ‘map’ to the control system and provide full data functionality and oversight to operators. The Megapack that caught fire had been in service for 13 hours before being switched into an off-line mode when it was no longer required as part of the commissioning process. This prevented the receipt of alarms at the control facility."

You simply poll the status registers all the time, MODBUS works well. You never have datapoints taking 24 hours of magic, what is this fake excuse they had no visibility of the packs?
When I'm doing site commissioning, it's the whole system and alarms that are mapped, never partials.

P.S. Did we ever find out if this is older Lithium or newer Lithium iron phosphate Megapacks?

[EEVblog]

[EEVblog]

Perhaps they mean that it only caught fire and didn't blow someone's head off. Dunno. Failed safely from what? It's no doubt worded so the lawyers and insurance ghouls don't go bonkers.

That's the only conclusion I can gather as well.

[EEVblog]

The big thing for me is given that this overheat occured AFTER the charging stopped, AND the pack was taken offline, what caused the heating that the cooling system had to cool down and couldn't because it was leaky? If no power incoming or outgoing, and the pack is "offline", nothing should be getting hot, surely?

And why isn't the monitoring and safety control done inside the megapack regardless of the remote connection?

[mikeselectricstuff]

The big thing for me is given that this overheat occured AFTER the charging stopped, AND the pack was taken offline, what caused the heating that the cooling system had to cool down and couldn't because it was leaky? If no power incoming or outgoing, and the pack is "offline", nothing should be getting hot, surely?

And why isn't the monitoring and safety control done inside the megapack regardless of the remote connection?

https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

The most likely root cause of the incident was a leak within the Megapack cooling system that caused a
short circuit that led to a fire in an electronic component.

Not lack of cooling - coolant leaking and shorting something

[gmb42]

What a bunch of bullshit.
"The supervisory control and data acquisition (SCADA) system for a Megapack took 24 hours to ‘map’ to the control system and provide full data functionality and oversight to operators. The Megapack that caught fire had been in service for 13 hours before being switched into an off-line mode when it was no longer required as part of the commissioning process. This prevented the receipt of alarms at the control facility."

You simply poll the status registers all the time, MODBUS works well. You never have datapoints taking 24 hours of magic, what is this fake excuse they had no visibility of the packs?
When I'm doing site commissioning, it's the whole system and alarms that are mapped, never partials.

Modbus is rarely used in the electrical generation\distribution world as with polling it's all too easy to miss transient events, unless there is the MacGyver Modbus "solution" of mapping log files into the 16 bit registers.  Generally IEEE 1815 (DNP3), IEC 60870 or IEC 61850 is used in these environments with event reporting.

[sandalcandal]

I think that was a pretty well done video summary of the reports.

As for why the safety/monitoring systems got inactivated, I would presume there is some sort of deactivation or "off" state used for shipping so that the monitoring and safety systems don't deplete the batteries below 0% SoC and cause damage (or danger) during transport and storage. Perhaps this state was improperly reactivated or they don't have a proper "in-commission but inactive" mode?

I have no idea about the SCADA stuff. I've no experience with these larger scale industrial plant systems.

[daqq]

Maybe very dumb solution to neighbouring racks catching fire:

Would some emergency spring loaded ejection system work? When overheating/fire is detecting, shove the module(s) outside of the rack? Possibly into some kind of concrete ditch in front of the system filled with sand/gravel?

[sandalcandal]

As for the inactive cooling system leak allowing fire, it makes sense that an external (to the batteries and coolant system) short circuit and heat generation event could cause sufficient heat input to overcome passive safety design for thermal failure propagation. I would have thought they would design any potential system which could be exposed to a coolant leak to not cascade into this sort of runaway thermal failure. Perhaps there are such systems in place but they were improperly deactivated? It seems they are changing it so these safety systems which could have prevented catastrophic failure can't be inadvertently deactivated.

The following actions have been put in place to prevent a recurrence of this incident.
[...]
*The high voltage controller (HVC) that operates the pyrotechnic fuse remains in service when the key lock is isolated.

[EEVblog]

The big thing for me is given that this overheat occured AFTER the charging stopped, AND the pack was taken offline, what caused the heating that the cooling system had to cool down and couldn't because it was leaky? If no power incoming or outgoing, and the pack is "offline", nothing should be getting hot, surely?

And why isn't the monitoring and safety control done inside the megapack regardless of the remote connection?

https://esv.vic.gov.au/wp-content/uploads/2021/09/VBB_StatementOfFindings_FINAL_28Sep2021.pdf

The most likely root cause of the incident was a leak within the Megapack cooling system that caused a
short circuit that led to a fire in an electronic component.

Not lack of cooling - coolant leaking and shorting something

Yep. Read that wrong.
The power to sustain it must have come from the battery, so it would be interesting to know if they safety systems that disabled when it went offline could have prevented that.
They didn't say that, but it's a "contributing factor".

[EEVblog]

As for why the safety/monitoring systems got inactivated, I would presume there is some sort of deactivation or "off" state used for shipping so that the monitoring and safety systems don't deplete the batteries below 0% SoC and cause damage (or danger) during transport and storage. Perhaps this state was improperly reactivated or they don't have a proper "in-commission but inactive" mode?

They would likely have something for transport, but I'd imagine that would be a big physical interlock or something? Once on site plug in the big red jumper link.

[trobbins]

I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit.  I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.

There is also the concern that the DC path did not include passive fusing or CB, as only the use of a pyro fuse was identified.  Perhaps the short circuit was not bolted per se, but if it wasn't a substantial over current then why did the battery need cooling - maybe there was a current range in the middle.

[mikeselectricstuff]

I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit.  I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.

Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length

[T3sl4co1l]

Between carbonization and electroplating, a substantial short circuit could be formed.  Remember this is DC, so ionic transport is a thing, and ions can be plated off one electrode onto the other, usually in a haphazard way (in a foamy deposit due to excess hydrogen production, or in dendritic crystals without an agent to encourage smooth deposits).  There shouldn't be much electrolyte available, at least not initially, but that just means it'll take longer and have higher resistance; and cell voltage is in no short supply here.

The coolant leak could've been among the battery itself, which logically should occupy the bulk of internal volume regardless of where the controls/inverter are placed, and obviously has to have lots of metallic connections, not necessarily exposed connections, but would they take the time to seal everything up when it's already supposed to be nicely contained in pipes, and the enclosure(s) already handle rainwater?

Tim

[floobydust]

So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.

The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?

In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.


"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."

So we don't why the leak existed and how it could not happen again 

[wraper]

So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.

The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?

In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.


"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."

So we don't why the leak existed and how it could not happen again 

You've seen the word "FIRMWARE", completely ignored the rest and made a huge sensation out of it  . Firmware change mentioned was just another thing among several measures to improve safety. Not to say watch Dave's video or read the actual article by NSV rather than watching CNBS.

The following actions have been put in place to prevent a recurrence of this incident.
• Each Megapack cooling system is to be fully functionally and pressure tested when installed on site and
before it is put into service
• Each Megapack cooling system in its entirety is to be physically inspected for leaks after it has been
functionally and pressure tested on site
• The SCADA system has been modified such that it now ‘maps’ in one hour and this is to be verified
before power flow is enabled to ensure real-time data is available to operators
• A new ‘battery module isolation loss’ alarm has been added to the firmware; this modification also
automatically removes the battery module from service until the alarm is investigated
• Changes have been made to the procedure for the usage of the key lock for Megapacks during
commissioning and operation to ensure the telemetry system is operational
• The high voltage controller (HVC) that operates the pyrotechnic fuse remains in service when the key lock
is isolated

Conclusion
The incident was most likely initiated by a Megapack coolant leak. The absence of a number of monitoring
and protection systems that would have been available had the initial Megapack not been subsequently
switched to off-line service mode allowed the initial fault to go undetected and resulted in the total loss of two
Megapacks

[floobydust]

Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.

Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?

[nctnico]

Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.

Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?

Well, that is why these battery units are designed to spit flames upwards and not to the side. In the end there is only so much you can do before a design gets very expensive. OTOH it is likely that the design goes through some further iterations but that is normal for any product.

[somlioy]

Hah, the same thing happened on a norwegian ferry with a 2MWh battery pack, where the battery was taken out of service as well, so the crew didnt get any warning on the control system before the thermal runaway was a fact.

A gasket in the cooling system was leaking causing a short circuit. In addition an explosion happened due to the seawater from the sprinkler system.

https://www.ctif.org/news/explosion-norwegian-battery-hybrid-ferry-may-have-been-caused-fire-extinguishing-system

[trobbins]

somlioy, that link does not identify 'the same thing happened'.  Certainly there was a similarity that alarms did not annunciate, but for different reasons.

[floobydust]

Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve

It's terrible there is so much pressure, perhaps too much on new renewables technology. The politics for a stable grid before summer hits.
Tesla is shaping things, not in a great way. I don't like the disposable/non-repairable Megapacks and the "consumer grade" design they rolled out. No firewall, too late to increase cabinet spacings and no thought of a coolant leak occurring in the design. These fires will be happening again.

1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.

[T3sl4co1l]

1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.

It wouldn't be too far out of place, I suppose -- recall what their first cars were like.  The products themselves had a number of issues, like panel alignment; everything was thrown together by hand, quality control was poor.  (They still do, with things like placement of the 12V battery being a microcosm of anything else I'm sure.)  Production was even more fraught, with frequent delays and poor utilization of automation systems.  The "throw everything at a wall and see what sticks" and "continuous integration" mentality of the software/startup crowd (and owner) was blamed for this.

Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market.  Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.

Tim

[trobbins]

Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve

It's terrible there is so much pressure, perhaps too much on new renewables technology.

I think there are two sad aspects of the recent windfarm legal battle.  Firstly, the HV distribution tower failures that initiated the wide area blackout shouldn't have happened - but the designers of those towers don't appear to have been brought in to public scrutiny.  Secondly, the windfarm disconnect scheme had obviously been tested and approved and given the authority to connect, but that aspect also appears to not have been brought in to public scrutiny, only the aspect that the wind farm operators should have programmed the disconnect scheme to endure more than six separate trip events before actually disconnecting.

[EEVblog]

I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit.  I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.

Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length

I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.

[sandalcandal]

Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve

FYI official press release here: https://www.aer.gov.au/news-release/hornsdale-in-court-for-inability-to-provide-contingency-services-as-offered

As per the above official statement, the issue is related to power not being supplied as stipulated due to a power failure of Kogan Creek Power Station (Coal fired power plant in Queensland). That's over 1 500 km (930 miles) away. I saw a comment else where saying the issue seemed more due to network capability and distance rather than Neoen.

Edit:
Similar lawsuits seem pretty common from AER checking their press releases here: https://www.aer.gov.au/news?f%5B0%5D=type%3Aaccc_aer_news Most of these are unreported in the media however. Probably just getting extra attention due to the usual Tesla/Musk perversion.

Issue might be inaccurate modelling of system capability over the Australian distribution network causing them to over bid and under deliver?
Edit2: See Trobbins' comment below.

[sandalcandal]

I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.

Metal cover holds the radiator system and maybe pumps I presume.

[sandalcandal]

Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market.  Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.

Yeah, automotive and aerospace markets are so lax in comparison

[trobbins]

Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision.  I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC.  I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support.  For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).

[sandalcandal]

Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision.  I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC.  I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support.  For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).

Yeah, sounds reasonable. I'm not too familiar with grid scale systems.

Given it has gone to court and Neoen isn't just paying a fine would that indicate there is some contention (or Neoen thinks it can otherwise reduce its penalty)?

[askji432d]

the radiator was inside metal cover.

[EEVblog]

I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.

Metal cover holds the radiator system and maybe pumps I presume.

It's moot anyway because that's the older PowerPack. The Megapack seems to be an entirely different design.

[JohnG]

engineers are used to very relaxed schedules --

 

[thinkfat]

Stack them.

[f4eru]

Burn, Baby, Burn

[Marco]

Who would want to be the general contractor doing site design and install? It's too risky.

Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.

If it was Tesla's fault, they can sue Tesla in return.

[wraper]

Who would want to be the general contractor doing site design and install? It's too risky.

Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.

If it was Tesla's fault, they can sue Tesla in return.

If you are consevative, you end up with stagnation and outdated infrastructure which is falling apart due to its age.

[TimFox]

Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.

[wraper]

Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.

Except it's a huge waste of effort and money to keep it maintained for for any practical purpose https://nypost.com/2016/11/11/brooklyn-bridge-repairs-expected-to-cost-811m/ The only value it has is architectural and historical. Not to say it was actually a quite innovative and beat the records of it's time. Not conservative at all as it was an early example of a steel-wire suspension bridge. Innovative stuff may fail here and there due to not yet known issues which may become apparent later, but it's a path to rapid improvements, and better results overall. FWIW in other circumstances it would be more effective to simply replace it, as most of outdated infrastructure.

[Siwastaja]

Maybe very dumb solution to neighbouring racks catching fire:

Would some emergency spring loaded ejection system work? When overheating/fire is detecting, shove the module(s) outside of the rack? Possibly into some kind of concrete ditch in front of the system filled with sand/gravel?

I love such overengineered fantasies when the reality shows that they fail at even such trivial thing as just reporting overheating/fire!

[thinkfat]

Just launch the complete pack into orbit. I'm sure SpaceX can lend a hand.

[TimFox]

Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.

Except it's a huge waste of effort and money to keep it maintained for for any practical purpose https://nypost.com/2016/11/11/brooklyn-bridge-repairs-expected-to-cost-811m/ The only value it has is architectural and historical. Not to say it was actually a quite innovative and beat the records of it's time. Not conservative at all as it was an early example of a steel-wire suspension bridge. Innovative stuff may fail here and there due to not yet known issues which may become apparent later, but it's a path to rapid improvements, and better results overall. FWIW in other circumstances it would be more effective to simply replace it, as most of outdated infrastructure.

One example of the conservative design approach to the Brooklyn Bridge is that Roebling estimated the "fudge factor" needed for specifying the cables that would be required since he knew political operators would cheat on the procurement.  The non-conservative design approach that doomed the Quebec Bridge was that the engineer remained in New York during construction, and had calculated the stresses with "theoretical" values of the structure weight (from the original design drawings) and did not confirm the weight from the detailed fabrication drawings.  That, and the construction supervisor ignored reports that the structure was yielding and sent workers out on the bridge: they would be out on the bridge when it failed with substantial loss of life.
So, maintenance repairs to the Brooklyn Bridge are estimated at $811 M.  Did the article mention how much a replacement would cost?  The Oakland Bay Bridge recently cost $6.4 B to replace.

[wraper]

The Oakland Bay Bridge recently cost $6.4 B to replace.

Don't underestimate the ability of California to blow the money. FWIW they are building the slowest and the most expensive in the world high speed rail system. And the costs will probably triple over current $80B estimate by the time they finish it (if they ever do), considering their track record.

[TimFox]

And you think the State of New York will be any more cost-efficient?

[floobydust]

Had to laugh, the Hornsdale court case was hinging on the "right to not generate" electricity. Renewables are coming in too late, they are mission critical when they really shouldn't be if the Grid was well-managed, had sufficient generation and reserve capacity, fat cats actually forecasted demand increases and acted accordingly.


The location of the radiator, fan and coolant pump seems to be the main difference between PP and MP.
MP has the radiator and fan located at the top of the cabinet, whereas the PP has it in the door with the distinguishing door grille. The battery packs seem to need names?

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

Boeing 787 lithium batteries they added ceramic spacers to block fire?
The irony, Musk 2013 tweet "Maybe already under control, but Tesla & SpaceX are happy to help with the 787 lithium ion batteries."

[wraper]

Boeing 787 lithium batteries they added ceramic spacers to block fire?
The irony, Musk 2013 tweet "Maybe already under control, but Tesla & SpaceX are happy to help with the 787 lithium ion batteries."

Very funny, especially considering that fire did not start at batteries as it was in multiple 787, and where battery was extremely tiny in comparison.

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

You literally have no idea how they are actually built, yet act if you are an expert in the field.

[floobydust]

It's a battery pack, not a space ship. You can look at the relevant patents. Tesla didn't start from scratch, you can see the Powerpack design internals and my assumption is they just upsized to the larger cells with minor changes.
None of this matters because at the end of the day, the cabinet burned up with only guesses as to why.

From my armchair, the battery pack spacings are terrible with no fire breaks, and nothing for the firefighters to do but sit around and watch it burn out. Tell me how great this is and how it will never happen again.
 
Megapacks are being sold like hotcakes all over the globe, if the design went too far we'll know within a year.

[T3sl4co1l]

One example of the conservative design approach to the Brooklyn Bridge is that Roebling estimated the "fudge factor" needed for specifying the cables that would be required since he knew political operators would cheat on the procurement.

Don't underestimate the ability of California to blow the money. FWIW they are building the slowest and the most expensive in the world high speed rail system. And the costs will probably triple over current $80B estimate by the time they finish it (if they ever do), considering their track record.

As projects grow in size, and thus political visibility, never underestimate our* capacity to skim off a little more, here and there, for our friends/contractors/cronies.  This effect, probably even more so than the direct benefits (or arguable lack thereof, across various similar proposals), is a huge reason why such projects may never get built in this country.  It's a systemic issue, and addressing that first, would seem to be a prerequisite to actually building these things.

*US specifically, yes, but it happens everywhere, democracy or otherwise.  Tendering contracts among friends, is the democratic version of more overt corruption under dictatorships (ref: _Rules for Rulers_).  Different mechanism, and absolutely, at a different rate/amount -- but the same basic effect.  So, ultimately it would seem, by "our", I mean humans in general.  Or at least, any large-scale system humans have yet developed.  So, this isn't a political point, or at least any more so than concerns that relative amount.

Tim

[sandalcandal]

The MP battery pack is much larger and spacings very tight between them in the rack. Somebody can do the math of the MP cabinet heat generated with 16 battery packs and high 80-90% efficiency of the batteries and the boost-converter, I wonder if they bungled the thermal design doing the copy'n'paste from the smaller PP for plumbing. The battery packs I have not seen a pressure relief valve, it could overpressure the common cooling system which would cause more drama. PP battery packs had lots of tubing with hose barbs inside.
Could not be made any smaller really. Great approach for designing cars, cellphones, laptops... but I say they need increased spacing and firewalls. Tesla's Lego block approach does not infinitely scale up.

Modern power electronics, particularly for these high end systems goes >90% with >99% efficient AC-DC systems available. The datasheet posted at the start of this thread has 87% to 90% round trip efficiency for 0.5E to 0.25E loading so efficiency each way would be \$\sqrt{0.87}=0.933\$ so ~93.3% efficiency each way.

So far as thermal design needing to be completely redesigned between the two, both are rated for the 0.5E to 0.25E loading so I don't see how that's a requirement, even then the cooling system setup is clearly not the same. The Powerpack datasheet has round trip efficiency rated at a slightly lower 85.5% to 89.5%.

Not that any of that matters in this case since the fire had nothing to do with operational heat load and was apparently due to an uncontrolled failure with resultant heat from the failure.

The design, layout and density of the Tesla systems don't look that far off other system I've seen but I wanted double check so here are some other grid scale systems below.

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Attached below:
LG Chem Rack System R1000- UPB4860  https://greenpowerco.com.au/products/batteries/lg-chem/lg-chem-rack-system-r1000-upb4860-gen2/

Samsung SDI http://www.samsungsdi.com/upload/ess_brochure/201803_SamsungSDI%20ESS_EN.pdf

ABB Enviline https://new.abb.com/medium-voltage/switchgear/railway-switchgear/dc-traction-power-supply/energy-management/enviline-ess

Fluence (Siemens AG and The AES Corporation) Cube https://fluenceenergy.com/energy-storage-technology/gridstack-grid-energy-storage/ (There's also photos of a very large system floating around but I can't find what it actually is. Only official use is as a banner on their about page)

GE Reservoir Storage Unit https://twitter.com/generalelectric/status/1154428597895122944/photo/1

[sandalcandal]

1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.

Again as per the datasheet, the supported protocols are Modbus TCP / DNP3 / Rest API.

[trobbins]

The SCADA mapping issue could have been due to using default settings for aspects like how long to wait for a response, or a myriad other technical and comms issues that obviously hadn't been adequately identified and then ticked off the punchlist.  This could also easily have been using a different PLC/software combination than previously used, which would reset prior experiences.

There are always stressful installation and commissioning timelines and milestones, whatever the project, and whatever the key reasons, and sadly decisions are made to keep going with not enough non-Tesla high level experience to put hold's in place for certain aspects.  There may also have been a lack of awareness of past megapack issues, such as can only be gained from other site visits by non-Tesla people when those sites have their key people available and who are not constrained to be open.

[EEVblog]

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Only Megapack photo I can find is:



Which looks nothing like their wanky render:

[sandalcandal]

Tesla Powerpack again for reference and we don't have any Megapack photos yet.

Only Megapack photo I can find is:
Which looks nothing like their wanky render:

Yeah, no real "complete" photos of the innards.

[EEVblog]

Twitter comes though:
https://twitter.com/sillybaboy/status/1443823137678897164

Video:

https://globalnews.ca/video/rd/92b2b6fa-30ce-11ea-9e92-0242ac110004/?jwsource=cl

[sandalcandal]

Twitter comes though:
https://twitter.com/sillybaboy/status/1443823137678897164

Video:

https://globalnews.ca/video/rd/92b2b6fa-30ce-11ea-9e92-0242ac110004/?jwsource=cl

Great stuff. Can see coolant lines going off to some manifold on the left, RJ45 daisy chaining between modules and it looks like the control systems are powered by an external 24V bus running off to the right. If they are powered by an external supply rather than deriving power internally from each battery pack then it makes a bit more sense the monitoring systems for the whole thing could be disabled at a single point. I guess these could also potentially be inverter modules?

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

[jonovid]

from seeing Dave's video about this  Telsa Victoria Big Battery Fire REPORT

so it takes 12 to 24 hrs just to log a coolant system feature! I find that level of poor design of a remote location telemetry system just astonishing!
it just as well tesla is not building or running nuclear reactors.   

[gmb42]

from seeing Dave's video about this  Telsa Victoria Big Battery Fire REPORT

so it takes 12 to 24 hrs just to log a coolant system feature! I find that level of poor design of a remote location telemetry system just astonishing!
it just as well tesla is not building or running nuclear reactors.   

Are Tesla responsible for the SCADA?  I would assume that's someone else, the MegaPacks are the field devices.  The issue could still be those field devices being slow to respond, but from the other reports, it seems that the SCADA system isn't very performant.

Does anyone know what the SCADA software is?

[floobydust]

Look at volume of data in one SCADA scan. How many analogs and digitals should there be from a single battery module? Do a back of the napkin calculation.
V, I, temperature, inverter V, I this that, alarms, status etc. Say 20 points and avg. 12 bytes/point in packets, that's about 240 bytes of data.
Multiply by 16 battery modules per cabinet, 11 cabinets/MP row and 2 rows back-back, and the site has 40 of 150 MP's in the commissioning phase. I think 14,080 battery modules so far? Gees that's a lot.
My rough estimate is 3.2MB of data every poll of 281,600 points. If you are polling over serial at 9,600 that is an hour lol.
With my math, it will be almost 4X larger/slower when the site is finished, 1 million points...

For reliability you would not use a single network, there would be many. Large utility sites can have a high failure rate for RS-485 transceivers, one short from a ground-fault or lightning etc. and you lose comms. To limit the loss of visibility and control, there would be multiple networks.

DNP is an antique utility industry (substation) protocol from the days of dial-up. It's stuck around and I think it's a gross protocol as DNP3 due to the drama with standardization, that it's obfuscated and inefficient. IEC 61850 over Ethernet is the modern way to go. But this is the utility industry. MODBUS is bare-bones minimum and fine over TCP/IP.

How Neoen decided to go... you would never use a PLC or low-cost simple HMI system due to the volume of data. Serial is way too slow here.
Over a million points I estimate, which is the league of distributed control systems DCS. I don't know of any factory or plant with that many points to read and log and display on an HMI. It's never done on a single computer system, it would be broken up but this costs money.

I'd say they (Neoen) went too cheap and ended up with turtle bus. After the fire, they've speeded it up to a one hour scan time, seems terrible. You won't know there is a fire alarm coming in, faster to just look for smoke?

[gmb42]

For systems such as this you wouldn't get in the door with your Modbus ideas, polling can easily miss transitions which are somewhat common and important in the electrical monitoring world.  You need event buffering and reporting, i.e. what DNP3 was designed for.

Yes, DNP3 has not adapted the data frame from the old serial days, so the 2 byte CRC for every 16 bytes of data is useless overhead when run over TCP, but then Modbus TCP frames drop the Station address and the CRC and add in the transaction ID, protocol ID and length words to make a frame with more overhead than the serial equivalent.  Both protocols still retain their small data size per packet so the raw frame size is a toss up.

With DNP3 you poll for events, much more efficient than Modbus polling, and if you enable unsolicited mode on the devices you don't even need to do that, they report events of interest.

Practical real world experience has shown me that effective data update rates in large systems are way better with DNP3 than Modbus.  When using polling protocols such as Modbus, SCADA systems end up having to implement "fast scans" to increase the poll rate on devices of interest, i.e. when viewing the info for a particular substation (reducing it for other devices) to satisfy operator requirements.

DNP3, due to it's longevity, is pretty well understood and implemented, there are always exceptions (I'm looking at you, Rockwell ML-1400 p.o.s) and thus is still specified and implemented in the industries that suit it.  There now seems to be a move to use DNP3 in water\waste water systems as pump stations are becoming increasingly complex and data "rich", and Modbus and other polling protocols just don't cut it.

[floobydust]

In the pics of the Saint John Energy (NB) Megapack, I can see daisy-chained RJ45's so it's a multidrop network between the cabinet's battery modules, and who knows how far they took that, if there is a data concentrator somewhere taking Megapacks beyond sluggish serial CAN or RS-485, up to Ethernet.
Improving things from 12-24hr to 1hr poll is great but still pretty much a bandwidth disaster... and the site is 1/4 finished. The LG system advertises supporting CAN2.0 and MODBUS TCP/IP. It should be ok with Ethernet speeds.

I don't think anyone has thought it through, how much data is there. Substation controls, decisions are done locally i.e. in the relays and devices, with the SCADA/HMI "eyes only".
So no decisions or logic/controls are running at the top level. If there's an overcurrent, the breaker trips on its own. Of course there are trip/close, local/remote, auto/manual switches on the HMI for operators to actuate if needed.

If the SCADA/HMI went down, the site would keep running.

[EEVblog]

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

Given that the render doesn't match the other panels, I wouldn't put any stock in it.

[sandalcandal]

No views of the inverter or radiator/pump system unfortunately. I note there is something that looks like a motor/pump in the bottom left of the renders. Also didn't get a better view of the "subpanels" which seem to sit in front of some rack modules.

Given that the render doesn't match the other panels, I wouldn't put any stock in it.

Really? I think the new images/video confirm the renders are accurate.

We can clearly see that mostly empty cabinet with what appears to be the main isolator switch which looks pretty much exactly like the renders. Right next this cabinet in the video (in the same location as the renders), they also show another cabinet filled with rack modules without the "subpanel" covering that look near identical to the render. Then as per my last post we can also see the same panels covering the front of some rack modules. The only differences seem to be some of the wiring in the main switch panel is still incomplete and missing wires/coolant lines for the middle right rack cabinet shown.

[floobydust]

The site is apparently now up and running  https://www.centralwesterndaily.com.au/story/7543112/victorian-big-battery-switched-on/
Not sure when it's really needed and will actually do some work to prove itself.

[Someone]

Not sure when it's really needed and will actually do some work to prove itself.

Battery (hydro, and other short term methods of) Storage "works" best when its being utilized every day, which it is doing:
https://opennem.org.au/facility/au/NEM/VBB/?range=7d&interval=5m
They'll likely want to deepen the cycling but much of the power capacity can be held back for FCAS:
https://reneweconomy.com.au/victoria-big-battery-now-registered-as-biggest-non-hydro-player-in-fcas-market/

[trobbins]

Someone, thanx for that opennem website link.  It will be interesting to see how Moorabool progresses, and whether it will be tasked with more aggressive daily energy shifting like Hornsdale is doing.

[Someone]

Someone, thanx for that opennem website link.  It will be interesting to see how Moorabool progresses, and whether it will be tasked with more aggressive daily energy shifting like Hornsdale is doing.

Looks like it did over 50% depth of discharge across the 09/01/2022, then similar on the 13th. They dont seem to be particularly reactive to market pricing, so are playing it safe/friendly.

[floobydust]

Did you have any heatwaves with related grid shortages?

Here, a cold snap where it was dancing around -30°C for three weeks and we ran out of electricity, demand around 13GW and pool price peaked over $1.00/kWh 
It's mainly people plugging in ICE car block heaters ~600W at night so their car will start, or electric heaters. Coal-fired plants were the usual workhorses, solar is terrible in cloudy winter weather, not much from wind.
Mental note: build a heat pipe from Canada to Australia.

[Faringdon]

people plugging in ICE car block heaters ~600W at night so their car will start

Thans interesting , its a shame they dont put these on a timer and have them warm up the car an hour before drive time....or does the car need to be kept warm all night i wonder

[Siwastaja]

people plugging in ICE car block heaters ~600W at night so their car will start

Thans interesting , its a shame they dont put these on a timer and have them warm up the car an hour before drive time....or does the car need to be kept warm all night i wonder

One hour in advance is really enough unless the engine or battery is pure shit. But people are lazy and stupid and simply do not care.

My engine block heater is broken and blows the fuse, but the car starts just fine at -25degC. Yeah, it might not be optimum for the engine or emissions, so better would be to heat it up for an hour beforehand. But keeping it on all the night is massive waste.

You don't even need a timer, just plug it in and start making breakfast.

Though, if everybody drives to work at the same time, then the heaters are all on at the same time, too. Maybe just heat for 30mins, that's better than nothing at all, and spreads out better.

Here, a cold snap where it was dancing around -30°C for three weeks and we ran out of electricity, demand around 13GW and pool price peaked over $1.00/kWh

Had the same thing here in December. Close call on energy sufficiency, pool price peaked at 1.24€/kWh, that's 1.42USD! We are used to 0.05€/kWh energy pricing so obviously this pissed people off. Meanwhile, government is heavily subsidizing removal of oil burners and transition into electric heating. At -25degC if not earlier, all those fancy air source heat pumps have turned into COP=1 direct electric heaters. I'm still running oil in parallel with the heat pump during <-18degC, and it even has lower CO2 footprint than the grid has during those peak days, when back-up stations are brought up, so it's win-win for absolutely everybody/-thing, climate included, but it isn't media sexy, even if you just burn 200 liters of oil for heating, per year, to avoid burning a lot larger equivalent amount of coal elsewhere!

[Marco]

Battery (hydro, and other short term methods of) Storage "works" best

Best being most economical. At some point best and most economical will have to start meaning long term storage if all the renewable plans go through, at which point short term storage will likely become uneconomical because you have the sunk cost of the long term storage any way.

[Someone]

Battery (hydro, and other short term methods of) Storage "works" best

Best being most economical. At some point best and most economical will have to start meaning long term storage if all the renewable plans go through, at which point short term storage will likely become uneconomical because you have the sunk cost of the long term storage any way.

Extreme way to intentionally misrepresent the quote, taking the original:

Not sure when [the battery is] really needed and will actually do some work to prove itself.

Battery (hydro, and other short term methods of) Storage "works" best when its being utilized every day, which it is doing....

If you think long term storage is so viable, show us the figures. Over provisioning of generation (as grids have traditionally done) still looks to be cheaper. Doesn't matter what the generation sources are. Even if long term storage becomes a thing or massive over provisioning is done, it won't remove daily (or day to day) fluctuations that battery and hydro storage will still daily cycle against and profit from.

Unless you are imagining some perfectly ideal long term storage that will eat 100% of excess power through the "off demand" season and perfectly return it as required during the "on demand" season. In which case your model is too simplistic and idealistic to actually work in the open market. At a basic level that would oversize the generation capacity of the hypothetical long term storage plant, that could likely be more economically done with a smaller peak load/generation along side short term storage to flatten it out.

[trobbins]

So far this summer there have not been any significant long duration hot periods in Victoria (the state the Moorabool battery is in) or in nearby states which are grid interconnected, whereby the news has indicated peak grid power issues.  Luckily our instantaneous grid demand from hot weather from air-conditioning is somewhat aligned to PV production.  Although imho we have a long way to go before a large majority of people better align their daily consumption events to PV generation (even with a large grid distribution to average out the PV contribution), including aircon use as a partial thermal store buffer during the middle of the day.  It seems that some of the commercial PV farms are using trackers to optimise the revenue from generating in to the early morning and late afternoon peak consumption demands, but I don't see residential households specifically aiming for that type of operation yet (eg. panels on east and west facing roof sections given inverters typically have 2 MPPT inputs, rather than just on north facing roof sections).