I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit. I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.
Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length
Between carbonization and electroplating, a substantial short circuit could be formed. Remember this is DC, so ionic transport is a thing, and ions can be plated off one electrode onto the other, usually in a haphazard way (in a foamy deposit due to excess hydrogen production, or in dendritic crystals without an agent to encourage smooth deposits). There shouldn't be much electrolyte available, at least not initially, but that just means it'll take longer and have higher resistance; and cell voltage is in no short supply here.
The coolant leak could've been among the battery itself, which logically should occupy the bulk of internal volume regardless of where the controls/inverter are placed, and obviously has to have lots of metallic connections, not necessarily exposed connections, but would they take the time to seal everything up when it's already supposed to be nicely contained in pipes, and the enclosure(s) already handle rainwater?
Tim
So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.
The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?
In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.
"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."
So we don't why the leak existed and how it could not happen again
So far I haven't seen a decent excuse for the fire... and it's a firmware fix with a hardware issue outstanding.
The Megapacks are are a massive blob of glue, you'd think pressure-tested for their coolant section leaking?
In submerged products, I put in a dummy wire or pcb to detect water. Any ghost voltage or leakage current and you know water is there and can send an alarm before the impending doom. Another approach is, assuming the coolant loop is galvanically isolated, is look for leakage current there.
"A Neoen spokesperson told CNBC that “the cause of the fire was identified as coinciding short circuits in two particular locations likely initiated by a coolant leak external to the battery compartment.” the spokesperson said.
“These occurred while the Megapack was off-line in a service mode that removed fault protections. Enabled by this unlikely sequence of events the fault was able to go undetected and initiate a fire in the adjacent battery compartment.”
Tesla took "mitigating actions" and has implemented changes to the Megapack firmware after conducting an analysis, according to Neoen."
So we don't why the leak existed and how it could not happen again
You've seen the word "FIRMWARE", completely ignored the rest and made a huge sensation out of it
. Firmware change mentioned was just another thing among several measures to improve safety. Not to say watch Dave's video or read the actual article by NSV rather than watching CN
BS.
The following actions have been put in place to prevent a recurrence of this incident.
• Each Megapack cooling system is to be fully functionally and pressure tested when installed on site and
before it is put into service
• Each Megapack cooling system in its entirety is to be physically inspected for leaks after it has been
functionally and pressure tested on site
• The SCADA system has been modified such that it now ‘maps’ in one hour and this is to be verified
before power flow is enabled to ensure real-time data is available to operators
• A new ‘battery module isolation loss’ alarm has been added to the firmware; this modification also
automatically removes the battery module from service until the alarm is investigated
• Changes have been made to the procedure for the usage of the key lock for Megapacks during
commissioning and operation to ensure the telemetry system is operational
• The high voltage controller (HVC) that operates the pyrotechnic fuse remains in service when the key lock
is isolated
Conclusion
The incident was most likely initiated by a Megapack coolant leak. The absence of a number of monitoring
and protection systems that would have been available had the initial Megapack not been subsequently
switched to off-line service mode allowed the initial fault to go undetected and resulted in the total loss of two
Megapacks
Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.
Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?
Lots of fixes but the root cause is the important one. Not enough sealant, bad assembly? It's a misleading notion I read in the press article that it's all good, firmware fixes.
It has to leak enough to get the electronics malfunctioning "Safety mode" or MCU control is then no longer an option once it's got water inside. Coolant is going to bridge any mosfet switches until the leads corrode, oh and how does this make a fire? If you're lucky you can send an alarm out.
Plastics are popular for car water pumps, thermostat housings etc. example Audi/Volkswagen plastic thermostat housing leaking is a huge problem. The plastics deform and warp with age.
I'd wonder if these Megapacks won't leak in service. Pressure testing during installation now is great, and a year or two later?
Well, that is why these battery units are designed to spit flames upwards and not to the side. In the end there is only so much you can do before a design gets very expensive. OTOH it is likely that the design goes through some further iterations but that is normal for any product.
somlioy, that link does not identify 'the same thing happened'. Certainly there was a similarity that alarms did not annunciate, but for different reasons.
Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserveIt's terrible there is so much pressure, perhaps too much on new renewables technology. The politics for a stable grid before summer hits.
Tesla is shaping things, not in a great way. I don't like the disposable/non-repairable Megapacks and the "consumer grade" design they rolled out. No firewall, too late to increase cabinet spacings and no thought of a coolant leak occurring in the design. These fires will be happening again.
1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.
1 hour for a SCADA "map" seems like they are using 1,200 baud or LIN bus lol. That's unheard of slowwwwwww. I've worked on big utility projects, generation, substations and the basics for on/off-line behavior and what you're doing during commissioning- this is many levels of amateur mistakes so far. I can't believe they had no SCADA visibility.
It wouldn't be too far out of place, I suppose -- recall what their first cars were like. The products themselves had a number of issues, like panel alignment; everything was thrown together by hand, quality control was poor. (They still do, with things like placement of the 12V battery being a microcosm of anything else I'm sure.) Production was even more fraught, with frequent delays and poor utilization of automation systems. The "throw everything at a wall and see what sticks" and "continuous integration" mentality of the software/startup crowd (and owner) was blamed for this.
Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market. Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.
Tim
Who would want to be the general contractor doing site design and install? It's too risky.
Australia sues Neoen for lack of power from its Tesla {Hornsdale} battery reserve
It's terrible there is so much pressure, perhaps too much on new renewables technology.
I think there are two sad aspects of the recent windfarm legal battle. Firstly, the HV distribution tower failures that initiated the wide area blackout shouldn't have happened - but the designers of those towers don't appear to have been brought in to public scrutiny. Secondly, the windfarm disconnect scheme had obviously been tested and approved and given the authority to connect, but that aspect also appears to not have been brought in to public scrutiny, only the aspect that the wind farm operators should have programmed the disconnect scheme to endure more than six separate trip events before actually disconnecting.
I am at a loss to appreciate how leaking coolant could lead to a (I assume) prolonged and effectively bolted short circuit. I can envisage that circa 1kVdc could lead to a leakage current (due to coolant acting as creepage pollution) that then progresses in to an arc, and perhaps that arc is across a metallic surface that then sustains the arc (or a charred insulation surface), but it beggars the question as to how a coolant system could be designed that a leak could allow such an outcome.
Yeah, you'd think that it should be designed so that that any leak wouldn't fall onto anything that mattered, though maybe putting all the electroics at the top would have led to excessive cabling length
I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.
I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.
Metal cover holds the radiator system and maybe pumps I presume.
Y'know, it must be a huge culture shock, for them to build a product in the power electronics market -- customers have high expectations on reliability, functionality, and especially safety, and engineers are used to very relaxed schedules -- it can take decades for projects to finally enter the market. Obviously, I know nothing about what's inside this thing, how it's built and all -- but the warning signs are definitely there, the same management at the top.
Yeah, automotive and aerospace markets are so lax in comparison
Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision. I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC. I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support. For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).
Sandalcandal, without details on what aspect of their FCAS support agreement was not complied with it is imho too open-ended to comment on with any precision. I'd doubt it was related to issues such as distance to the Kogan Ck gen that glitched, as that doesn't make any sense with respect to grid frequency monitoring and agreed response at the Hornsdale PCC. I could anticipate some types of lack of support compliance could relate to delayed or no support, or not a sufficient level of support. For example, Hornsdale may have been undergoing some level of maintenance or service that it shouldn't have (eg. it may have said it would be on-line and supporting 24/7 except for defined maintenance periods), or had some functional limitation being imposed that it shouldn't have had (eg. too low a SOC due to other money making support agreements).
Yeah, sounds reasonable. I'm not too familiar with grid scale systems.
Given it has gone to court and Neoen isn't just paying a fine would that indicate there is some contention (or Neoen thinks it can otherwise reduce its penalty)?
the radiator was inside metal cover.
I'm trying to find photo of the Megapacks internals, but they are as rare as hens teeth. The only one I found is I posted in this thread a whiel pack.
The old Powerpack design has the tank above a bunch of stuff beind a metal cover.
Metal cover holds the radiator system and maybe pumps I presume.
It's moot anyway because that's the older PowerPack. The Megapack seems to be an entirely different design.
Who would want to be the general contractor doing site design and install? It's too risky.
Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.
If it was Tesla's fault, they can sue Tesla in return.
Who would want to be the general contractor doing site design and install? It's too risky.
Risks can be mitigated and high risk comes with high rewards. If you are being contracted to build something for high reliability backup, be conservative. This wasn't conservative, both in design and timeline to delivery. They need to be disciplined in the only way companies can be disciplined, take their money.
If it was Tesla's fault, they can sue Tesla in return.
If you are consevative, you end up with stagnation and outdated infrastructure which is falling apart due to its age.
Conservatively-designed infrastructure, such as the Brooklyn Bridge (completed in 1883), may not "fall.. apart with age" if properly maintained.
Non-conservative designs, such as the Quebec Bridge (failed in 1907), may collapse before completion.