Dual redundant MCUs for controlling two separate circuits.

[rokspy]

I'm at works of designing a system that has two separate circuits that have to be controlled by separate MCUs. This system should have a relatively high reliability and should occupy as small of a space as possible. Reliability involves the necessity to survive an MCU failure.

I have thought about three MCU voting system, but to control two circuits independently would require six MCUs, which is way too much of a space.
Using a three MCU voting system, where MCUs control both circuits at the same time would get too computationally heavy as well.

My current idea would be to have two MCUs and by default one MCU would be responsible for the one circuit and the other MCU would be responsible for the other circuit. If fault occurs and one MCU gets dead, then the other MCU takes on the responsibilities of the other MCU at reduced capacity.
To implement this, it would be required to have a pair of firmware images for an MCU, each image responsible for controlling a circuit. Then the I/Os of both MCUs would be connected to two multiplexers. Output of one mux is connected to one circuit, output of the other mux is connected to the other circuit.
There would be periodic resets,startup routine and some fault detection circuitry. At the startup, the fault detection circuitry would require a signal from each MCU. if both MCUs answer, the fault circuitry sets the channels of one mux to one MCU and the channels of the other mux to the other MCU. If one MCU does not respond, the fault circuitry sets both mux channels to the MCU that responded. There would be some indication to the MCU from the fault detection circuitry as well.

Does this makes sense??
What are the issues that Im not seeing?
Maybe there are some other alternatives that could help me solve this?

Thanks in advance!

[voltsandjolts]

This system should have a relatively high reliability

Why?

For safety reasons, then try reading IEC 61508 and see if that helps.

If just for operational up-time / cost reasons then you are more free to choose, but the above is still worth reading.

[nctnico]

I'm not sure whether using a dual MCU is going to do much good as adding muxes and extra electronics is going to add additional points of failure. I'd use a highly reliable, extended temperature range microcontroller and make sure all the inputs & outputs are properly protected. Adding more is usually going to end up in becoming less reliable. Several decades ago I worked as an intern at a company that sold & repaired mainframes. At some point we spend days trying to find the cause of why a computer (=full height 19" cabin filled with boards populated with 7400 series logic and fans) kept resetting every now and then. It turned out to be a faulty ADC in the mains voltage monitoring system.

Probably something worth implementing is to bring the system to a safe state in case none of the MCUs is doing something meaningfull. As an example: in a reasonably safety critical system I designed a while ago, I used a watchdog chip to enable a servo control output through a opto switch. Without a control signal triggering the watchdog, the attached servo goes into an idle state. I designed this as minimalistic as possible to make it work reliable without adding too many additonal points of failure.

Either way: I'd research the design of redundant systems carefully. There are likely some good books on the subject.

[SiliconWizard]

If you're going to go redundant, ideally don't put the redundancy on the same PCB. Ideally have two (or more) equivalent systems designed by different people. Yes that's a cost and may be overkill for your application, which we don't know much about.

Using several identical MCUs on the same PCB with the same firmware written by the same people will only cover the probablity that one MCU fails on a hardware level, and will do very little, if nothing for all the other possible failures. In other words, you're probably wasting your time if this is what you have in mind.

[woofy]

As others have said, multiple controllers is likely a waste of effort. What if your majority voting system fails? It's just more silicon and just as likely to fail. MCU's are reliable and in my experience they very rarely die, but they can be killed. Your efforts will be much more effective looking at other aspects of the design such as emc immunity. Attention to good layout, PCB design and protection circuits will be much better.

[rokspy]

To give a bit more context, it would be for a student satellite project targeted for LEO. The amount of space in the satellite and funds in general are limited in our case. So with the current question I'm mostly trying to deal with MCU failures due to single-event-burnouts and total ionizing dose accumulating over time, which essentially is inevitable (though the time it takes for COTS MCUs  is quite controversial). Limited funds, commercial-off-the-shelf, acquiring rad-hard MCUs are too expensive in our case. Other rad-hard devices are more likely and in this case the fault detection circuit would be redundant itself and probably of the highest priority. Currently the agenda is to work with individual MCUs on individual systems, where the MCUs are periodically power cycled by an external watchdog if the MCU does not clear it, which does deal with latch-ups. However, it does not deal with cases where MCU takes the long nap, and a whole system is lost, so we are also exploring and looking for new ways to increase reliability and come up with new designs to test extensively. I agree that adding multiplexers and additional set of components is adding additional points of failure and software based failures are more likely than hardware so they would end up in both MCUs.

I will post this reply once to not populate this page.
Thanks for all of the answers and the insight, it is much appreciated.

[woofy]

To give a bit more context, it would be for a student satellite project targeted for LEO. ...

That changes everything, now I understand the wish for redundant MCU's.
Let me say I have Zero experience of satellites.

I agree multiple MCU's and a mux. could work. The trick will be making the mux and its failure detection reliable.
I would start by getting each MCU to check itself. Dedicate a pin to an "I'm OK" signal. The MCU will check its sub-systems an set the pin high if all is well, Ie. interrupts are firing correctly, timers, communications etc are all running. It will also check its critical data with parity checks, CRC's and against multiple copies, and set the pin low if all is well. The idea is to generate a pulse train only if all is ok. If it gets stuck high or low then its failed.

For the mux. keep it simple. Use older technology as that will use wider line widths on the chip, that will give better immunity against SEU's. Inputs don't need muxing, just individual buffers to each MCU. Outputs will need muxing. I would feed each "I'm OK" signal into a simple capacitor/diode/resistor/schmitt buffer circuit for a logic high when all is well. These can control your muxes.

Only use combinatorial circuits in the mux system, no flip flops.

Well that's where I would start.

[rounin]

It might be worth looking at ASIL B / ASIL D rated microcontrollers, eg NXP S32K, S32S.

For a few dollars more you get ECC ram & flash in ASIL B, and a full lockstep core in ASIL D. The lockstep core will let you detect some more SEU events than ECC alone. They support a failure model similar to your requirement, eg detect fault internally & go quiet when faulted and let another module take over. They usually have some extra internal support for self testing which can be helpful, usually an extra fault management peripheral that can drive an external fault line, sometimes BIST reference software libraries.

A while ago I worked on making an MPC5777M based autopilot module, with the goal of also selling to applications like cubesats. Project got canceled unfortunately. But some of the ASIL B parts are not really any more expensive than anything else. The ASIL D parts might be 20$ each.

[Silenos]

though the time it takes for COTS MCUs  is quite controversial

What do you mean by "COTS MCUs"? I do consider hard-rad/aerospace mcus perfectly COTS.
If you mean common mcus like eg STM32, well, I disadvice. I didn't do aerospace, just dealt with... more artificial sources of ionizing radiation and these kind of mcus were first to break in just a flash of gamma flux.
And whatever hardware you chose, you should test the specimen in expected environment in some radiation lab and see if it conforms any regulatory stuff. Consider this if you are limited on funds.

[fchk]

You might have a look at this one:
https://www.ti.com/product/TMS570LC4357-EP

This controller is qualified for functional safety:
- all memories are ECC secured
- there are two CPU cores working in lockstep
- build in self test (BIST)
"Supports Defense, Aerospace, and Medical Applications:

    Controlled Baseline
    One Assembly/Test Site
    One Fabrication Site
    Available in Extended (–55°C to 125°C) Temperature Range
    Extended Product Life Cycle
    Extended Product-Change Notification
    Product Traceability
"

This is the next best thing to rad-hard chips.

[Berni]

Yep what you actually want is a lockstep executing MCU.

A lot of the big microcontroller manufacturers make these for safety critical applications, like TI, ST, NXP..etc

Not only does this solve the complications of joining together two MCUs, but these also have more advanced fault detection. For example if a cosmic ray flips a bit in RAM then the CPU will know it is corrupt upon reading it and drop into a error handler before it even has a chance to perform a calculation with the corrupt data.

The one thing this wont save you from is a hardware failure since it is a single MCU, so if a voltage spike blows up a IO pin or a solder joint on a pin lets go, then there is no way to work around that. The ultimate way to work around that is indeed having 3 MCUs and have voting for every output. The voting logic could be implemented inside some programmable logic to make it smaller, even a ROM chip could be abused to implement it by feeding voting outputs into the address pins and using the data pins to get the final output out of it.

But keep in mind that safety critical design involves both hardware and software. No amount of hardware reliability will help you if your software does something stupid and destroys the mission.

[Psi]

Consider if you really need a 3x MCU system.
It may actually be better and more reliable to put the money into a more robust powersupply and IO protection and then conformal coat the PCB

[Kleinstein]

Consider if you really need a 3x MCU system.
It may actually be better and more reliable to put the money into a more robust powersupply and IO protection and then conformal coat the PCB

On a satelite conformal coating is likely less critical - not much humidity there. The point is more about radiation, temperature variations and vibrations (during start).
With the limited space and power, a single high reliabilty µC with a good watch dog may be an alternative to multiply µCs. A more complex system also adds more points of failure.

[Psi]

As soon as you want to add redundant MCUs you really need to duplicate everything rather than adding redundancy to individual subsystems. Especially if the device is something that can never be on-site serviced.
 
You wouldn't have subsystem A that has internal redundant MCUs and does one part of the system, and subsystem B that also has internal redundant MCUs and performs some other task in the system.
Instead, you would have two totally separate systems that each have their own A and B.
Those A and B subsystems wouldn't have internal redundancy, the overall system has it because there are two entire systems.

One of the systems is always in a standby mode and just listens for the correct RX command to trigger it to take control and become the active one. Also it might take control automatically after N hours of not getting an RX remote watchdog command that indicates everything is ok.
Also, having two totally separate systems tends to make it easier for one to diagnose faults in the other, or reflash its chips and run tests before releasing control back to it.  You can offload the checking and comparisons between the two systems to a ground based task and not have the satellite trying to do that itself. Self checking systems get compilated and you want things as simple as possible to reduce the risk of hidden bugs.

If you want to add even more redundancy you add the ability for the systems to be able to swap each of their subsystems for that same subsystems in the other system. (eg it could be a relay that physically rewires them like a crossover switch) 
Then you have some way to permanently flag a subsystem as faulty. That way if subsystem A on system 1 is faulty and both systems have been told about this they can both continue to work fine and each can take control as per normal. They just have to always ensure they switchover to using the good subsystem when they take control.

[bookaboo]

While not directly what you asked for, this may be worth a read. A good WDT could help provide some redundancy regardless of the architecture you go for: http://www.ganssle.com/watchdogs.htm

[Infraviolet]

I'm aware its pretty common for cubesats to avoid rad-hard chips due to costs, but would it be too mass-hungry for you to somehow enclose the most sensitive chips in your system in lead sheeting? Perhaps a small PCB with your MCU on it, lead sheeting folded around it with just some twisty paths (so no straight line for high energy particles to enter along) out for wires to run from it to the less sensitive PCB areas?

[WatchfulEye]

I'm aware its pretty common for cubesats to avoid rad-hard chips due to costs, but would it be too mass-hungry for you to somehow enclose the most sensitive chips in your system in lead sheeting? Perhaps a small PCB with your MCU on it, lead sheeting folded around it with just some twisty paths (so no straight line for high energy particles to enter along) out for wires to run from it to the less sensitive PCB areas?

This may not work depending on the radiation environment. High energy (e.g. cosmic) rays may be scattered by the lead and result in a shower of lower energy particles with an overall higher cross section, potentially increasing the risk of SEU.

This phenomenon used to be used for analogue radiography. A lead foil in front of the film could actually increase the sensitivity of the film, by photoelectric/Compton absorption of photon radiation and conversion into electrons which have a much greater cross section.