Saving state on power off

[Axk]

Given a circuit with an FPGA and power disappearing how do you implement triggering of saving the current state?

I'm thinking about adding a capacitor between the power supply and FPGA's power pins while having a GPIO pin monitor the power supply bypassing the capacitor.
When the GPIO pin goes low then it would save the state. Assuming the capacitor will give it enough time to save the state before the FPGA loses power.

Are there any standard approach(es) to dealing with "save on power off"?

[NiHaoMike]

That can certainly work and is done in quite a few popular 3D printers. Another solution is to constantly save the state to FRAM or SRAM with battery or supercap backup.

[Ian.M]

It doesn't matter whether its a MCU, FPGA or other digital processor - the key ingredients are the same:

* Some sort of non-volatile memory.

* A power supervisor chip (or internal module) to either hold the processor in reset  when its supply drops, or to de-assert the NVM Chip Enable so the processor doesn't corrupt the NVM with spurious writes as it drops below its min. operating voltage

* Optionally, for write endurance limited NVMs like FLASH or EEPROM, or just to make maintaining data integrity easier,  advance warning of power failure e.g loss of AC input, low battery detection, etc. to give enough time to save the state, or complete any pending writes, then the firmware locks out further writes until the incoming power is OK again (with or without compete powerdown in-between).  If you don't have a power fail interrupt, you'll have to store multiple copies of the data with checksums as any copy of the data set may be corrupted by power loss while updating it.

ultra-low-power MCUs can take an end-run around the problem: Maintain power to the CPU with a backup battery or supercap, and when incoming power-loss is detected, enter a low power sleep state that preserves internal RAM contents.

Caution: if using FLASH or EEPROM NVM in mains powered or removable battery applications, its advisable to rate-limit NVM updates, otherwise a very noisy mains supply or intermittent battery contact due to vibration can cause rapidly repeated updates and chew through the NVM'ss expected write endurance lifetime in weeks or months rather than the expected years or decades.  The trade-off is the possible loss of any state changes in the last X minutes before power loss, where X is the interval you chose to rate limit updates.  This does not apply to effectively unlimited write endurance NVM, e.g. FRAM and battery backed static RAM

[jbb]

Using an FPGA may make things more difficult, because you may be using internal block RAMs etc. Any technique which relies on pushing data into non-volatile (or battery backed) memory will need some hardware support to a) write the important data out periodically and b) get it back in again after a blackout.

This will be easier if your FPGA has a microcontroller inside and all the relevant data is mapped to the micro’s memory space.

If you need to maintain all of the state, you might need to simply deploy a backup battery large enough to keep the whole FPGA alive. (Note: clock gating in the FPGA could be very helpful here in terms of reducing power consumption / battery size.)

[jbb]

Oh, yes. I forgot...
In terms of system design, you can also use a specially designed power supply that can keep the DC rails up for a little while after the main supply drops out (i.e. a large capacitor somewhere). Then use a monitor circuit (with de glitching) on the power input  to send a signal to the FPGA saying ‘power’s down, shut off loads and save critical states NOW.’

I have seen a few systems which go so far as having a little LiPo or LiFePO4 cell to provide ‘last gasp’ features including state saving and even sending a power loss message via cellular modem.

[DaJMasta]

Similar technique, different implementation, you can do your power off monitoring on the mains side of your power supply, so that as long as you have enough cycles to save state on an interrupt, you get to use the capacitance in your power supply output without adding extra or trying to sense it very early on in the discharge cycle.


It sort of comes down to how long you need to save state, though.  If there's no capacitance or the chip may be in sleep or very slow operation, you probably want something that is just regularly saved or an additional supply (capacitor) to give it the time to save.  If you keep your save requirements for the state to just a few cycles and the power sensor is quick and on a high priority interrupt, you may be able to get away with just the power sensor, no extra storage or hardware.  With traditional power supply designs, you likely have at least 100uS, and probably have several ms before the power drops low enough to prevent operation, so you just have to be able to sense the fault and react in that window.

[Axk]

I just need to save around 10 bytes of state and planning to save it into the FPGA's (LCXMO2-1200HC) internal flash.
So I hope to get away with a couple of hundred of uF of capacitance to give it enough time (a couple of milliseconds at around 20mA max) to write to the flash.

[mariush]

How often does that state change ?
Maybe have a flag which is set if any of the bits in those 10 bytes changes, and start a timer at that point... when 5-10 seconds go by, dump the bytes to flash and reset the flag

[Axk]

up to 1.5Meg times a second

[jbb]

Eek, that’s a lot.  Now we can guess why you’re using an FPGA

You should check out the erase / write lifetime of the internal flash. As someone said earlier, you could get a lot of write cycles if the incoming AC power goes intermittent.

[Axk]

Thinking about the implementation details now.
How do I detect the loss of power as close to the FPGA as possible, without going all the way up to the input of the circuit (which is 54V)?

Putting a Schottky diode right before the backup capacitor and then sensing before the diode comes to mind (so that when the input current is lost the FPGA continues to work from the capacitor but sensing input doesn't see the voltage on the capacitor).

Will this approach work?

[Ian.M]

That depends on how much margin you've got between the normal FPGA operating voltage and the minimum voltage at which it can operate normally + write to FLASH.  If you tap off earlier in the PSU, before the regulator, (possibly via a potential divider to a micropower comparator), you get a lot more warning  time for a lot less capacitance as the input side of the regulator can drop far further below its normal voltage before the FPGA browns out. You'll probably still want a diode so the comparator doesn't sense the slowly dropping bulk cap voltage after the supply input goes away.

[Axk]

Thinking about it more, I already have voltage measurement before the regulator and the voltage divider (VSEN) that that is measured is disconnected when the circuit is switched off.
So I suppose should be able to use this voltage measurement provided I add enough capacitance (not added in the schematic yet) to ground after the regulator to give it time to write to flash.

[NorthGuy]

FPGA is likely to be high power which makes it difficult to power with a capacitor. I would use a small low power MCU instead of flash fed by a separate linear regulator. You would pass the state to the MCU continuously. MCU would detect voltage sag and then would store the state in the internal flash. You don't need a big capacitor to maintain an MCU and it would cost you the same as flash IC. You can also use the same MCU to power down your FPGA gracefully.

[Axk]

To write one page of flash it takes 0.1ms.
I can erase the flash on power on, so I only need about 0.1ms to save the state.
100uF (a couple of 1206 X5R 100uF ceramics considering derating) should be enough considering the 75mA total current draw of the circuit (3.3V).

[Ian.M]

Its much more effective to add capacitance before the regulator.   Assuming your regulator will work down to 25V (I'm guessing its a switching regulator as dissipating 3.8W in a linear regulator would need heatsinking), and starting from 10% low input voltage, so 48.6V,  there's 868 Joules per Farad between those two voltages (from E=C*V²/2)   You need 75mA, @3.3V, which if the regulator is 80% efficient is 0.31W input power.   To hold the 3.3V rail up for 1 second only needs 356uF, or 0.356uF for 1ms. 

If you are using a linear regulator the maths is easier.   Assuming it drops out at 8.6V, the input can fall 40V.  From Q=CV, allowing 5mA quiescent current for the regulator and 75mA load current,  it would tahe 2000uF to hold it up for 1 second, or 2uF for 1ms

The tradeoff is the higher voltage caps required are larger and usually more expensive than low voltage ones.  OTOH the 3.3V rail is maintained in spec. for the holdup time.

 

[Axk]

@Ian.M, thanks!
I didn't think in terms of the regulator giving a lot of margin to how far a capacitor on the high side can discharge before the low side starts sinking.
I though I would need an electrolytic on the high side, and I want the board to be low profile.
So as you point out considering the margin I can use a couple of 100V chip ceramics instead.

I'm planning to use a linear reg with an external TO-220 NPN with the whole board attached to a somewhat longer 2mm aluminum sheet and the transistor screwed to the sheet where it sticks out to keep it level with the board.
I could use a switching converter but I want to keep things simple.

[Axk]

How do I determine if my PMOS switch can withstand the current surge needed to charge the 10-20uF of capacitance?
The maximum instantaneous current is specified at 3A?
Can it withstand more to charge the caps or should I use a current limiting circuit?
I 20 ohm resistor in series with the capacitors?

[Ian.M]

Choose a gate resistor to limit the PMOS turnon speed so the capacitor charging current doesn't exceed Id_peak.  Some experimentation may be required with a current probe and a DSO,

[RGD2]

I'm just hitting this one myself: Needed the reminder about possible corrupt data writes.

Signed up here just to chip in.

My solution was a small FRAM chip, and I was going to also use the same sort of loss-of-unreg-input detection. Just a resistive divider to input, looking for the 5V rail to start dipping.
So I was just going to use a trimpot, so as to bring the 5V down to just where the pin would register it as definitely a low if it dips.

Will still give it a try, but the other circuits look more reliable (and reproducable). I'll report back in if it does.

I decided to go with an FRAM chip - these write extremely fast, could fill a whole 2Mbit chip at 25MHz in only 84 ms or so, let alone a paltry few bytes.

I was looking to try 'wear levelling' the thing, but it turns out you actually can't: FRAM's wear on both read and write cycles. Anything you try to do to wear-level therefore won't work: Every boot you would need to read to the 'last used' place somehow, and that just puts more wear over the front of the chip every time anyway.
The reason I was hoping I could, was with 10^12 cycles (assuming basically only writes wear), I could just update on change, and keep appending-writing, with a 'only increments' counter tacked on, that would make the chip look like a circle buffer: recovery would be looking for the sudden downstep in that counter on boot to find the 'last' write, so as to pick up where it left off, and this would wear level nicely, and allow a small packet to gradually use up write cycles across the chip, thus allowing keeping a short section of critical data always up to date.

But alas, can't do it: Reading also wears the chip, and you're reading as often as you write, or more so if you're trying to wear level with a scheme like that. Please anyone who can see how I'm wrong: show me that I'm wrong, I'm not 100% sure. Intuitively though it feels like wear levelling only works if reading is 'free' and only writing hurts. The problem is, on boot, how do you know the next spot you should wear, without causing wear? On eeprom that's easy - you can read without wearing. But if reading wears...

So my solution is:

Just write vector of critical data in a burst, packed to a multiple of 8 bits, when it is suspected that a power outage is about to occur, then disable the chip.

Remember the 'write enable latch set' command first.

This could go out at 20 - 40 MHz depending on which FRAM chip you get.

Even with how 'high power' an FPGA is, this won't take long, since you don't have to wait for the slow writes of EEPROM with FRAM.

On boot, just read from address zero your burst of bits.

No wear levelling, and the chip will only last so many write cycles, so tune to where it doesn't write unexpectedly.

Maybe count them and keep the number in your critical bits, and occasionally read/check how it's going?  Just want to notice if it has started to hammer on the FRAM - an FPGA could easily wear it out unexpectedly in a few months. So long as you avoid that, life should be so very long that any wear levelling scheme is just not worth the effort.

There was a guide to FRAMs for engineeers here: https://www.eetimes.com/an-engineers-guide-to-fram/. (Which of course I only though to look for after asking myself, "Hey, do reads maybe also wear FRAMs?"... yes!    ).

[Ian.M]

@RGD2,

Welcome to the EEVblog forums.  May I be the first to compliment you on the quality of your first post here.

Yes, I think your wear levelling argument is valid - you can't do it for FRAM without an auxiliary NVM using different non-destructive read technology to hold at least some of the wear data. 

However even a humble 24C00 16 byte EEPROM or a few words of self-writable FLASH memory in your MCU^* could let you do that - divide your FRAM into blocks, track each block's usage within itself, and once it reaches your chosen wear threshold, copy that block to the free block with the lowest wear, update root block pointers, and flip the corresponding bit in the  auxiliary NVM from 1 to 0 to disable the worn out block.  It goes a little differently if the root block wears out, as  it has to be possible to find the root block from only the auxiliary NVM contents.  You could store a pointer to the root block in the auxiliary NVM (which should change rarely enough that endurance for the pointer isn't an issue), or you could define the root block as the first block that hasn't worn out, and shuffle blocks to keep that so whenever it wears out. 

As each bit in the auxiliary NVM's block map only has to flip from 1 to 0 once, wear on it shouldn't be an issue.  Some higher level checksums and error correction codes, and even tell-me-three-times redundancy may be required for the auxiliary NVM to handle cases where corruption is possible e.g. due to power loss during an auxiliary NVM write.

* FPGAs aren't very efficient at implementing complex but slow logic directly as it typically requires as much resources as full-speed logic of comparable complexity.  Once the complexity goes over a certain threshold, an 8 bit soft-core MCU to manage supervisory tasks may well be the best option.

[AndyC_772]

Intuitively though it feels like wear levelling only works if reading is 'free' and only writing hurts. The problem is, on boot, how do you know the next spot you should wear, without causing wear? On eeprom that's easy - you can read without wearing. But if reading wears...

It's all down to how often various things happen.

Suppose you have to write a state at regular intervals. You can spread out the wear by treating the NV device as a circular buffer, divided up into blocks with a sequence number attached to each one. The last state is always the one just before the discontinuity in sequence number.

You might write the state many times per second, and this wear levelling algorithm is effective at distributing the write cycles across the entire device.

When the system boots up, it has to read the device - possibly the entire device - but that's OK. It's still only one access to each location per power cycle.

Statistically, how often do power cycles occur compared to the number of times the circular write buffer wraps round? This gives an idea of the relative wear caused by the start-of-day search for the last written block, compared to the writing of the block themselves.

[RGD2]

When the system boots up, it has to read the device - possibly the entire device - but that's OK. It's still only one access to each location per power cycle.

Good point AndyC_772. Most of the wear will still be levelled, bootup access contribution should be negligable.

In my app reboot is rare enough for it not to matter: the problem I need to cover is maintaining a performance guarentee of the system despite a short-duration power failure: the equipment being controlled won't tolerate a 'homing' sequence on boot: it will break something. There is only one open-loop stepper axis, and annoying, the 'safe' position is farthest from the only homeable endstop, so current position needs to be maintained despite power cycles. This whole requirement was one 'tacked on' to the end of everything else: The main job of the system is actually DI type fuel injection timing.

[RGD2]

@Ian.M

Thanks! You make a good point, and the scheme you detail certainly would solve that problem.

* FPGAs aren't very efficient at implementing complex but slow logic directly as it typically requires as much resources as full-speed logic of comparable complexity.  Once the complexity goes over a certain threshold, an 8 bit soft-core MCU to manage supervisory tasks may well be the best option.

By that point, I have previously sprung the on-FPGA resources for a swapforth/j1a core (or my personal j4a core): Almost full featured ANS forth compatible 16 bit core, that implements the Forth stack machine in hardware, not by 'stack is in ram, pointer in register', but 'both stacks exist in place of registers'.

Runs easily up to about 60MHz, and that's one forth word call per clock, most of the time. Every indirect costs you a clock, but in general it executes standard 16 bit forth hilariously well. The hard-limited stack depths hurt less than you'd think, and it's a joy to use. (See for Jonathon's demo of swapforth j1a on a 1k gate ice40 chip).

More recently, I'll just co-install an Arm SBC of some kind running some variety of debian. Then usually just communicate either by serial or USB (or both). Plain python is fine. Can then have stuff like tensorflow / openCL if really heavy lifting is needed: or just talk over the internet to a supercomputer.

Certainly USB 2.0 HiSpeed does up to 30M bytes/sec reliably: So long as you have at least 8MiB of SDRAM on the FPGA side, at any rate. Through an SBC that does drop to only around 18MB/s or so, but I've found 16MB/s to be enough for my needs. This gets punted over the network, just using netcat on the SBC, and is 'caught' the same way elsewhere on a server with the space to put it to disk.

I haven't looked into it deeply, but I do have a $20 FPGA board on my desk: A Colorlight 5A-75B, having a LFE5U-25F (25k gate) FPGA and 2x gigabit ethernet interfaces. There's already a open source RISCV soft core SoC project for this that works (including gigabit ethernet!)  but I have yet to check what *sustainble* dataflow it can reliably manage, and whether the couple MB's sram it has is enough.

Either way, the data capture pipe I tend keep completely separate from the 'control' FPGA: My business is basically getting evidence of running new equipment, and if it fails, we need to be able to learn from it. When it finally all works, the reports look totally boring, but that's what you want to see. Boring reliability. A whole bunch of pulses that look identical, apart from the natural noise. Better not to mess with what works, and has proven reliable so far. So the control FPGA is generally on a completely separate board, let alone chip. It's just easier that way.

At most there's an additional link between them, which lets the control FPGA 'snoop' on selected live data.

Most of the stream is somewhat of a black box: 8 channels of 12bit 0 to 3.3V data at 1MS/s, mostly watching hydraulic pressure sensors. Acts somewhat as a cross between an oscilloscope and and strip-chart recorder, but one which is more like a highspeed camera that is just running all day long: Really good for catching things breaking, really makes it easy to see what went wrong, since any 'triggering' can be done entirely post-process.

You end up with some big files, but hard drives are really cheap these days. I use the 'snd' program to deal with them: works really well with arbitrary sample rate, bit depth, channel count and file size: Never crashes, just takes a while to read over the file sometimes, especially if you zoom out too far. But it comes back with a really neat overview, where you can find any glitch, no matter how small, and zoom in on it - it feels a bit like using google earth, because you can go from 'all day' to '100 microseconds' or so.

But all the above is the recording / science side of the job: For controls, just putting the stuff that needs precision timing in the FPGA, with some dumb state machines, and setting up a low-latency link to where more complex code can run works well enough. The job is basically DI engine ECU, where I do DI injection and ignition timing, plus some other things as needed. The FPGA is just doing the job that the mechanical fuel system does - whilst being able to be a bit smarter about dealing with known injection latency, for example. This also does some correlation of deskewed pressure data with engine position, and having all that corrected for latency so it's collated in terms of observations at a give engine position make analysis much easier:  separate data link and save flow for that.

What really impresses me is how good the open source only FPGA tools are getting: At least for the chips they have good support for.

The main thing, apart from being able keep all the toolchain as well as source code within the same box, is that it eliminates all the silly 'IT' failure modes. I've found such systems far more stable to bring out of mothballs after a couple of years. With everything else, there's another two battles: reinstalling the software, getting it relicensed (possibly repurchasing a time-limited license again) and then needing to fix any breakage that occurs because of the new versions.

With the open source box, you can just turn it off, walk away, at let it sit for years. When you dust it off and turn it on, sure the OS is out of date, but it all just works, exactly as it was. Updating it is easy and quick, and usually painless.

The OSS FPGA tools on a fast new PC are breathtakingly fast, especially compared to the older vendor tools. See for instance icestudio.io : It's like the arduino 'just an app' setup experiance, more or less, except going from changed verilog code to configured FPGA with blinking LED is literally seconds. Faster even than the same wait for a changed code on an Arduino platform design, or so it seems to me.

But even on an embedded SBC, they're still about as fast as the vendor tools are on said fast PC: maybe 15 to 20 minutes, but doable, for that 8k gate embedded forth soft SoC FPGA recompile.

With quicker turn around for development than anything but a live forth system, along with easy immunity to interprocess interlocks - which can be nasty intermittent problems to solve; I think even quite low end control cases should go to FPGA from microcontrollers, especially where servicability matters at all.

It's quite feasible to now put the whole dev tools into an sd card, and have that card served in usable state via a web page running from a $4 esp microcontroller, where the user BYO's some wifi-connectable smart device which then provides all the RAM and compute needed to not only run a GUI, but also the whole toolchain including recompilation! I haven't done it yet, but it would be a good student project. esp-link , maybe some use of emscripten etc.

That would save the cost of the SBC. Just stick a QR code in there to let whoever opens the door have their smart device connect up and point it at the entry web page. Can work entirely without internet access. Could use something like the cloud9 web-hosted IDE to make that quite comfortable too (I have hosted that from an embedded system, and it's fairly nice), or just use jupyterlab or so. IDK: not a web dev.

At this time the FPGA tools are available already in architecture portable executables which run using WebASM (not in a browser).
You can be up and running in a minute or two, on any platform. (YoWASP project). I do this using microsoft's WSL2 and it has been good. A little lower level / more flexible than icestudio.io, but very comfortable if you're comfortable in linux.

As a side note: I've found dumb relay ladder logic, and dumb analog alarm relays (And a pulse-presence alarm relay) is all worthwhile for anything safety critical (especailly safety of people/environment, but also safety of plant). Should all be fail-safe and 'programmable' by turning a trimpot, with readily available E-stop buttons. No matter how 'smart' the programmable stuff works, day to day - safety is just a whole 'nother ball game! A couple threshold/alarm units, maybe some interlock switch-physical access locks go a long way to keeping absent-minded people safe.

I did at first feel like I was going out on a bit of a limb, just using all open source toolchain stuff, but I've had excellent results doing so, and it's proven itself time and time again on some older apparatus that gets only intermittent use. It's easy to recommision, tailor to the new job, and be up and running in less time than it takes to just install/update the heavyweight windows software otherwise needed.

projects Symbiflow, yosys, icestorm, trellis, apycula, YoWASP,  etc are all very much like "gcc for FPGAs", and only improving with time. Probaby you'll get better design performance with vendor tools still, especially if you're chasing maximum clock rates or minimum space usage, but for what is typically otherwise 'arduino' territory, I think there's a very strong case for skipping over microcontrollers (including RTOS) entirely these days, in favour of a hybrid FPGA digital logic / Linux OS SBC approach.

I expect eventually there'll be even smaller cheaper 'lower end' FPGA platforms, and there kind of is - c.f. Fomu FPGA.
Support for some of the nicer older FPGAs: Xilinx spartans and machXO etc, is slowly appearing as well.

Were I intel, I'd be trying to get on top of it.

Anyway - I've got stuff to do. Sorry for the Rant, it's probably just I do have to deal with some colleages stuck in the 90's sometimes.

[IDEngineer]

How does the FRAM "wear out"? If it fails to a known state, say all bits stick to 1's, you could accommodate a wear algorithm. Say you start at the top of the address space. The first valid byte going down in address is the first one not equal to 0xFF. As the current byte ages, eventually you write it to 0xFF and start using the next lowest byte. Further reads of the aged-out bytes will always read as 0xFF (in this example), either because that's the last value you wrote or because the cells are dying. Either way you will always read 1's and the algorithm works until you wear out the last byte, which should take a very long time.