TDD embedded.

[paulca]

I have a fairly large firmware project on the bench at the moment.  The hardware doesn't exist yet.  A lot of the code relates to the process above the peripherals. 

So, instead of using CubeIDE and writing for a dev board, I created a basic Unix C project.  Added unit tests and started down that route, more like I would in work.

I was fairly surprised to write a custom fifo module.  Then write a full suite of unit tests for it and ... it worked first time with one minor bug.  My "design" expected if you call "take" on an empty fifo you get the last entry and the pointers dont move.  Of course the pointer has already moved and points at head, so instead of working out what that will be, i just replaced it with "Undefined".  URUN is set though.

I then moved to the "Bus Interface".  The part of code which handles the external bus communications. 

Here it introduces the concept of test surrrogates.  The bus_interface requires a GPIO API.  So, again, rather than jump to an actual MCU, I wrote an interface.  Classic C, no fancy abstractions.  gpio.c doesn't exist yet, but gpio_test_stub.c  does.

I implemented functions as "What I need from here".  "read_address", "read_select", "isEnabled()" read/write_data and so on.  "release_data" "drive_data" of course.

The gpio_test_stub has extra static methods the unit test can "extern" in, which allow it to "prime it" with test data.  When the actual bus_interface calls "read_address" a test address is returned.

I am still curious as to how far I can take this before it meets the micro and ... how much good it did me in terms of bugs and struggles.

The hardware arrived as a PCB blank yesterday, so it will be a while and for it's first few days I don't expect it will be running real firmware, rather debug and diagnostics firmware to prove out the hardware.

Anyone else use TDD (Test driven development).  Basically you isolate out code away from the hardware, so you can write tests even before you write the code, or in parallel with 2 people (1 and an AI).

The "interface" here is a very low level and very old one, I doubt any of it will survive the compile optimisation in the wild.  Using swapped "implement.c" files for headers is a standard C feature and there should be very little other "glue" around when its deployed for real.

For context.  The firmware - this version - exposes an STM32 CDC peripheral on the 68000 parallel "8x8" IO Bus.  8 address, 8 data, ctl.  Because the STM32 - in my hands anyway - is not going to be comfortably fast enough for various portions of the bus cycle, I used up an FPGA I had in stock as "Bus Controller".  It treats the STM32 like a peripheral IC.  Give it an ENABLE line, an IO_ACK line and choreographs the databus and data_ack dance at nano second speeds.  It itself is not written yet.  Well, it's "drafted" in Verilog and the simulation "looks right".  Buts that only 10% the hurdle.

The entire chain for "Hello world" here is custom from 68k ASM, FPGA Verilog bus control, STM32 C code peripheral facade... the more I  can test in advance the better!  All of it has to work right for it to say "Hello".

The firmware model has two sides.  The hot looped "in thread" bus facing portion and the interrupt and flag signalled back end actual peripherals.  The touch in careful places only.  For this first version the fifos are the interface between them.  The plan is to just shut interrupts off while a bus transaction is in flight.  Signal by flags things like "TX_START" for an empty fifo transitioning to not empty for example.

[NorthGuy]

I was fairly surprised to write a custom fifo module.  Then write a full suite of unit tests for it and ... it worked first time with one minor bug.

I wonder how would you approach this if the FIFO was DMA-filled?

[PlainName]

Anyone else use TDD (Test driven development).

Yes, but perhaps at the wrong level. For example, I was building an ESP32 project using the ESP-IDF, so there were quite a few calls to that. I wound up prototyping out the ESP-IDF code (that is, replacing <esp>.c with <fake esp>.c) which wasn't too bad once the basics had been done.

There were also modules that didn't hit the ESP code, of course, and the test modules for those were simpler to implement. But at some point you're going to hit libraries and then you gotta fake them.

Having code pass the TDD tests doesn't mean it's good code, just that it passes those tests. The tests will initially be for things you need and think of, but if you haven't defined an edge case (and a test for it) then your code could fail there. But once you fixed that, and created a test for it, it shouldn't happen again if you make a change.

Is it worth it? It can significantly lengthen the project (and you need to keep maintaining the tests), but there is the promise that the testing will catch screwups further down the line. Apart from that, it does make you think about your code more, which is probably a good enough reason on its own to at least set out down that route even if you don't keep with it.

[hans]

I use TDD quite a lot in embedded.

What I like is thinking about how a software part should operate before writing it.
Sometimes its necessary to build something from the RTL up. But I often find myself at a point: okay the BSP works, I've some messy simple test code, now what?

I like TDD because I can run all business logic outside of the hardware. You can simulate some software state to be exist at a flipover point, overflow/underflow conditions, etc. And it runs basically all the time.. so when fixing a bug, write some reproduction code for it, then keep it around to prevent regressions.
I program in C++ so I use GoogleTest for this.  I suppose you could also make it work in C by swapping out gpio.c with gpio_test_stub.c as a linker trick..

The main downside of TDD in embedded is that it requires some extra abstractions.. I think of everything now as a sequence of events. These events can be bumping a statemachine (verbatim) or series of function calls that are processed in order. Put these in a test case, job done.
But when it comes to interrupt <=> main() communication, it can potentially require a little bit more extra code to verify. Or when main() is executing Class.MethodA() and an IRQ is executing Class.MethodB() that have some shared state.
It can work but it also cannot. How to test for race conditions? That's why I take the overhead of an event queue (which should be reentrant, race condition free, etc.) and then process those in-order as a general design pattern.

Then the trouble becomes testing code that has some variation with timing. Execution latency cannot be tested in TDD (easily) ... Code thats too slow to meet deadlines needs hard metal.

And finally.. writing test code can easily result in a lot of "simple" code that executes a lot of methods over and over. I often see 3-4x LOC in tests than in the real implementation. It also becomes real easy to write useless tests, for example, for setters and getters.. (no logic, no branching, what can go wrong except for a typo?)
I've always seen (by far) the largest benefit from testing edge cases. You can sit down and actually solve and "document" that problem in a reliable test that can be reran over and over again..

The nice thing of having these tests is that in case of refactoring, you can change implementation and verify everything runs fine.
But I often find the reverse to be true: feature creep for some software module, and to account for these new (separation of) concerns, its 'API' needs modifications.. and now you'll be modifying the business logic code alongside 3-4x LOC worth of tests too.
So tests can help but its not like they are exactly free to use.

[paulca]

Interesting point on concurrency/threading testing.  Usually when writing MCU code that part is 100 times easier.  You typically only have one core.

Though I can think of test strategies for big iron and multicore for threading/multiprocessing, the tools required would be overkill for an MCU test.

The general rule is to make your code "Thread safe" and not worry.  In MCU world you are  basically, "almost" thread safe as it is, but thats only because you have a single core.  So the question is "Does the unit access any shared state?  Does the unit in test have any state which is mutated when called by multiple threads?  Does the unit pass an exclusion/monitor/synch zones?"

The Fifo for example. 
Does the unit access any shared state?  - no.  The fifo's state is internally managed.
*Does it have state mutated by multiple threads - yes.  By design - but by a single CPU core - so interleaved.
*Does it pass and monitor zones?  - yes.  This is how the above is supposed to be handled.

The fifo is shared between two "threads".  Thread one is the permenant "main" thread.  The second is the temporary ISR thread - if you permit me to call it that.  Synchronisation is required.  Note.  I need to verify the STM32 only supports, or can be made to only support, single ISR without re-entrancy and without nested interrupts.  I need it to just be 2 threads.

The plan is.... when the main thread is triggered out of "wait and maintain" by a bus transaction "Enable line" goes hot.  It first has to wait on any running ISR, disable interrupts, mutate the FIFO, enable interrupts. 

The details of that synchronisation (both sides of it) is still in the making.  "Wait on running ISRs" - doesn't have a solution yet.  Can I detect them by register flags?  Can I spin on an atomic "isrBusy" safely?  Do I disable ALL interrupts or just mask out the one which touches "this" FIFO?  Still open questions.

On "If the FIFO was filled by DMA".  How does the DMA channel 'feed' the fifo?  If you mean it writes directly to it's underlying RAM... then it's not a FIFO in my opinion and would be a completely different mechanism entirely.  The FIFO is a contract, a write to it's backing memory would be a violation of that contract.

However.  The talk of DMA comes later.  Not for CDC or UART, but for SPI which I hope to use as a underlyer to present a "block device" later.  512byte blocks to match most backend media blocksizes.  There using the STM32 SPI DMA is very obviously the right way, but note, it's not a FIFO, it's a "Buffer request" mechanism.   Command Out, Buffer or error back or "Command+Buffer out, error or ok back".  I test those by writing a facade around "SPI_Transmit_DMA", "SPI_Recieve_DMA" lower levels and test they get called correctly and swap in test data when in the harness.  The "CPU Side" of that interface is probably not going to really be held back if the STM32 waits for the full 512 byte SPI transaction to complete before it even "enables" the Rx.  (Not least as in period it would be waiting on a 512 byte transfer + seek time on a Floppy in 1980s.

The key thing to point out.  If you don't own it, didn't write it, then you aren't testing it either.  Get it OUT of the test.  The more you can remove the more its testing just your unit.  I am not for instance testing the STM32 Interrupts or DMA.

EDIT:  To dispel the slight miss conception about what exactly it DOES test, a unit test that is.  It tests the unit does what the developer intended it to at the time the unit and or test was written.  Nothing more.  However, that is often a sizable about of code that can be tested that way, which would otherwise have to be tested in "Integration" testing with a much larger more complex scope to find bugs in.

[Jeroen3]

Yes. I currently see 5040 Tests.
This is a C# project to operate a test bench using various meters and io. Not that embedded perse.
But writing a module with tests allows you to only test the module using the tests and not have to spin up the entire simulator or have an actual product on hand. If your design is properly SOLID this should work very comfortable.

Yes, sometimes you have to write prompt convoluted mocking or simulator classes. But there are limits. I found a bug in a modbus driver. As none of the tests actually used it, we abstracted it away for tests and simulator, only in the full integration this was present.

TDD also guards your requirements. If you make modifications that break other rules, you should be notified by a failing test.

I also wrote some integration tests where I keep a config file of all product variants, load it, and run tests on all of them. Although this only tests the business code, it does give insight if new feature causes issues with older hardware.

TDD is especially powerful for collaborative work if you are able to utilize the github/gitlab automations to run the tests.

[NorthGuy]

The Fifo for example. 

My take on this is a little different. Typically FIFO state is represented by two numbers - head and tail. If you have one producer and one consumer, then producer modifies the head only, while consumer modifies the tail. This makes it thread-safe unless you read/write the state in some awkward way (for example, you may declare head and tail as 64-bit numbers making reads and writes non-atomic). Therefore, there's no need for any synchronization except perhaps memory barriers between load/stores and counter updates which is necessary if producer and consumer run on different cores.

As the speed of data arrival increases, the number of ISRs needed to store them becomes large and the load on the CPU may become too high. In such case the only way to eliminate the interrupts is using DMA, in which case DMA will store data to the buffer. With your viewpoint, DMA works as a different core, so would require synchronization. Therefore it's important to have a FIFO model which doesn't require synchronization. The consumer's code doesn't need to change. BTW: There was a thread several years ago where one of the posters pointed out that STM32 DMA had a bug - speaking in CPU terms, they didn't have a memory barrier between stores and counter updates, so the counter was incremented before the corresponding data was actually stored in memory.

I would typically figure the timing first, then would use the project-to-be as a test bed and added the producer code. Then I would check that the data arrives as expected. Perhaps would try to break it with varying timing of data arrival events. Perhaps would add an unrelated interrupt load for testing purposes. Perhaps would add tests at higher or lower speed. Then I would code the consumer taking data out of FIFO, perhaps by outputting them somewhere. Then I would work further through the data flow coding what consumer is designed to do with the data. And so on. Not exactly TDD, but moving in small increments with frequent testing. This way you don't need to build unit tests and test beds, but you're testing real hardware and real timing.

[Shenhe]

WOw

[paulca]

Yes I normally work in micro-goals the way you describe.  Start with step 1.  Make it work.  Then step 2.

The reason I started with TDD on Unix for this is the hardware didn't exist yet.  It exists today though.  Step 1 was blinkie.  Step 2, CDC Hello World.  And so on and so forth.

Your point on a fifo making it possible to do reader and writer concurrently is something I remember from ring buffers.  (Leaky fifos).  I recall doing that in Java.  Java actually makes it easier as you can pass an object reference from the backing array and even if you delete the array, the object reference is valid until it's disposed of or out of scoped.  In C, you either have to "deep copy" the value back out, or pass a pointer, the later then exposes more than you intended to the client, so you need public/private pointer patterns, it's ugly.

Also, I believe I store "Full" and calculate "Empty".  I'd need to calculate Full too or it could cause issues.

[NorthGuy]

Also, I believe I store "Full" and calculate "Empty".  I'd need to calculate Full too or it could cause issues.

I always store head and tail. Then you can easily calculate the rest:

Code: [Select]

n_used_slots = head - tail;

// producer side
do_we_have_enough_space_for_write = (n_used_slots + n_items_to_write) <= fifo_size;

// consumer side
can_we_read_the_desired_amount = n_items_to_read <= n_used_slots;
can_we_preview_an_item_x_slots_away = x <= n_used_slots;

[paulca]

Well.  I mashed everything together so far into a Cube project.  Commited it to git and with it mostly half written, let claude loose on it.

It was all going well, the end to end IO worked!  It said "Hello" out the USB port.  Only occasionally it says "Hell" or "Helo" and thats tracing deep into the hardware as it appears that the DTACK contracts are being broken somewhere, somehow.  The output suggests, to me and I have seen this before the IO board existed... when CPU RESET is spammed often enough, a different end result occurs ... when running deterministic software which should 100% of the time end up bus halted on an IO Write.... somehow ends up bus halted on the following IO Read.

Claude went down a rabbit hole, in a branch, I let it.  It's using the STM32 edge tricker timer to count the number of edges and it's getting even more bizarre data back there too.  Like the MCU servicing more IO requests than it acked... meaning something else ACK'd it and the bus transaction was closed.

This by the way is the opposite of TDD... it's YOLO and only stop when it doesn't work.

Popcorn and coffee.  While I keep the sane and nearly working version in master.  Claude is now fully spanning the code bases and PCBs.  It has CDC access to the MCU only though.

UPDATE:
Meh.  It's back to me now.  It wants the Digital Discovery wired up and a 10K on IO_DTACK to ground.

The fault being circled is that currently "IO_DTACK" is a bare, unpulled direct wire from the FPGA pin to the header, through the pins, along the IO Board and into the  IO Buscontroller FPGA pin.  It is never pulled and the FPGA pins are not even setup with their weak downs.  The STM32's 40K pulldown seemed to only "dim" the problem and make it rarer.  In the motherboard FPGA that IO_DTACK is a "direct async" fire to nDTACK.  No sync, no check, no validation.  "Any edge is good".  A design choice that has persisted from the 74Series OG design and a liability here, even if it had proper formal pull downs on PCB.

But... Im not fixing that IO_DTACK signal until I prove it's the ONLY fault there.  So.... digital discovery wire up first. 

[hans]

I have written multiple FIFO queues in my codebase...

For byte FIFOs it is often acceptable to leave 1 slot empty to mark when the buffer is full. So my checks are:
empty => head==tail
non-empty => head != tail
full => (head+1)%N == tail.

However, I also have other FIFOs for larger objects (like protocol packets, IPC messages, etc.). Leaving a gap may take a considerable amount of RAM. In this case I add an 'empty' flag that is written by both reader and writer. The readers set it to true (even when there is still more data to be read) and writers set it to false.
Then the empty check becomes head==tail&&empty, and the full check becomes head==tail&&!empty

Some care must be taken when to write this empty flag. I do so at the end of the read/write routine so it is always the most 'fresh'. However, it may also be a bit easier to just make it a critical section so that two threads (IRQ vs main, or 2 RTOS tasks) cannot create a race condition here.
Another risk is that multiple writers can cause some FIFO index to be used twice. In this case, the whole FIFO write/read routines are better to be put in a critical section.

I've tried to use exclusive flags on ARM to start reserving slots and carrying out the writes, but it became pretty complicated to verify the housekeeping of the reserved and committed head or tails. I also found these kinds of race conditions are notoriously hard to test on real metal. So I tried to run these tests in my own RISC-V emulator, but its still something to be sorted out for another day..
So till now I will just use a less than optimal implementation.. However, in general, testing these kinds of objects is highly useful for TDD :-)

[NorthGuy]

It was all going well, the end to end IO worked!  It said "Hello" out the USB port.  Only occasionally it says "Hell" or "Helo" and thats tracing deep into the hardware as it appears that the DTACK contracts are being broken somewhere, somehow. 

USB (or UART for that matter) won't lose characters because of bad contacts. So, this looks like a bug somewhere.

[NorthGuy]

For byte FIFOs it is often acceptable to leave 1 slot empty to mark when the buffer is full. So my checks are:
empty => head==tail
non-empty => head != tail
full => (head+1)%N == tail.

However, I also have other FIFOs for larger objects (like protocol packets, IPC messages, etc.). Leaving a gap may take a considerable amount of RAM. In this case I add an 'empty' flag that is written by both reader and writer. The readers set it to true (even when there is still more data to be read) and writers set it to false.
Then the empty check becomes head==tail&&empty, and the full check becomes head==tail&&!empty

You needlessly overcomplicated your calculations. This led to a gap in your buffer. Then you added even more complications by introducing "empty". Instead of adding "empty", lose +1 and %N. I have posted simple formulae a few posts back. Use them. They won't leave a gap.

It is always better to remove things than to add.

[hans]

I agree that simpler is better, especially in routines that are used as often as queue read/writes. For UARTs these can be called tens of thousands times per second.

Yes the "empty"-flag variant can be a bit complicated at times. I may actually swap that implementation with these formulas..

But I'm not so sure about the 1 slot gap situation. The formulas you posted rely on a 'rewind count' being stored in head/tail, so if fifo_size=256, then head/tail need to be uint16 (or fifo_size=65536, then head/tail need to be 1-bit wider, or uint32 in C).  Otherwise the n_used_slots becomes 0 when the buffer gets to a full state.

I've tried to keep the data types of head/tail as small as possible for the implementation to remain fast on e.g. AVRs. 258 vs 260 bytes SRAM usage for a 256 byte queue is neither here or there, but AVRs would need to push/pop more registers and data to update these counters. So I use this 1 slot gap instead. For 32-bit MCUs it doesn't matter as much..

The +1, %N calculations etc. are required in all FIFO implementations to update head/tail positions and/or index data within the buffer array.

I'd actually say these kinds of exercises are excellent material for unit tests. Especially in languages like C++ it becomes trivial to spin up a new variant of a queue with new fifo_size values and the like.

[Fire Doger]

TDD is so much easier now! I would never go back to develop on hardware from first line.

What is really hard to do is "core" things, interrupts, DMA, cache coherency, volatility, bus connections etc...

For these cases you can either try to simulate all these (not recommended) or introduce Hardware In the Loop (HIL)

For example, currently I design a dimmer, I also designed a Lux to USB adapter.
The plan is to connect both mcu with programmer and the usb lux meter to same pc and let it figure out the graphs...

An other example was the bootloader firmware update, for this I simulated the whole mcu flash contents in PC ram, so the data decryption, decompression, transfer, etc.. was 100% tested.
But being able to respond to UART with busy while the flash was busy during erasing, required HIL to test again and again.

Ps: ofc all these on linux, it's much more easier to integrate them after to your git CI

[NorthGuy]

But I'm not so sure about the 1 slot gap situation. The formulas you posted rely on a 'rewind count' being stored in head/tail, so if fifo_size=256, then head/tail need to be uint16 (or fifo_size=65536, then head/tail need to be 1-bit wider, or uint32 in C).  Otherwise the n_used_slots becomes 0 when the buffer gets to a full state.

Yes, the number of charcters in 256-byte FIFO is from 0 to 256 which would require 9 bits to store this information. If you want 8-bit head and tail, the max fifo you can have without wrap entanglements is 128 elements.

The +1, %N calculations etc. are required in all FIFO implementations to update head/tail positions and/or index data within the buffer array.

Not really. If your FIFO size is power of 2, you just do

Code: [Select]

head += whatever_number_of_items_you_have_just_written;

Code: [Select]

tail += whatever_number_of_items_you_have_just_utilized;
The natural integer wrapping will take care of everything.

Note that this is a "separation of concerns" here as paulca puts it - one things is counting items, another thing is actual reading and writing. You can use the same counting, but different storage methods. Typically you would use an array, but you can come up with different methods of storage if you need to. If you use arrays, making FIFO size a power of 2 helps with storage access as well:

Code: [Select]

buffer[head&(fifo_size-1)] = item_to_be_written_at_head;

Code: [Select]

item_stored_at_the_tail = buffer[tail&(fifo_size-1)];
item_at_offset_after_the_tail = buffer[(tail+offset)&(fifo_size-1)];
item_at_any_index = buffer[index&(fifo_size-1)]; // consumer "owns" everything between tail and the head

[paulca]

It was all going well, the end to end IO worked!  It said "Hello" out the USB port.  Only occasionally it says "Hell" or "Helo" and thats tracing deep into the hardware as it appears that the DTACK contracts are being broken somewhere, somehow. 

USB (or UART for that matter) won't lose characters because of bad contacts. So, this looks like a bug somewhere.

Its definitely a bug.  Most likely signal integrity of that IO_DTACK line.  It looks like I have "Ghost" data acks on the bus.  Something is "ack'ing" bus transactions it shouldn't be.  Like 1 in a million bus transactions.... but thats more than enough to show up.

Claudes timer counter build produced interesting results.  Stuff like 2.8 million IO_DTACK edges, but the IO board counted a dozen or so less IO_DTACKS sent than detected.  The UART output was short of a 2 or 3 characters over 200 bytes or so.

ROM/RAM accesses, they are fast enough to have the data ready in almost all cases without a "wait state", so its highly likely a "ghost" data ack would have no visible effect there.  But on IO addresses, the CPU in nano-second land is waiting for the MCU in microsecond land and thats a very wide time window for an IO_DTACK glitch to cause the FPGA to actually latch the CPU DTACK data ack signal.  The data it gets depends on how well the pull downs did.  0x00 if they won, 0xXX is they don't.

[paulca]

Bit of a ride that one to find more of my dodgy verilog at fault again.  This time with a little help from YoSys.  Same rule applied again.. "Don't trust edges other than your own clock".

The bot was particularly handy when it comes to logic analyser traces.  I would take a boot capture, 100Mhz, 200M samples, 2s, export the events only to CSV 33Mb a go.  The bot never opened it.  It started out of the gate with a python stats analyser with a window size it scanned over the data, then zoomed it in... almost like I was doing with the GUI in parallel.

It all "looked fine" to me at a glance.  But it found 10-17% 'ghost acked' transactions.  What it did, would probably have been were I ended up, either scripting the LA GUI to pattern match the fault out or writing python to do it.  The bot does that all in the time it took me to zoom out and in a few times, 'sample' a dozen transactions and determine it looks fine.  That single little session of 4 traces probably saved me a few days, maybe more.

Anyway.  Hardware bus issue rabbit hole now addressed, I'm climbing back up the MCU and back into C.  Phew.

UPDATE

Code: [Select]

68010 Mini ROM (Paul C_ll 2026) post BIT....
* PASS - CPU BIT
* PASS - EXCEPTION HANDLING
* PASS - TRAP HANDLING
* PASS - RAM CHECK
* PASS - RAM PROBE: 255 KiB RAM Detected

>
With the RAM PROBE disabled it reads 1 quarter region.  It has got 1024MiB.

The hopefully last hurdle was found to be an unsoldered FPGA pin on D7.  As D7 doesn't show up in most ASCII, but does in a carriage return, things got very confusing.

"That's yer problem right there boy!"