Microcontroller Development Advise

[nctnico]

The only way is by measurement, and hoping that you have captured the worst case performance. While that may be acceptable in some circumstances, it is unsatisfying.

If you are running a fully-fledged operating system such as linux, then the effects of that have to be assessed and measured.

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

[tggzzz]

The only way is by measurement, and hoping that you have captured the worst case performance. While that may be acceptable in some circumstances, it is unsatisfying.

If you are running a fully-fledged operating system such as linux, then the effects of that have to be assessed and measured.

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Maybe that's what you have seen[1] on your system. Now prove there won't be any problems under any conditions.

If a deadline was missed, what would be the consequences? Now answer the same question with the OP's application, hardware, and external system.

Given that the OP has highlighted the importance of Rdson in devices being switched at accurate times, I suspect relatively high energies are being controlled.

[1] BTW, that's not what I see on my (rather old) system.

[SiliconWizard]

That's cool. Talk is cheap though.
So, just show us an example of a "real-time task" running with a few µs latency on Windows and vanilla Linux. Measure real latency, not self-convincing tricks.

On Linux, it's just not possible with the vanilla kernel and base options due to how the scheduler works. Even interrupts are all handled in a single thread by default - that's not the case if you either use a RT-patched kernel, or if you enable the 'threadirqs' option on the kernel boot line. That helps. You can use benchmarks from the rt-tests package to get reliable figures. The 'cyclictest' program for instance.
With a RT-patched kernel, I can get max latencies (it's the max that matters, of course, not the min or average) of about 15-20µs, it's already pretty good and I've never seen better than this on any machine I've dealt with - maybe it can happen, if all stars are aligned. With a vanilla kernel (latest) with the threadirqs option, I can get a max. latency of about 150-200µs. All that with pretty fast machines.

That's a benchmark, and of course, to get that in your applications, you need to carefully craft your programs, configure your kernel properly and create threads with a high priority.

On Windows I don't have recent benchmarks on latency. I've never seen it have lower latencies than the Linux kernel though up to Win 7.

[PCB.Wiz]

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Of course, the examples you just gave, both use significant peripheral assistance, they are not purely OS delivered.

Yes, I have used Windows to design a DAC and logger, (no remote MCU) and that was able to stream continually and have defined timing to sub-milliseconds, but it was not the OS alone at work here.
It used a USB-UART bridge and those have significant buffers that tolerate the clumping effects of the OS.

It works both ways, if you see the CP2102 thread, that shows how just bad 'USB+OS' are at fine granular half-duplex communications.
There are many things to get in your way.

[Siwastaja]

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Audio & video is a specialized task for which the OS has been hand-tailored. Besides, things like smearing is commonplace, especially on linux, has always been. Missed frames, audio playback stutter, loss of syncronization etc. also happens every now and then. You just don't always notice it, and when you do, it's not catastrophic because audio/video on general purpose computer is entertainment.

Look at projects like linuxCNC and you'll see how difficult it is. It tries to bit-bang stepper motor direction/pulse controls on the good old parallel bus ("printer port"), using a customized linux kernel build and heavily tuned userland packaged into its own "distribution" (a debian fork IIRC), and it still usually fails so people got tired to it and started building actual motion controllers, many based on surprisingly simple microcontrollers like 8-bit AVRs, which buffer commands and act on their own.

[tggzzz]

That's cool. Talk is cheap though.
So, just show us an example of a "real-time task" running with a few µs latency on Windows and vanilla Linux. Measure real latency, not self-convincing tricks.
...
With a RT-patched kernel, I can get max latencies (it's the max that matters, of course, not the min or average) of about 15-20µs, it's already pretty good and I've never seen better than this on any machine I've dealt with - maybe it can happen, if all stars are aligned.
...
That's a benchmark, and of course, to get that in your applications, you need to carefully craft your programs, configure your kernel properly and create threads with a high priority.

Yup.

Even then you are still crossing your fingers and hoping you've spotted the max latency. I don't know how you can ensure all the caches are in the pessimal state required for max latency. Start with branch-predictions cache, move on to L1, L2, L3 caches, don't forget TLB caches, nor superscalar instruction pipelines - and then there's all the software techniques such as hashmaps.

Fundamentally, if you care about worst-case performance, then you have to avoid every technique used to improve average performance. Best way is to start with simple hardware and software - preferably designed to guarantee hard realtime.

Or you can say the equivalent of "you can use an elephant gun to kill a fly".

[tggzzz]

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Audio & video is a specialized task for which the OS has been hand-tailored. Besides, things like smearing is commonplace, especially on linux, has always been. Missed frames, audio playback stutter, loss of syncronization etc. also happens every now and then. You just don't always notice it, and when you do, it's not catastrophic because audio/video on general purpose computer is entertainment.

Yup. And that's even with heavily buffered i/o peripherals. For that application the increased latency won't matter.

Look at projects like linuxCNC and you'll see how difficult it is. It tries to bit-bang stepper motor direction/pulse controls on the good old parallel bus ("printer port"), using a customized linux kernel build and heavily tuned userland packaged into its own "distribution" (a debian fork IIRC), and it still usually fails so people got tired to it and started building actual motion controllers, many based on surprisingly simple microcontrollers like 8-bit AVRs, which buffer commands and act on their own.

I didn't realise many people are wasting their lives trying (and failing) to do things that way. Can't say I'm surprised though

[nctnico]

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Audio & video is a specialized task for which the OS has been hand-tailored. Besides, things like smearing is commonplace, especially on linux, has always been. Missed frames, audio playback stutter, loss of syncronization etc. also happens every now and then. You just don't always notice it, and when you do, it's not catastrophic because audio/video on general purpose computer is entertainment.

What you are describing are typically buffering problems stemming from hard drives not keeping up / being busy with something else. I'm watching quite a bit if video on my Linux computer and what you describe doesn't happen because I set the prebuffering to a larger size.

I have done a couple of ultra low latency video player projects using Linux and in those missing frames is not allowed to happen. Data and parameters need to be moved around at precise times governed by the display's refresh rate (which ultimately stems from the vertical retrace interrupt). I actually did measurement by wiggling I/O pins + oscilloscope to verify timing is met and it does. Response times into user space applications are actually quite good and consistent.

[tggzzz]

It is funny that people keep on clinging to this myth. The reality is that OSses like Linux and Windows are perfectly capable of doing realtime tasks requiring single digit microsecond timing accuracy. If they wheren't, they wouldn't be suitable for playing audio and video in a synchronised fashion. Video playback -for example- is tightly coupled to the monitor's refreshrate using the vertical interrupt at the lowest level to avoid flickering and smearing between frames.

Audio & video is a specialized task for which the OS has been hand-tailored. Besides, things like smearing is commonplace, especially on linux, has always been. Missed frames, audio playback stutter, loss of syncronization etc. also happens every now and then. You just don't always notice it, and when you do, it's not catastrophic because audio/video on general purpose computer is entertainment.

What you are describing are typically buffering problems stemming from hard drives not keeping up / being busy with something else. I'm watching quite a bit if video on my Linux computer and what you describe doesn't happen because I set the prebuffering to a larger size.

So? It is sufficent to counter your statement.

I have done a couple of ultra low latency video player projects using Linux and in those missing frames is not allowed to happen. Data and parameters need to be moved around at precise times governed by the display's refresh rate (which ultimately stems from the vertical retrace interrupt). I actually did measurement by wiggling I/O pins + oscilloscope to verify timing is met and it does. Response times into user space applications are actually quite good and consistent.

Remove the adjectives and provide the numbers of the worst case behaviour. Demonstrate why those numbers are more than the worst observed case.

I ran into this data point 15s before reading your response. It is about the PostGres database, and serves to illustrate the kinds of problems that must occur

I think we're starting to hit quite a few limits related to the process model, particularly on bigger machines. The overhead of cross-process context switches is inherently higher than switching between threads in the same process - and my suspicion is that that overhead will continue to increase. Once you have a significant number of connections we end up spending a *lot* of time in TLB misses, and that's inherent to the process model, because you can't share the TLB across processes.

https://lwn.net/SubscriberLink/934940/3abb2d4086680b78/

[nctnico]

Why would a video player suffer from TLB misses? By nature every frame is new data which must be fetched from an SSD and can be disgarded after showing it. That is repeating the same process over and over by a limited amount of code which easely fits in the cache. The limit is the data throughput of the system. A Postgresql database is a different beast entirely; you can't even begin to compare that.

An interesting experiment is to take a multicore SoC and create a square wave (based on the systick counter / monotonic counter) on a GPIO pin using a high priority process that is locked to a core. I'm pretty sure it will have very little jitter as very little gets in the way of executing it and SoCs typically run at >1GHz. I've seen surprisingly good behaviour on bit-banged I2C busses as well (not my choice to make it that way) on low end (single core, 400MHz SoCs) running Linux. It is not like the clock rate is allover the place or messages get interrupted a lot. Not that it matters from I2C though.

[tggzzz]

Why would a video player suffer from TLB misses? By nature every frame is new data which must be fetched from an SSD and can be disgarded after showing it. That is repeating the same process over and over by a limited amount of code which easely fits in the cache. The limit is the data throughput of the system. A Postgresql database is a different beast entirely; you can't even begin to compare that.

An interesting experiment is to take a multicore SoC and create a square wave (based on the systick counter / monotonic counter) on a GPIO pin using a high priority process that is locked to a core. I'm pretty sure it will have very little jitter as very little gets in the way of executing it and SoCs typically run at >1GHz. I've seen surprisingly good behaviour on bit-banged I2C busses as well (not my choice to make it that way) on low end (single core, 400MHz SoCs) running Linux. It is not like the clock rate is allover the place or messages get interrupted a lot. Not that it matters from I2C though.

Go on then. Demonstrate the jitter when bit banging a 1MHz 50% duty cycle square wave.

Then add 205619Hz 732997Hz and repeat. Then add comms to remote controlling PC.

And then provide an answer as to whether being able to use an elephant gun to kill a fly in the desert (but not a town) is a worthwhile exercise.

Don't forget to respond to the points made by various people, not just cherrypick one example.

[nctnico]

Why would a video player suffer from TLB misses? By nature every frame is new data which must be fetched from an SSD and can be disgarded after showing it. That is repeating the same process over and over by a limited amount of code which easely fits in the cache. The limit is the data throughput of the system. A Postgresql database is a different beast entirely; you can't even begin to compare that.

An interesting experiment is to take a multicore SoC and create a square wave (based on the systick counter / monotonic counter) on a GPIO pin using a high priority process that is locked to a core. I'm pretty sure it will have very little jitter as very little gets in the way of executing it and SoCs typically run at >1GHz. I've seen surprisingly good behaviour on bit-banged I2C busses as well (not my choice to make it that way) on low end (single core, 400MHz SoCs) running Linux. It is not like the clock rate is allover the place or messages get interrupted a lot. Not that it matters from I2C though.

Go on then. Demonstrate the jitter when bit banging a 1MHz 50% duty cycle square wave.

Then add 205619Hz 732997Hz and repeat. Then add comms to remote controlling PC.

And then provide an answer as to whether being able to use an elephant gun to kill a fly in the desert (but not a town) is a worthwhile exercise.

No time to set that up right now. Maybe later. But the reasoning to use Linux is that it makes it easier to create a sophistigated GUI (if that is required) without needing a lot of programming experience.

[tggzzz]

Why would a video player suffer from TLB misses? By nature every frame is new data which must be fetched from an SSD and can be disgarded after showing it. That is repeating the same process over and over by a limited amount of code which easely fits in the cache. The limit is the data throughput of the system. A Postgresql database is a different beast entirely; you can't even begin to compare that.

An interesting experiment is to take a multicore SoC and create a square wave (based on the systick counter / monotonic counter) on a GPIO pin using a high priority process that is locked to a core. I'm pretty sure it will have very little jitter as very little gets in the way of executing it and SoCs typically run at >1GHz. I've seen surprisingly good behaviour on bit-banged I2C busses as well (not my choice to make it that way) on low end (single core, 400MHz SoCs) running Linux. It is not like the clock rate is allover the place or messages get interrupted a lot. Not that it matters from I2C though.

Go on then. Demonstrate the jitter when bit banging a 1MHz 50% duty cycle square wave.

Then add 205619Hz 732997Hz and repeat. Then add comms to remote controlling PC.

And then provide an answer as to whether being able to use an elephant gun to kill a fly in the desert (but not a town) is a worthwhile exercise.

No time to set that up right now. Maybe later. But the reasoning to use Linux is that it makes it easier to create a sophistigated GUI (if that is required) without needing a lot of programming experience.

And that will be a world away from your 205619Hz bit-banged output. It will drag in all sorts of software that will make hard realtime behaviour difficult to guarantee by design (or measurement).

Have fun waggling the mouse and typing/entering many many numbers to see ever increasing worst case latency

[Siwastaja]

LinuxCNC did include a test program which instructed the user to move mouse cursor quickly while browsing web and watching Youtube videos trying to catch worst-case jitter. Quite obviously this does not work out in practice because one would need to move mouse cursor while watching Youtube videos for hours straight to get any meaningful idea of worse-case performance, which is quite critical because the CNC tool snaps on one single incident.

As usual, nctnico has not tested his hypothesis, which is not supported by practical experience by all the others.

[nctnico]

LinuxCNC did include a test program which instructed the user to move mouse cursor quickly while browsing web and watching Youtube videos trying to catch worst-case jitter. Quite obviously this does not work out in practice because one would need to move mouse cursor while watching Youtube videos for hours straight to get any meaningful idea of worse-case performance, which is quite critical because the CNC tool snaps on one single incident.

As usual, nctnico has not tested his hypothesis, which is not supported by practical experience by all the others.

I guess you read past my experience doing low latency video players which rely on realtime performance on Linux. The most complicated one involved putting the output on an EGL canvas on Wayland which has several threads in parallel (one for drawing, one for decoding and one for management) all tightly knit together while running a GUI application in the background that can be brought up using a magic key press. Ofcourse this wasn't build on some crappy SoC or is supposed to do all kinds of other random stuff in the background.

[sokoloff]

I guess you read past my experience doing low latency video players which rely on realtime performance on Linux. The most complicated one involved putting the output on an EGL canvas on Wayland which has several threads in parallel (one for drawing, one for decoding and one for management) all tightly knit together while running a GUI application in the background that can be brought up using a magic key press. Ofcourse this wasn't build on some crappy SoC or is supposed to do all kinds of other random stuff in the background.

If you were early or late getting the frame to the SoC or GPU's HDMI driver, your system didn't crash and nothing broke. You and the user might in fact never even notice a micro-stutter. Getting a usually smooth 50 or 60Hz display is a pretty different problem from driving a bunch of GPIOs directly with sub-microsecond jitter. If you were driving the video display directly from GPIOs rather than using a video driver, disregard my comment here.

[nctnico]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

Meanwhile I did a test with an iMX8 platform (I designed a couple of years ago) using this simple program:

Code: [Select]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <time.h>
#include <stdint.h>
#include <fcntl.h>
#include <unistd.h>

uint64_t time_us()
{
struct timespec now;
uint64_t retval;
clock_gettime(CLOCK_MONOTONIC, &now);
retval = (uint64_t) now.tv_sec * (uint64_t) 1000000 + (uint64_t) now.tv_nsec / (uint64_t) 1000;
return retval;
}

#define INTERVAL_US 1000

int main()
{
uint8_t h = '1';
uint8_t l = '0';
int fd = open("/sys/class/gpio/gpio66/value", O_WRONLY | O_SYNC);
if (fd < 0)
{
printf("fd failed");
return 1;
}

uint64_t next = time_us() + INTERVAL_US;


while(1==1)
{
while(time_us() <= next);
write(fd, &l, 1);
next += INTERVAL_US;
while(time_us() <= next);
write(fd, &h, 1);
next += INTERVAL_US;
}

close(fd);

return 0;
}
Then I used cset to make the program have one of the 4 CPU cores for itself entirely which results in this square wave:



While running I decided to start Eclipse on the system and compile a piece of C++ code (about 16k sloc) which took about 4 minutes (make -j 2 to use 2 parallel build processes). This takes the system to it's limits where it comes to processing and memory usage.

Without doing all the system calls and mapping the GPIO controls + reading the systick counter directly into the user space program things can likely be made even better.

[sokoloff]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

Jitter and pulse width are two different specs; the only thing they share in common is the unit.

[tggzzz]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

Jitter and pulse width are two different specs; the only thing they share in common is the unit.

Tee hee  Beat me to it!

Conflating the two does, ahem,  raise doubts about his understanding the points made by several people in this thread.

[nctnico]

I'm perfectly aware of what jitter is but in this case the number Sokoloff threw on the table just doesn't apply to what the OP requires. OP wants 0.5ms with 6 bit resolution. 2^6 = 64. 500us / 64 = 7.8 us.

[NorthGuy]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

OP said 1%, aka 5 us. Your results definitely don't meet this. By a factor of 10.

Even if they did, why bother with 4-core Linux when even a simplest PIC16 can do better? Is this a "justify government spending" contest?

[nctnico]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

OP said 1%, aka 5 us. Your results definitely don't meet this. By a factor of 10.

Even if they did, why bother with 4-core Linux when even a simplest PIC16 can do better? Is this a "justify government spending" contest?

Because the Linux system can also drive a super fancy GUI application while your PIC16 can not. I'm using the particular iMX8 system as a desktop system to do development on. Anyway, I wanted to show that realtime performance is way better than people expected even with something super simple and as I noted there is much room for improvement by bypassing the kernel entirely.

[PCB.Wiz]

OP said 1%, aka 5 us. Your results definitely don't meet this. By a factor of 10.
Even if they did, why bother with 4-core Linux when even a simplest PIC16 can do better? Is this a "justify government spending" contest?

Both have their places.
The OP's project has both research and production phases.
It is often useful to use a more powerful host, to allow better testing and data collection, and then for production deploy a correct sized-MCU.
It also comes down to what the developer is most used to (and thus most productive on).
 
On the general idea of using a more powerful initial host, here is another find from google.

https://www.eevblog.com/forum/testgear/cy7c68013-based-bus-pirate-and-logic-generator/msg1582012/#msg1582012
It uses a FX2 HS-USB bridge, to
generate up to 8 channels of PWM signals is a welcome addition. It is capable of producing frequencies from 1Hz up to 1MHz at a sample rate of 4 MHz.

[tggzzz]

Nobody ever specified sub-microsecond jitter here. Original spec was in the 0.5ms ballpark.

OP said 1%, aka 5 us. Your results definitely don't meet this. By a factor of 10.

Even if they did, why bother with 4-core Linux when even a simplest PIC16 can do better? Is this a "justify government spending" contest?

Because the Linux system can also drive a super fancy GUI application while your PIC16 can not. I'm using the particular iMX8 system as a desktop system to do development on. Anyway, I wanted to show that realtime performance is way better than people expected even with something super simple and as I noted there is much room for improvement by bypassing the kernel entirely.

And even more improvement by throwing the Linux kernel out completely.

[mawyatt]

Didn't mean to create such a controversy, just requesting some help with the development system and microcontroller selections

Anyway, to clarify the pulse details. Each pulse of the 8 sequential pulses just need to match each other within ~1%, and the pulse-width needs to be accurate time wise overall within ~5% over the 0.5 to 5ms range. Regarding the pulse resolution of ~6 bits, this is for the pulse-width range and just needs a resolution of ~5ms/64, or ~78us, or another way of saying the pulse width increment needs to be better than ~78us.

These pulse characteristics are easy to achieve with just a couple 555 timers and support circuitry, and we will be sending off single channel test PCBs today for fabrication, which we'll utilize to evaluate and model load characteristics, many of which are unknown at this time.

Later while we are "learning" about microcontrollers with the kind help of folks here, we'll be developing another custom PCB which will incorporate all 8 pulse channels, load drivers, and some other functions. This will also use 555 timers, and a CD4017 to create the sequential 8 pulse train, and a CD4060 for the single long duration (60~1800 sec) timer pulse. This will be an entry level system, mainly to gather more information, and hone the final specifications/features (hopefully with customer input) for the microcontroller application(s).

Thanks again for all the microcontroller and development system advice & information.

Best,