Benefits of using Embedded Linux in real world projects

[DrGeoff]

If you want real-time performance (eg rate-monotonic tasks) then you are better off using a RTOS. Linux is not a RTOS.
Many applications don't require precise realtime performance so Linux work fine out of the box.

[peter-h]

NXP's iMX series is a good (and about the only) choice where it comes to versability and long production runs

That is probably why few people do this for industrial stuff. Actually the old Intel 186 was around for many years for this sort of thing, running some sort of embedded windows.

But every box I have seen running linux was crap in performance. Draytek went from their own OS to linux and even just logging into the config page takes many seconds. I have a 4G ethernet modem right here which also runs linux and is incredibly slow. Putting linux in a box seems to be the standard chinese way of doing stuff (easy to rip off another design, and bring out a product in weeks or a month or two) and it produces very slow products.

[shapirus]

Putting linux in a box seems to be the standard chinese way of doing stuff (easy to rip off another design, and bring out a product in weeks or a month or two) and it produces very slow products.

There is nothing inherently slow in Linux, so it must be a result of poor configuration of the userspace programs.

[nctnico]

If you want real-time performance (eg rate-monotonic tasks) then you are better off using a RTOS. Linux is not a RTOS.
Many applications don't require precise realtime performance so Linux work fine out of the box.

For that purpose many modern day SoCs have an extra ARM microcontroller (typically Cortex-Mx where x is a number) inside that can do tasks that require sub 1ms timing. Still, Linux has quite a few realtime tasks in order to make things work and you can get quite far with using Linux for realtime stuff as well. It all depends on how many processes you run and their priorities. Remember that on an embedded Linux system you have far better control over what runs when compared to a desktop PC / server.

Putting linux in a box seems to be the standard chinese way of doing stuff (easy to rip off another design, and bring out a product in weeks or a month or two) and it produces very slow products.

There is nothing inherently slow in Linux, so it must be a result of poor configuration of the userspace programs.

In my experience slowness on such systems is mainly caused by using a slow/crappy flash storage medium that makes loading/starting applications slow.

[zilp]

> However, you want to go for a full-custom design. Don't use modules as you don't have control over the changes and you end up having to modify the software to make it compatible with new module versions.

Well, either that, or split the design into one long-term "core hardware" that does all the low-level hardware interfacing and one "control plane" processor that runs linux and interfaces the "core hardware" through a standard interface like ethernet or even just RS232, so it's trivial to swap it out for essentially any other processor that has an ethernet interface or two and that can run linux.

After all, one of the big advantages of linux is that it comes with a ton of drivers that get abstracted to a common API, so if you write software that uses an AF_PACKET socket to interface your low-level hardware via ethernet, say, you can just swap out the CPU for a different architecture, the NIC for a different chipset (or SoC MAC or whatever), recompile, and things just work without any need to touch your software.

Also, Olimex sells a buch of boards intended for industrial use with long-term support, some based on iMX, but also others.

[nctnico]

That won't help because you typically want to customise the bootloader and kernel to match the hardware. And then you have to do extensive regression testing. I've been down the module road a couple of times and it sucked every time. It is most certainly not a simple as you think. The best way is to keep the hardware as identical as you can over the lifetime of a product.

[zilp]

> That won't help because you typically want to customise the bootloader and kernel to match the hardware.

Hu? Match what hardware, when there is no hardware integrated with the "control plane board", other than what's mounted on the board itself and presumably supported by the bootloader/device tree that comes with the board? I mean, the whole point of that sort of setup is to make that part as un-specialized as possible, ideally to the point where you could just hook up a laptop instead or something!?

[nctnico]

> That won't help because you typically want to customise the bootloader and kernel to match the hardware.

Hu? Match what hardware, when there is no hardware integrated with the "control plane board", other than what's mounted on the board itself and presumably supported by the bootloader/device tree that comes with the board? I mean, the whole point of that sort of setup is to make that part as un-specialized as possible, ideally to the point where you could just hook up a laptop instead or something!?

That goes out of the window if you need a specific form factor, thermal management, interfaces (*), or other special requirements like a centered boot logo, etc. One particular project I designed/developed has a setup you outline for safety reasons but still both the embedded Linux and controller board are full custom to meet the form factor requirements. Also keep in mind that a module relies on making contact reliably with a carrier board. Reliability of interconnects is not a given and are best to be avoided where possible.

* Try to find an exact replacement for a module with a different SoC for example. You'll see that this is near impossible due to technology progressing and new interfaces replace the older ones. For example: try to replace a module with a Sata interface with a modern day module.

[zilp]

> That goes out of the window if you need a specific form factor, thermal management, interfaces (*), or other special requirements like a centered boot logo, etc.

Well, yeah, obviously you design to avoid all of that?!

Like, you leave enough space that you could easily fit in any reasonable future SBC, probably don't mount the board directly, don't do a tight thermal design, avoid specialized or likely to be deprecated interfaces, ... and forget about a boot logo? ;-)

> Also keep in mind that a module relies on making contact reliably with a carrier board. Reliability of interconnects is not a given and are best to be avoided where possible.

Well, yeah, depends on the application, of course, but obviously you don't want to use unreliable connectors ...

> You'll see that this is near impossible due to technology progressing and new interfaces replace the older ones. For example: try to replace a module with a Sata interface with a modern day module.

Well, yeah, there is a reason why I mentioned ethernet and RS232 ;-)

As for storage, I guess I would consider that part of the "SBC assembly"? I mean, all of this obviously depends on the application, there obviously are many applications that really aren't a good fit for this approach, but it's an option to consider that can considerably reduce design effort where the limitations of the approach don't matter much.

[peter-h]

it's trivial to swap it out for essentially any other processor that has an ethernet interface or two and that can run linux.

Ah, yes, very easy

Like so much, it depends on whether you are just working for someone (bread on the table = bread on the table) or it's your own company

The best way is to keep the hardware as identical as you can over the lifetime of a product.

Exactly, and hard to do this if you need many megabytes of RAM when a more normal embedded version might not need any external RAM.

[nctnico]

Industrial grade DDR memory tends to stick around for a long time. I designed a product around a TI Omap processor + DDR memory little over a decade ago which is still in production and it is not a problem to buy the memory today.

[nctnico]

As for storage, I guess I would consider that part of the "SBC assembly"? I mean, all of this obviously depends on the application, there obviously are many applications that really aren't a good fit for this approach, but it's an option to consider that can considerably reduce design effort where the limitations of the approach don't matter much.

You have to keep in mind that if you have 2 seperate systems, you'll need to have a very clear seperation of functionalities. If that is even possible! In the project where I used this approach a lot of time was spend on defining the commands between the HMI and controller board. And then there is dealing with error conditions and recovery. The safety requirements from the customer made this seperate setup a good choice. However, it is not a good choice when the only goal is to be able to swap SBCs later on. Dealing with the mechanical side alone (mounting brackets, thermal, power) makes this an economically unwise decission. Better use components that have long availability from the start and design a solution that fits without too much fuss.

[matrixofdynamism]

For multicore ARM processors, is the approach that one core runs Linux while the other core runs bare metal code or RTOS so we can get best of both worlds in a single system?

[SiliconWizard]

For multicore ARM processors, is the approach that one core runs Linux while the other core runs bare metal code or RTOS so we can get best of both worlds in a single system?

Depends on what the "other core" is. If it's some Cortex-Mxx, sure.
But if it's the same kind of core, a Cortex-Axx, good luck writing "bare metal" code for it. While it can be done, it's usually not supported by anything official and is often a major PITA, riddled with potential pitfalls.

[matrixofdynamism]

We can't write bare metal code for A series ARM? Why would that be? I did not know that.

[nctnico]

We can't write bare metal code for A series ARM? Why would that be? I did not know that.

Ofcourse you can. You could use u-boot as a starting point for example.

There are also domain specific RTOS for SoCs. And you can go even further using virtualisation. The iMX8 -for example- allows to run virtual machines so you can have a couple of cores running safety critical engine management while the other cores are handling the instrument cluster / entertainment / navigation system of a car.

[SiliconWizard]

We can't write bare metal code for A series ARM? Why would that be? I did not know that.

Who said that?
I said it was usually a major PITA in practice. Please have a go and come back with your results. We're just exchanging words here, reality takes a bit more time.

On top of what I mentioned, if you have a multi-core Cortex-A and want to run Linux on it, you'll have to configure and build the kernel in order for it not to grab all cores but leave one alone that you can use with your bare metal code, that's an additional burden also riddled with pitfalls. The bootloader will also be fun to write/configure.

Sure going with VMs is probably "easier" to work around the above, although that's now something else to configure and set up.

[Nominal Animal]

Putting linux in a box seems to be the standard chinese way of doing stuff (easy to rip off another design, and bring out a product in weeks or a month or two) and it produces very slow products.

There is nothing inherently slow in Linux, so it must be a result of poor configuration of the userspace programs.

Most companies hire wild monkeys or hormone-crazed high-schoolers to do Linux systems integration, based on the end results.  Spit, bubblegum, and cargo cults.  "Unix philosophy?  That is so outdated, we live in the 2020s now."

Proper systems integration –– that is, taking a Linux kernel, configuring it, picking the userspace tools and services for the base OS, and the applications or services the appliance needs, and combining them all into a reliable, working system –– takes expertise; generalists who know what they are doing.
Unfortunately, very few organizations are willing to spend the time and resources to get an actual professional to do this.

[Simon]

For multicore ARM processors, is the approach that one core runs Linux while the other core runs bare metal code or RTOS so we can get best of both worlds in a single system?

You need a separate subsystem or you will always have a problem with managing the resources. What we did was use an M0+ micro controller on a board that did all of the electronic stuff that our RPI/Rock4 board just would not have as a SBC and then talk to the SBC over serial. So we had a tacho wheel going around, that was done in embedded.

[Thomas]

If you want real-time performance (eg rate-monotonic tasks) then you are better off using a RTOS. Linux is not a RTOS.
Many applications don't require precise realtime performance so Linux work fine out of the box.

For that purpose many modern day SoCs have an extra ARM microcontroller (typically Cortex-Mx where x is a number) inside that can do tasks that require sub 1ms timing. Still, Linux has quite a few realtime tasks in order to make things work and you can get quite far with using Linux for realtime stuff as well. It all depends on how many processes you run and their priorities. Remember that on an embedded Linux system you have far better control over what runs when compared to a desktop PC / server.

Putting linux in a box seems to be the standard chinese way of doing stuff (easy to rip off another design, and bring out a product in weeks or a month or two) and it produces very slow products.

There is nothing inherently slow in Linux, so it must be a result of poor configuration of the userspace programs.

In my experience slowness on such systems is mainly caused by using a slow/crappy flash storage medium that makes loading/starting applications slow.

I use pigpio in Python on a Raspberry Pi for measuring pulse width from a 433MHz radio module.
5us resolution, pretty impressive. I thought I had to use an external uC for this, but it works fine on the Pi itself.

https://abyz.me.uk/rpi/pigpio/

Remains to be seen how well it works when I pile on other things