Rasberry PI and clones as industrial SBC's?

[SiliconWizard]

Odroid boards use socketed eMMC that you need to buy separately but they are replaceable, and I estimate more reliable and faster than microSD card.

Yes, but they are plugged in with "flimsy" connectors with absolutely no locking mechanism. I wouldn't trust it for an industrial use in any kind of harsh environment.
I'd definitely favor the boards that embed soldered eMMC flash (and of course use "read-only" Linux distributions.)

That said, you still have absolutely 0 guarantee with those "hobbyist" SBCs. They are not designed to be reliable. The other factor is software. One of the points of using those is having a Linux OS. Problem is, practically none have Linux distributions that are dependable enough to be used in an industrial setting IMO. Are you skilled/willing or have the time to be rolling your own?

As always, don't be fooled by the low cost. The real cost for any serious use is another story entirely.

[rx8pilot]

As always, don't be fooled by the low cost. The real cost for any serious use is another story entirely.

You are correct on this one. The hidden costs of applying endless band-aids on the low cost hardware will ultimately make me wish I just designed and built a a system from the ground up to be in an industrial application.

The efforts I have seen so far to put RPi systems into a hardened shell are interesting, but sadly end up being the similar cost as many industrial options. The big question I need to answer on my own is this: How robust does the solution need to be in order to be effective at achieving the stated goals. Technically, I could design and build to aerospace standards with multiple levels of redundancy and tolerance to the harshest of environments....but does that help solve the problem or simply add enormous cost? Is it reasonable to simply have enough redundancy by way of super low cost hardware that faults can be managed?

My basic intent is/was to leverage the ultr-low-cost RPi clones in a fairly large non-real-time, non-safety-critical application of gathering and displaying data related to process control. In that concept, serious reliability can be sacrificed to some extent in favor of having ready to deploy backups that can be dropped in for very little money. The larger scheme can tolerate numerous dead nodes and still be effective. I would think that with some basic precautions in power, IO, and software - the reliability could be at a level where having a few plug-in spares that are used periodically could be enough to actually benefit from the lower costs.

From the customer point of view: This is the $100,000 solution vs the $10,000 solution where the $10,000 solution is advertised and expected to require more hand-holding than the big-boy option but still deliver 95% of the $100k system.

[rx8pilot]

Some Linux based devices create a ram-disk at boot, copy a firmware image into it and run the software from there. This means that the system gets a fresh install at every boot so having a read-only root filesystem isn't needed.

I love this. 

Boot time is not really an issue, so the time needed to create a ram disk is not a problem. This concept would eliminate any need for battery backup or super caps.

[westfw]

I dunno.  Sort of standard "industrial SBC" is like PC104, right?  Not a lot different from RPi boards in some vague hand-wavy way.  You can even get linux SBCs and such in PC104 format.

Except for "support."  Most PC104 vendors exist to support the industrial SBC market, while RPi aims at the hobbyist and educational markets.  That's likely (unfortunately) to end up being a significant problem at some point.  Just the "churn rate" is awful; how'd you like to have a really successful product based on an RPi2 board?  (we used to have heavy debates on replacing custom hardware with "commodity PC stuff" like some of our competitors.)  Industrial product end-customers tend not to be happy with with churn either, and there are all those certification issues and such...

[frogg]

My only real issue with using raspberry Pi's in an industrial application is that the NVRAM on which the OS resides is not particularly robust, nor is it particularly deterministic in terms of performance. That could probably be fixed to some level by hacker types, but depending on the hard real-time  or high availability needs, the amount of effort may quickly eclipse the convenience/cost of the cheap SBC.

Many real-time systems are not designed to have backup power systems, even though many of them end up having them attached.  Rather, they are designed to start up 100% reliably to a known, operable state no matter how badly power has been cycled to the unit.

Like others have said, hard real-time capability is hindered by the OS, but for very hard real-time systems, the CPU may have to provide some guaranteed parameters in how deep the execution pipelines are, the type of speculation, the cache/memory arrangement, and how preemption occurs with interrupts. This is why research is done on "Worst Case Execution Time" as a sort-of-standard metric on real-time hardness.

So I guess in summary - it all depends on how real-time or robust you want it to be. Keeping in mind, of course, that some industrial SBCs have very soft real-time requirements and some are not particularly robust at all.

[rx8pilot]

Like others have said, hard real-time capability is hindered by the OS, but for very hard real-time systems, the CPU may have to provide some guaranteed parameters in how deep the execution pipelines are, the type of speculation, the cache/memory arrangement, and how preemption occurs with interrupts. This is why research is done on "Worst Case Execution Time" as a sort-of-standard metric on real-time hardness.

I definitely do not see a Pi type device as real-time or mission critical. For the real-time tasks, I am expecting to design and manufacture custom MCU based interfaces that run in a stand-alone configuration. The MCU modules will be able to tolerate all sorts of poor treatment and still deliver excellent availability. Those modules would be sending status updates to the Pi on request. The expectation is that the MCU based devices have enough on-board memory to easily deal with any of the typical latencies in the Pi and will continue to operate normally if the Pi burns to the ground.

Essentially, the task of the Pi would be to handle periodic operational updates to the various connected devices and gather data as needed and report that to a server. This arrangement ensures the Pi's are not going to stop production. With that said, if they are fiddly and need regular attention - the crew on the factory floor will likely start a mutiny.

[Zucca]

In my experience a Raspi work fine for about 5 days 24/7 top, then I had always to fight some sporadic issues  .
Not to blame the raspi platform, but me: in my eyes you need to get deep into linux to debug or understand what is going on or to prevent crashes.

Surely a learning experience as you pointed out in another thread, but you need to plan (maybe?) 1 year in advance to get the desired industrial production stability.

[PlainName]

I have a RPi2 at a remote site that I forget about until I need it for some reason (VPN down, so remote access to internal infrastructure via SSH, perhaps). Always there regardless of anything else happening except for two circumstances: 1. mains powercut, and 2. broadband down.

Before installing I put in a software watchdog. Don't know if that's ever tripped, and it wouldn't handle the worst case scenarios, but all I can say is the RPi2 has never let me down. (A quick check indicates it's been there for four years, completely unloved and unmaintained.)

[Zucca]

Interesting to know if I am an unlukcy user or you are a lucky user.
I hope it is me to have bad luck.

[PlainName]

So do I 

[SiliconWizard]

Apart from the mass storage potential issue (SD cards are not all that great, bad things can happen at power loss, etc.), there's one definite thing to consider when using those small SBCs: heat management.

Many of them, RPi included, are sold without any kind of heatsink. And if you think it doesn't need any, think again very hard. Unless you're running some code that is very light on CPU (idles most of the time) AND GPU, the processor (and RAM) gets very hot. You can't let it run for any extended period of time like this. So add a proper heatsink. That will certainly help reliable functioning a lot.

[rx8pilot]

. So add a proper heatsink. That will certainly help reliable functioning a lot.

For sure....before the first boot, I designed and machined my own heat sink for my test Pi.
It has a fan option, but I don't think it is needed unless the ambient temp is unusually high. I did some days long stress tests with CPU/GPU/RAM all running at max, and the temp stayed very comfortable in roughly 90F/ 32C ambient. Not too bad. The RAM did not have a direct connection to this heatsink, but they did not get all that warm either (to my surprise).

A future rev would make thermal contact with the RAM chips.

[Zucca]

For sure....before the first boot, I designed and machined my own heat sink for my test Pi.

You should sell them.

[Zucca]

Did anyone spot the Compute module?

https://uk.farnell.com/raspberry-pi/rpi-compute-module/compute-module-raspberry-pi-dev/dp/2427122?gclid=Cj0KCQiAvJXxBRCeARIsAMSkAprZTE20cRp9w0tYqE-BDVx7agjYIzoF6jtV50PMtYzleN5F6b-XSKEaAorVEALw_wcB&mckv=sYzmG8S34_dc|pcrid|377992558200|kword|rpi%20compute%20module|match|p|plid||slid||product||pgrid|13887941408|ptaid|kwd-90106367354|&CMP=KNC-GUK-GEN-SKU-MDC

https://www.raspberrypi.org/blog/raspberry-pi-compute-module-new-product/

Interesting, please read also the comments:



 

[nctnico]

. So add a proper heatsink. That will certainly help reliable functioning a lot.

For sure....before the first boot, I designed and machined my own heat sink for my test Pi.
It has a fan option, but I don't think it is needed unless the ambient temp is unusually high. I did some days long stress tests with CPU/GPU/RAM all running at max, and the temp stayed very comfortable in roughly 90F/ 32C ambient. Not too bad. The RAM did not have a direct connection to this heatsink, but they did not get all that warm either (to my surprise).

Typically memory is not much of a problem where it comes to heat. On the board I designed I use a standard quarter brick DC-DC converter heatsink which is screwed to the board:


There is a bit of soft thermal interface material between the heatsink and the SoC chip. Ofcourse in my case I could design the PCB to match the heatsink.

[Zucca]

Interesting, maybe a SSD will improve reliability?

[rx8pilot]

Interesting, maybe a SSD will improve reliability?

I would think this configuration would be considerably more reliable. Being faster is a good by-product as well.

From an error situation where the OS has been corrupted, I wonder how difficult it would be to have another partition boot if the primary fails. The alternate partition could rebuild the primary partition from an image and then reboot from the primary. Not sure how to accomplish that in Linux, but surely it is possible, right?

[nctnico]

I have implemented a dual-boot system in the past. If you boot from a (Q)SPI flash rom for example you can have a bootloader (like u-boot) figure out which device (for example SD or Sata) is bootable. Another option is to have the bootloader in the SoC sort it out or both. In a current project the SoC can boot from both eMMC and SD (it will try to read a bootloader in that order) so if the eMMC fails an SD card can serve as a backup. From there the bootloader can figure out which device should be booted from.

In the system I implemented things got complicated because both software updates and data had to be kept synchronised on both devices.

[rx8pilot]

In the system I implemented things got complicated because both software updates and data had to be kept synchronised on both devices.

I suspect that is a common scenario. Although the data updates would have an element of risk - the core OS data would be safe. Perhaps you lose some percentage of the last incoming data, but that seems like a workable risk for many applications.

[nctnico]

Well, in that particular case the data was part of the product (electronic art). But if you are dealing with live incoming data then there isn't much you can do to prevent data being out of sync on two storage devices.

[Zucca]

From an error situation where the OS has been corrupted

It is still obscure to me how the first corrupted OS will then automatically reboot to the second good OS partition.
Maybe it's a kernel panic and it reboots anyway.

[PlainName]

Your OS sets a boot-protected flag once it's running properly. The bootloader checks the flag on boot and if it's not set then the OS is borked and you use the other one. An alternative, if your CPU has it, is to check for the reason of a boot - if it's a power or user event then it's probably OK, but a watchdog timeout is suspicious. You can use a time-from-boot to figure out if it's a one-off thing the OS should handle or the OS itself having issues.

[Zucca]

The bootloader checks

So you can mod the bootloader to do that? Wow, I need to move to Linux soon, it's becoming too attractive...

[PlainName]

It depends on your particular bootloader. We are talking embedded systems, don't forget, and not your average PC. Something has to start the OS load from whatever storage it is on, and that something is usually your bootloader. It will be specific to your system, not the OS.

[SiliconWizard]

The bootloader checks

So you can mod the bootloader to do that? Wow, I need to move to Linux soon, it's becoming too attractive...

The bootloader of what? What platform are we talking about here in the end?

On conventional PC motherboards, the "BIOS" is what you could call this (at least it's one of its tasks). You could take a look at the OpenBIOS and Coreboot projects for instance. They of course don't support everything, and it's not for the faint of heart.

For SBCs, things are not necessarily much easier. The "bootloader" of the RPis, for instance, actually runs on the Video chip, and it's closed source, they only provide binaries. I'm not aware of any existing and fully working open-source replacement, if you find one, please share.