EEVblog Electronics Community Forum

Computing => Embedded Computing => Topic started by: technix on May 14, 2021, 05:41:10 am

Title: Stock Debian armel/armhf for ARM-based embedded systems?
Post by: technix on May 14, 2021, 05:41:10 am
As the title suggests, it is about using stock Debian userland for embedded systems. Is this a good idea?

IMO using an established Linux distro for embedded means you are not obliged to maintain a lot of support code, and you do not have to host a lot of open source code yourself for GPL compliance - I can simply say my firmware is Debian + my own secret sauce, and since the distro is largely stock I can just point users to Debian for source code of any GPL code that is not the kernel.
Title: Re: Stock Debian armel/armhf for ARM-based embedded systems?
Post by: Nominal Animal on May 15, 2021, 05:44:03 am
Is this a good idea?
The only problems I see are technical – a lot of Debian userspace "assumes" a desktop/server environment, and may not be fit for an embedded environment.

I would, however, strongly recommend you also publish your exact recipes for rebuilding the firmware image, from the get go.  Make this an integral part of your development cycle.  This may sound intimidating, but there are several very good reasons for this.  The downside is that you really cannot then charge for software licenses anymore (like e.g. Mikrotik RouterOS, Microchip XC compilers based on GCC, and so on).I write this as both an embedded dev myself, as well as an end user.  If you disagree with my main three points above – feel free! –, I don't think basing your embedded dev on a public distribution like Debian makes sense for you.  (I am trying to describe a pattern that seems to work in practice.  If you don't like the things that I believe make this pattern work well, I do not believe the pattern is fit for you. Then, rather than try tweaking the pattern to fit for you, I would recommend starting with a completely different pattern.)

For a very specialized appliance, constructing the userspace from (forked) GPL'd sources (that you'd keep in customer-accessible git, for example), can be a better choice.  I believe it depends exactly on the set of GPL services/applications you need, as the core userspace stuff is actually rather easy; see e.g. Linux From Scratch (  In particular, you can (SHOULD, in this case!) customize the OS for your embedded needs, and drop parts you really, really don't want or need in an embedded device (systemd, for example).  You don't need a complicated build machinery like e.g. OpenWRT, because you only have a small set of hardware you need to support, and a simple automated build system will suffice.  (Although, I prefer to do the firmware builds in a layered form, with the core OS "image" built ALFS-style, with different appliance subsystems or services as "overlays" on top.  When building a firmware image, each layer is in a separate tree, then just combined into one and compressed into the firmware image.  Modularity for the win.)

So, if your embedded dev can use majority of GPL'd applications, and is complex rather than simple, I'd definitely go with Debian as the base, using the suggestions earlier in this message.  Otherwise, I personally would assume the full maintenance burden.
Title: Re: Stock Debian armel/armhf for ARM-based embedded systems?
Post by: technix on May 16, 2021, 11:20:38 am
I'd start with a Debian Server build and build from there, since it is pretty minimal. Obviously the kernel has to be my own, packages as source-available debs the same way Debian did theirs. As of the proprietary blobs, I'm going to research how Canonical implemented their Advantage system and package them as binary-only debs.

I intend my products to pull packages not only from my own servers but also Debian official servers, so bugfixes upstream can be automatically applied without me responding.
Title: Re: Stock Debian armel/armhf for ARM-based embedded systems?
Post by: DiTBho on May 16, 2021, 11:51:11 am
I wrote an entire OS just to play Tetris on the Playstation, so using Debian should make sense, anyway, a lot of products for 3D printers made by ACME System are actually based on Debian/ARM server + Octoprint-something.
Title: Re: Stock Debian armel/armhf for ARM-based embedded systems?
Post by: Nominal Animal on May 17, 2021, 01:36:18 am
I intend my products to pull packages not only from my own servers but also Debian official servers, so bugfixes upstream can be automatically applied without me responding.
Yup; sounds like the device is closer to a server than a single-purpose appliance (say, a TV or a talking toaster) with relatively "lots" of software, so this approach should make sense.

I've built binary only packages using the naïve direct method (dpkg-buildpackage -b -us -uc, nowadays debuild -b), so the result does not have a source package at all – for example, for things like Plymouth bootsplashes, which do not contain ELF executables, just images and scripts, and therefore do not have any real "sources" anyway.

Another way to look at the situation is to consider it a custom Debian derivative, with specific proprietary blobs (your own) enabled (and also provided by your own repo).

The reason the way you or any business approaches this situation is so important, I believe, is that it dictates the unstated expectations for both yourself/the business, as well as the customers.  In all cases in the spectrum – from a Debian derivative, to your own full stack source three –, any licensing issues are rather simple to solve while still keeping the core product features proprietary; it's just that often the solutions depend on the approach.  (There are some difficult licensing questions when you need a kernel with proprietary drivers, that you do need to talk with a lawyer or some experienced kernel developers like Greg KH to solve; typical solution is a two-part driver with the GPL part exporting a clear interface the proprietary part uses, so that a "knowledgeable" developer could replace the proprietary part with a GPL'd one performing the same functions.)

In particular, lurking on LKML or related mailing lists to see how the core devs have reacted to similar drivers in the past, gives a good idea how not to do it.  Placing oneself as a kernel developer, and thinking of how to do it with the least amount of pain (for oneself as a kernel developer), then working on top of that, seems to work.

Similarly, incorporating proprietary code into GPL packages is always problematic.  Usually, the proper/acceptable answer is a modular GPL'd interface, to which a clearly defined (but just happens to be proprietary) part can interface to, always with a clear description of the full interface.  The last is the key: you do NOT want to reopen cans of worms and do it like graphics drivers or ALSA does.  For kernel drivers, it is always a bit of a tight rope walk to try and open up the interface, while keeping only the implementation proprietary.  (I wish more developers realized the last: for me, it well balances the needs of proprietary licensed software, with the copyleft requirements of GPL; so much so that I have no problem working with either.)

On the other hand, having a distribution (or equivalently, a GNU/Linux based firmware image) with both GPL and proprietary packages, is no problem at all, because the standard interfaces (standard C library, other standard dynamic libraries with multiple compatible implementations; inter-process communication, sockets, etc.) are considered a boundary for software licensing.  (The US Supreme Court decision in Oracle v Google case (,_Inc._v._Google,_Inc.#Decision) is informative.  The vast majority of C/C++ developers in particular operate wholly upon the expectation of interfaces described in public header files and elsewhere to be such a license barrier: that writing your own code from scratch, but implementing the same interface as some other software, does not create a derivative work of that other software.)

One very tricky/problematic/hard/prickly question remains: will you try to stop your end users from replacing the firmware image with something completely different?

On Intel/AMD x86 land, there is a feature called Secure Boot.  When enabled, the hardware refuses to boot unless the boot image has been cryptographically signed by Microsoft.  (This is because most BIOSes only contain a Microsoft root certificate.)  As you can imagine, if this feature cannot be turned off in the BIOS, most Linux users are very, very unhappy.  (There is a reason why ASUS server motherboards are basically Windows-only, and unheard of in Linux datacenters, clusters, and HPC, and it is not really a technical one.  ASUS's choices when Secure Boot came out burned a lot of old Linux greybeards, a lot of whom are in charge of building said datacenters, clusters, and HPC in general, and don't want to go through the same troubles again.)

I believe it is good to understand the long-term implications of such choices, and make sure they align with your business strategy.

As to regulatory requirements, make sure your documentations specifies that the thing you're selling has two components, software and hardware, and that you guarantee nothing for either part alone, only for the combined product.  This means that your regulatory responsibility ends if an user replaces either the software stack, or the hardware – this is similar to the "non user serviceable parts inside" sticker, or "guarantee void if broken" stickers on hardware; neither is actually legally exactly correct, but they convey the right idea, which is enough.  This has already been solved for basically all countries in the world, wrt. radio frequencies.  Basically, a majority of WiFi Access Points and routers run Linux. A decade or so ago, there was a lot of discussion on how to restrict the hardware to the frequency bands allowed in each geographic jurisdiction, since these vary a lot.  Some suggested that it would be impossible to allow Open Source implementations, because the users could then just modify the frequency band selection, and therefore break the restrictions; stricter hardware restrictions were required, according to them.  I guess everyone slowly realized that an user could just buy the hardware elsewhere, and achieve the exact same thing..  In any case, if regulatory compliance comes up, looking up the Linux (and OpenWRT/DD-WRT) mailing list posts and blog posts and tech magazine articles, shows up how it ended up being resolved in the WiFi land.  In short in practice, the device should be compliant by default, and yell (either in the UI or in the documentation; in the documentation suffices, although not without grumbles) at the user when they fiddle with knobs that may affect regulatory compliance.  So, if you expect to sell under different regulatory requirements, do design your product so that the user only needs to select the geographic location, which you internally map to a set of regulatory requirements/rules, and apply to all parts.  That's how other existing global products do it, anyway.

For any litigation-happy USians, I am not a lawyer, and none of my posts anywhere are legal advice.  (In the rest of the world, writing opinions about various laws in various jurisdictions tends not to require a license or specific utterances to avoid liability.  And people are expected to know without an always-visible label that using a keyboard strains our wrists, or that putting an animal in a microwave oven will kill them. The unskippable FBI warning in DVDs and BDs mastered for Nordic audiences and sold in Finland – just try to sue me, G-man, please – turned me from an avid video renter into an illegal downloader a couple of decades ago; nowadays I use legal streaming services like Youtube, though: I find better entertainment than Hollywood produces there.)