SBC with 64-bit Linux

[ve7xen]

If your requirement is just development, why not just use a cross-compiler and/or virtualization?

Because cross compiling sucks. It is much easier to develop on a platform directly where you can run a debugger straight from the IDE. Typical Arm64 platforms are powerfull enough nowadays to use for development.

Eh, I disagree, but to each their own. Cross compiling is trivial to set up (aside from the mentioned cases, but that can generally be resolved with a better build process), generally faster, doesn't require a remote box, is much easier to scale to multiple platforms, and can be done anywhere. It also lends itself much more easily to CI/CD.

Nothing wrong with building on real hardware, but it's a much bigger pain in the ass to maintain an entire separate machine for each platform than installing a cross compiler and updating your build scripts, assuming you don't need to run tests on real hardware, anyway.

But if this is for your home playground and the code is only ever going to run on the same box you're building it on, then yes, this makes way more sense. That doesn't sound like the case here, which to me reads like existing code that needs to run on AArch64 too, which would be the poster child for cross-building.

[nctnico]

I would start at the the Single Board Computer Database.

I'm missing lots of boards on this website. For the iMX6 and iMX8 boards maybe 5% is listed.

[nctnico]

If your requirement is just development, why not just use a cross-compiler and/or virtualization?

Because cross compiling sucks. It is much easier to develop on a platform directly where you can run a debugger straight from the IDE. Typical Arm64 platforms are powerfull enough nowadays to use for development.

Eh, I disagree, but to each their own. Cross compiling is trivial to set up (aside from the mentioned cases, but that can generally be resolved with a better build process), generally faster, doesn't require a remote box, is much easier to scale to multiple platforms, and can be done anywhere. It also lends itself much more easily to CI/CD.

Nothing wrong with building on real hardware, but it's a much bigger pain in the ass to maintain an entire separate machine for each platform than installing a cross compiler and updating your build scripts, assuming you don't need to run tests on real hardware, anyway.

I've been down the cross compilation road many times. You'll need a hardware platform to test on anyway so why don't use it for development too? A dedicated platform also offers a somewhat fixated development environment. Over the years I've seen Buildroot break on every new Linux install making it hard to support older products. Using a VM is a good option to keep a fixed development environment but a development system image which runs on the actual target (or a dev. board if you want) is just as easy. On most ARM64 platforms you can run a standard Linux distribution out of the box and storage is cheap so creating your own environment using Buildroot or Yocto/Openembedded is no longer a hard requirement. And you also need to think about the ARM64 platform in general; you can compile binaries for several ARM64 platforms using the same hardware if you like. So you aren't compiling on a hardware target perse but are using an ARM64 computer instead of an x86. It is an entirely different mindset towards doing embedded computing development.

[PlainName]

where you can run a debugger straight from the IDE

You can do that with a cross-compiler. When developing for Linux on MPC5200 I did source-level debugging in the IDE running on Windows. Running code that might crash or otherwise screw up the system you're writing it on seems a bit daft, kind of like sawing a tree branch when sitting on the same branch

[ve7xen]

I'm not really trying to take a stand here - do what works for you. Just pointing out that it is a viable (and free) option that can make managing the development process a lot simpler, especially if you are working on the same code on multiple architectures/platforms. qemu well supports aarch64, so unless you need to test hardware peripherals attached to your SBC, or are doing kernel development or something, you should be able to do testing in a VM even if you are cross compiling, or build in the VM too if you need access to distro build tools etc. Running a VM like this is effectively the equivalent development process as having a real machine, you just don't need the real machine. Debugging 'remotely' is a well-solved problem at this point too, it is built in to gdb, and you're probably going to need to do this if you want IDE integration from your editing machine to your aarch64 hardware anyway.

[nctnico]

where you can run a debugger straight from the IDE

You can do that with a cross-compiler. When developing for Linux on MPC5200 I did source-level debugging in the IDE running on Windows. Running code that might crash or otherwise screw up the system you're writing it on seems a bit daft, kind of like sawing a tree branch when sitting on the same branch

It isn't a microcontroller... Linux systems have full protected mode just like your Windows system. And ofcourse I'm not writing about kernel debugging but debugging an application. Remote debugging sounds nice but in practise it can be finicky in some cases.

[PlainName]

Before that I used Borland C++3.1 to develop an app that ran on an 80186, and we did source-level debugging of that on the PC. No OS involved at all - it was bare metal all the way.

I think it was that particular setup that made using a native target compiler (as opposed to a cross-compiler) the 'norm'. Pretty much anything we developed for before then involved either a cross-compiler or assembler, and many had seriously expensive ICEs. The use of common-or-garden BC++ to develop for a bare-metal microcontroller AND be able to do source-level debugging for the cost of a PC serial port was a big thing. Huge. Instead of an investment of many tens of thousands just to get hello world going, it just needed a £30 compiler and serial lead.

[I am talking embedded systems here. Clearly, developing on Windows, DOS or CP/M for the same machine was not unusual, but no way were they embedded systems.]

[nctnico]

Time has moved on. Nowadays you can hook up an HDMI monitor + keyboard and mouse to an embedded Linux board and execute 'apt-get install <your favourite IDE here>' and start developing on it.

[SiliconWizard]

Time has moved on. Nowadays you can hook up an HDMI monitor + keyboard and mouse to an embedded Linux board and execute 'apt-get install <your favourite IDE here>' and start developing on it.

It takes much more desk space than just hooking the SBC to ethernet and SSH'ing it from your favorite workstation, and the latter is much more comfortable too IMO. And if you're working with several different SBC's, even more so obviously.

You don't need to cross-compile with this setup (cross-compiling would just have the benefit of being usually much faster than compiling on the target, but unless you work on big code bases, that won't matter much.) Just need to send files and issue commands through SSH. It can easily be fully automated. The benefit, apart from being more convenient and comfortable (to me) is also that it's more reliable. I never trust mass storage that much on those SBCs, even for the short term (especially the ones with SD cards, mishaps can happen pretty often, especially when you're fiddling with the boards a lot), so I wouldn't directly develop on them either - you could lose several hours of work in a second.

Oh well, to each their own as ve7xen said.

[nctnico]

Working remote is an option too (which I do as well; I have some headless Linux machines as well) but if you need screen output anyway then using a monitor is just as easy. Git and good quality SD + socket / eMMC make a setup pretty reliable. Most of the things I design are for industrial use so going for high quality connectors isn't a problem. I agree the SD cards and sockets you find on a typical dev. board are low quality / unreliable.

[David Hess]

Time has moved on. Nowadays you can hook up an HDMI monitor + keyboard and mouse to an embedded Linux board and execute 'apt-get install <your favourite IDE here>' and start developing on it.

I am waiting for one with lots of PCIe and memory slots and ports.

It takes much more desk space than just hooking the SBC to ethernet and SSH'ing it from your favorite workstation, and the latter is much more comfortable too IMO. And if you're working with several different SBC's, even more so obviously.

For embedded work, Ethernet interfacing is especially attractive because it is galvanically isolated, at both ends no less.

[andersm]

I am waiting for one with lots of PCIe and memory slots and ports.

You can buy workstations based on the Ampere eMag and Marvell ThunderX2 today. The ClearFog CX LX2K/HoneyComb LX2K aren't quite as expandable, but should be less expensive.

Some of the AArch64 server CPUs apparently can't run 32-bit code, which is one thing to watch out for use as a development platform for embedded software. Also, you will still want to use an x86-64 box for eg. a full Yocto builds, as they're still significantly faster for significantly less money.

[SiliconWizard]

It takes much more desk space than just hooking the SBC to ethernet and SSH'ing it from your favorite workstation, and the latter is much more comfortable too IMO. And if you're working with several different SBC's, even more so obviously.

For embedded work, Ethernet interfacing is especially attractive because it is galvanically isolated, at both ends no less.

Yup, that too. But if you need galvanic isolation, you also need to take care of how you power your board and whatever else is connected to it, many potential chances to break the isolation here. I know it sounds obvious, but even seasoned engineers can occasionally get bitten by that.

[brucehoult]

Some of the AArch64 server CPUs apparently can't run 32-bit code, which is one thing to watch out for use as a development platform for embedded software. Also, you will still want to use an x86-64 box for eg. a full Yocto builds, as they're still significantly faster for significantly less money.

I'd expect 64 bit only CPUs to become more and more common. Running three or four different instruction sets (A64, A32, T32, maybe Jazelle) on the same CPU is expensive in area / cost.  Apple has been using A64-only CPUs in iPhones and iPads for a few years now -- T32 and A64 were both supported starting with iPhone 5s, and I think A64 only starting from iPhone 7.

All ThunderX{2} and Fujitsu CPUs are A64 only.

I don't know whether Android vendors have been able to drop 32 bit support yet. I think I recall one of my ex-colleagues at Samsung telling me that's coming in the next release (purging all 32 bit code, dropping 32 bit libraries and ART)

[NorthGuy]

I'd expect 64 bit only CPUs to become more and more common.

My guess is, five years from now, everything is going to be 64-bit only. Maintaining multiple instruction sets (in hardware and software alike) is just a waste. IMHO everything was just fine in 32-bit world, but you cannot stop marketing, and here we go, well on the way to 64-bit. I don't know whether they are going to switch to 128-bit some day, but I'm sure this will not be on my lifetime. So, 64-bit it is!

BTW: Does RISC-V have SIMD support which is as big as Aarch64 or x64?

[brucehoult]

I'd expect 64 bit only CPUs to become more and more common.

My guess is, five years from now, everything is going to be 64-bit only. Maintaining multiple instruction sets (in hardware and software alike) is just a waste. IMHO everything was just fine in 32-bit world, but you cannot stop marketing

32 bits is certainly enough for many or most embedded purposes, or even 8 bits (with 16 bit addresses). The issue with individually-packaged microcontrollers (as used by hobbyists and small scale manufacturing) is that as process sizes shrink and/or the number of I/O pins increases, the size of your silicon die becomes determined purely by having enough perimeter to accommodate all the I/O wires. If you use an 8 bit core then it can end up as a tiny island in the middle of the chip surrounded by an ocean of unused (but still paid-for) silicon. The incremental cost of putting a 32 bit or even 64 bit CPU core in there can be exactly zero, and they do have benefits in ease of programming and work done per instruction, and even work done per Joule.

and here we go, well on the way to 64-bit. I don't know whether they are going to switch to 128-bit some day, but I'm sure this will not be on my lifetime. So, 64-bit it is!

RISC-V has been designed from the start to accommodate an eventual move to 128 bits, with instruction encodings reserved for the necessary extra instructions. It turns out there are some people who actually would like to have this bigger address space very soon. It's certainly easy enough to build if someone does.

I also don't expect regular desktop or handheld machines to outgrow 64 bit addresses in my lifetime.

If you consider that home computers graduated from 8 or 12 bit to 16 bit addressing around 1976, to 32 bit around 1984 (Mac) or 1985 (Atari ST, Amiga) or 1987 (Compaq 386), and to 64 bit around 2002 (Athlon64) to 2006 (Core 2 Duo) we have a pretty steady increase of around 1.6 bits per year which implies home computers will start to switch to 128 bit addresses around 2046.

My heart says it's slowing down now and will take longer, but who knows?

BTW: Does RISC-V have SIMD support which is as big as Aarch64 or x64?

The currently ratified RISC-V instruction set has no SIMD support at all. There are two complementary standardization efforts underway:

1) the "P" packed SIMD / DSP extension. This was penciled in from the start but when work actually commenced the Chinese company Andes stepped up and said something along the lines of "We have an existing industry-proven over a number of years SIMD/DSP instruction set from our NDS32 ISA which we have already ported into our new RISC-V cores we are shipping to customers and we are prepared to donate it to the RISC-V Foundation".

I believe the Working Group has made some changes to it, but I think not a lot. I haven't paid close attention.

This does SIMD in the 32 integer registers. Maybe that's another reason some people might want 128 bit registers, even though 32 bit might be enough for their memory addressing needs.

2) the "V" Vector extension. This is based on work at UCB through the 2000s and into the 2010s, inspired by the original Cray vector supercomputers. ARM has recently specified the very similar (but more limited) SVE and MVE (aka Helium) vector extensions, but they haven't yet released any cores implementing them (not sure about MVE, but certainly not SVE). Fujitsu have a prototype chip for supercomputers implementing SVE.

The V extension provides a new register file with 32 registers of size anywhere from 32 bits to 2^32 bits with the same program binary working on vector registers of any size. I'm sure no one will ever build physical registers with 2^32 bits, but it's possible someone will make some kind of streaming implementation that appears to the program to. ARM SVE also has 32 registers but they are architecturally limited to the range 128 bits to 4096 bits. I think we'll likely see some RISC-V implementations with vector registers of 16k or 64k bits in the fairly near future.

It's likely many early implementations will use 512 bit registers, as this matches the cache line size, which means it can make near optimal use of the memory subsystem.

Remember Ken Batcher's ~1970 quip: "A supercomputer is a device for turning compute-bound problems into IO-bound problems". The same is true of properly-implemented Vector processing units, except IO has become memory-bandwidth.

[langwadt]

I'd expect 64 bit only CPUs to become more and more common.

My guess is, five years from now, everything is going to be 64-bit only. Maintaining multiple instruction sets (in hardware and software alike) is just a waste. IMHO everything was just fine in 32-bit world, but you cannot stop marketing

32 bits is certainly enough for many or most embedded purposes, or even 8 bits (with 16 bit addresses). The issue with individually-packaged microcontrollers (as used by hobbyists and small scale manufacturing) is that as process sizes shrink and/or the number of I/O pins increases, the size of your silicon die becomes determined purely by having enough perimeter to accommodate all the I/O wires. If you use an 8 bit core then it can end up as a tiny island in the middle of the chip surrounded by an ocean of unused (but still paid-for) silicon. The incremental cost of putting a 32 bit or even 64 bit CPU core in there can be exactly zero, and they do have benefits in ease of programming and work done per instruction, and even work done per Joule.

and the ginormous NRE of modern processes can be spread over a bigger volume with a single design 

[wizard69]

I really can't understand the PI foundations reluctance to get on board with a 64 bit distro.    As such I will not use their boards.   Instead consider the options offered up such as the Odroid C2.    If it is fast enough of course.   The only real question is how well are the boards supported for a specific chip and kernel version, that changes almost daily.

What's your intended use?

My usage is very light. I need to port some software to Aarch64, and I need a platform where I can run gcc, gdb etc. My main requirement is to minimize the time spent settting up the system or chasing system bugs (that's what I mean by "Stable" ). There are lots of SBCs around and they offer images for download, but most of them don't even mention if their images are 32-bit or 64-bit.

Thanks to Bruce I found the Ubuntu distro for RPi. At least this one says it is 64-bit, but it doesn't feel like something which is heavily used by many people (as Raspbian is), so I'm afraid may not be as "stable" as I hope.

[NorthGuy]

I really can't understand the PI foundations reluctance to get on board with a 64 bit distro.

The sentiment in favour of 64-bit is very strong. It  is only a matter of time until they cave in.

The 64-bit Ubuntu works Ok. The 8Gb SD card which I salvaged from my old phone was hopelessly small, so I had to buy 32Gb card. Other than that, there's no problems.

[brucehoult]

I really can't understand the PI foundations reluctance to get on board with a 64 bit distro.

The sentiment in favour of 64-bit is very strong. It  is only a matter of time until they cave in.

I don't know. Not only have they stuck with 32 bit, they've also stuck with pure A32 instruction set, not the more modern Thumb2, just because the original Pi and also Zero don't have Thumb2. That costs them a lot in code size -- 64 bit code would actually be a little smaller, I think.

The 64-bit Ubuntu works Ok. The 8Gb SD card which I salvaged from my old phone was hopelessly small, so I had to buy 32Gb card. Other than that, there's no problems.

I expected nothing else. It works fine for me.

[SiliconWizard]

I think they didn't really have a big incentive to go 64-bit until they released boards with 4GB of RAM (and maybe more in the future.)
I don't think they really cared about code size either, when you can just stick a 64GB SD card for a couple bucks and call it a day...

The Pi was an exercise in optimizing hardware cost, not really anything else. Not that you can't do a lot with them already.

[David Hess]

I think they didn't really have a big incentive to go 64-bit until they released boards with 4GB of RAM (and maybe more in the future.)

You can get a real earful from Linus Torvalds about the advantages of a 64 bit address space which go beyond addressing more than 4 gigabytes of physical RAM.  Study what mmap does for an example.

[brucehoult]

I think they didn't really have a big incentive to go 64-bit until they released boards with 4GB of RAM (and maybe more in the future.)
I don't think they really cared about code size either, when you can just stick a 64GB SD card for a couple bucks and call it a day...

Sure, code size doesn't have a meaningful impact on mass storage or even RAM needs when you have hundreds of MB or a few GB of it, given that few programs have more than a handful of MB of code. On a desktop system it's data, especially GUIs and media files, that eats your RAM.

The main effect of large code size is on energy usage in instruction fetch from cache (few would notice/care unless battery powered) and more importantly the miss rate of the instruction cache which affects both performance (very noticeably!) and again the much more considerable energy consumed by cache refills.

It is interesting what has happened in the RISC-V world. RV32IMA and RV64IMA are perfectly serviceable ISAs (plus FD if you want hardware floating point), with code size very comparable to traditional ARM32 (both are quite a bit bigger than Aarch64). The C extension which adds 16 bit opcodes duplicating some of the 32 bit opcodes is completely optional. But I'm not aware of any production cores that don't implement the C extension -- there are some very very minimal cores for use in FPGAs or for teaching that don't. Everyone finds that the additional few hundred gates are well worth it to get 25% to 30% code size saving. You only need to have a couple of KB of code before the size of core+ROM+SRAM is smaller if you implement C than if you don't.

The Pi was an exercise in optimizing hardware cost, not really anything else. Not that you can't do a lot with them already.

They certainly started a revolution at the price point! Even if various Chinese companies have outdone them now, with Linux-capable boards under $10.

I remember when the entry price for Unix was over $10k! Or a lot more.

It was a big thing when Sun introduced the $4995 SPARC ELC in 1991: 33 MHz, FPU, MMU, 64 MB RAM, 64 KB cache, built into a 17" mono monitor (1152x900?). I can't remember now whether $4995 was without a hard disk (external SCSI) and needed to be network booted, or if it was $3995 without the disk and $4995 included a 20 or 40 MB one.  I still have one I acquired for $50, though it's probably 15 years since I fired it up.

At the time the ELC was cheap even compared to a 386 or 486 PC, and Linus hadn't yet (by a few months) announced the start of his eventually game-changing efforts.

[wizard69]

...
One thing I have noticed, the SD card is going away. The OS purveyors who do not acknowledge this as prolly the ones to avoid.

This is a huge point, things like eMCC, M.2 and other SSD type approaches can dramatically improve performance.   Probably the next important issue is GPU support.   If a board does these two things well it should be considered for any project that has an interactive operator.