Crazyness: C and C++ mixing

[Nominal Animal]

In essence, what I explained above, is that in this "freestanding C plus subset of C++" environment, we can define the programming language pretty much as we want.
If we do not, and rely on existing standard libraries or subsets of them, we will have to make the kinds of compromises Arduino environment suffers from.

Therefore, I once again claim that it is better to start from minimal set of assumptions and expectations, or you'll bind yourself to suboptimal solutions.

I have shown above that const pointer parameters can be redefined to mean "the target is in ROM/Flash".
With C or C++ specific features and the linker, we can redefine const to mean "this object is in ROM/Flash", and have the compiler generate correct address space access instructions.

For code compiled using gcc, we can use the __flash attribute, and the compiler will handle everything for us.  For example, on AVRs, if you declare const __flash struct mystruct  foo = { ... };, then foo will be placed in the .progmem.data section, and all accesses to members of foo will use the LPM instruction (or equivalent).  The compiler knows it is in a different address space, and will honor that.  _Generic does include __flash in type specifications, so we can create generic functions (like strcmp() etc.) with variants for the different address space parameters.  Again, avr-libc already has these _P() and _PF() variants, so "that will bloat code" is not really a valid argument; all this does is let the programmer choose how they balance RAM and Flash/ROM use, instead of forcing a specific choice on them.

For code compiled using g++, we cannot use __flash even inside extern "C" { ... } as g++ is a C++ compiler and not a C one, and does not have named address space support.  This means that while we can trivially have const objects and variables placed in ROM/Flash, it is much harder to get g++ to generate the correct accessors.  One solution is to implement dedicated types/classes for objects and variables stored in ROM/Flash.  An even better solution is to use Clang for AVR, since it supports named address spaces in C++ also.

Obviously, it is a completely different question whether one should do that to const or not.  (The way I'd prefer is a bit more complicated.)
My point is that it is possible, if you just release your fixation on the standard hosted environments, and accept starting fresh.

[netmonk]

From https://gcc.gnu.org/onlinedocs/gcc/Named-Address-Spaces.html

As an extension, GNU C supports named address spaces as defined in the N1275 draft of ISO/IEC DTR 18037. Support for named address spaces in GCC will evolve as the draft technical report changes. Calling conventions for any target might also change. At present, only the AVR, M32C, RL78, and x86 targets support address spaces other than the generic address space.

So i dont think such thing is available for esp32 from what i read. Or may be i misunderstood something.

[westfw]

use Clang for AVR, since it supports named address spaces in C++ also.

Really?  I thought the C++ folks had explicitly rejected the idea.  Was that just the Gnu GCC C++ project?

i dont think such thing [as named address spaces] is available for esp32 from what i read.

I didn't think ESP32 needed anything?  at least, the the Arduino "AVR Compatibility" features have null macros for things like PSTR and PROGMEM.
ESP8266, OTOH, is worse than AVRs (IIRC, flash can only be accessed 32bits at a time, or something like that.)  It's somewhat to Arduino's credit that their F() and related "hacks" can be made to work on ESP8266.

[Nominal Animal]

So i dont think such thing is available for esp32 from what i read. Or may be i misunderstood something.

Only Harvard architectures, with separate address spaces for RAM and ROM/Flash, need the support.  Architectures where there is a single address space, like ARMs (and RISC-V's, I believe, like ESP32-S2), you do not need it at all.

On Harvard architectures, the same address can refer to different things, depending on which instruction is used to access it.  On AVR, normal machine instructions access RAM, and the LPM/eLPM instruction accesses ROM/Flash.

use Clang for AVR, since it supports named address spaces in C++ also.

Really?  I thought the C++ folks had explicitly rejected the idea.  Was that just the Gnu GCC C++ project?

Yes, Clang does support named address spaces, even in templates.  See the rationale paper (PDF).

You only need to add __attribute__((address_space (1))) to the variable or the type, and Clang/LLVM will generate the correct access instructions.  A minimal example, example.cpp:

Code: [Select]

const volatile __attribute__((address_space (1))) int  foo = 5;
int get_foo(void) { return foo; }
Compile using clang++-10 -ffreestanding -O2 -Wall -target avr -mmcu=atmega32u4 -c example.cpp, and avr-objdump -D example.o will tell you

Code: [Select]

Disassembly of section .progmem.data:

00000000 <_Z7get_foov>:
   0: e0 e0        ldi r30, 0x00 ; 0
   2: f0 e0        ldi r31, 0x00 ; 0
   4: 85 91        lpm r24, Z+
   6: 94 91        lpm r25, Z
   8: 08 95        ret

0000000a <foo>:
   a: 05 00        .word 0x0005 ; ????
as expected.  (The address of <foo> shows as zero, because avr-objdump -D/-d does not apply relocations; this too is as expected, and will be fixed when linked to the final binary.)

Similarly, defining e.g. typedef  int  rom_int __attribute__((address_space (1))); will put rom_int x; in .progmem.data section, and generate LPM instructions for accessing it.

I am not sure which version of clang (on Ubuntu, which this particular machine I'm using runs on) added AVR support; clang++-6.0 does not have AVR support, but clang++-10 does, and both versions are standard (Universe) Ubuntu packages.  So somewhere in between.  G++ simply ignores the attribute (with a warning if -Wattributes is enabled), and obviously generates wrong code.

As I understand it, the C++ standards committee and GCC C++ frontend developers have rejected named address spaces as "too hard to implement correctly for templates et cetera", but the clang/llvm folks just went and implemented it nevertheless.

It is quite likely that right now, clang is the better C++ compiler for AVR, but I haven't done enough testing (or even tried to find related LLVM/clang/GCC bug reports) to be sure; but I definitely would not use GCC 9 or later on AVR because of the known-but-not-yet-fixed bugs.

My own (limited!) testing on Pro Micro clones (AVR, ATmega32u4, with native USB) does indicate that for embedded code doing command/response and data transfer work on behalf of the host computer, the C++ overloaded code with literal strings in Flash (and essentially a PROGMEM/ROM/FLASH attribute that differentiates types at compile time as being located in ROM/Flash instead of RAM), yields simpler and more intuitive code, and uses less resources.  The fact that some base functions (especially print family, for formatting string data to a serial connection, be it USB or UART or SPI or whatever buffer) do end up compiling to more than one variant, does not seem to cause the bloat you feared would occur.

Then again, that –– sensor interfaces, small external displays for embedded devices running Linux on an SBC, and so on –– is the sort of thing I'm interested in, so I do not claim it is a surefire fix for all the faults in Arduino!  I only claim it leads to more intuitive code, and gives the programmer full control over where the data is placed.

Hopefully, others can see how powerful this is, if one is not tied to existing code, especially existing standard library implementations.  Me, I'm quite happy to implement my own low-level routines using extended inline assembly, so I'm happy as a clam here.

[magic]

FYI, modern AVRs have memory mapping of flash, accessible with LD/ST.

And I'm not sure if I understood the previous discussion correctly, but I'm pretty sure you cannot substitute 'const' for address space identifier in absence of proper address spaces support in the compiler. Too easy to screw things up with casting. Nevertheless, I think some ancient GCC actually worked like that anyway, before they added 'progmem'.

[Nominal Animal]

FYI, modern AVRs have memory mapping of flash, accessible with LD/ST.

Perhaps you should clarify what you mean by "modern", considering ATmega32u4 is still in production and heavily used, and sadly affected by Chipageddon too.  (I know which ones you mean – not the 8-bit ones –, but others reading this thread may not.)

[magic]

So-called 1-series and 0-series. Perhaps XMEGA too. They aren't fully compatible with traditional AVR, hence somewhat obscure. They are 8 bit, but peripherals have changed.

[DiTBho]

They aren't fully compatible with traditional AVR, hence somewhat obscure. They are 8 bit, but peripherals have changed.

Yup, two years ago, I abandoned them for exactly this reason.

[Nominal Animal]

So-called 1-series and 0-series. Perhaps XMEGA too. They aren't fully compatible with traditional AVR, hence somewhat obscure. They are 8 bit, but peripherals have changed.

Thanks!  I wasn't aware that e.g. TinyAVR 0, 1, and 2 all have an unified address space!  I thought it was only with the very new special-purpose AVRs.

The chip naming is odd.  I guess for ATtinyNFV and ATtinyNNFV, N or NN is Flash size, F is the family (0, 1, 2), and V is variant indicating number of pins and possibly SRAM size.

At the hardware level, they still have separate SRAM and Flash address buses; the TinyAVR core just maps them to a single unified 16-bit address space accessible with the standard LD/ST machine instructions.  In the unified address space, I/O registers, EEPROM, and SRAM are in the lower half of the address space (0x0000 to 0x7FFF), and Flash is in the upper half of the address space (starting at address 0x8000).

The Flash address space can still be accessed using LPM, but then the addresses start at 0x0000.  Essentially, the data address space is shrunk to 15 bit addresses, and the most significant bit enables Flash address space translation, so that Flash addresses 0x0000-0x7FFF are accessible in the data address space at 0x8000-0xFFFF.
(According to the TinyAVR 2 datasheet, indeed, loading a byte from address 0x8000 using LD is the same as loading a byte from address 0x0000 using LPM.  Funky!)

So, the complex answer is that TinyAVR 0, 1, and 2 families do have Harvard architecture and do support LPM for Flash access, but they also have address magic in the hardware so that Flash is also available in the normal data address space.

[magic]

It's FFPI: flash, peripherals, I/Os.
RAM is proportional flash.

Yes, it's more or less a standard AVR core but something traps loads above 0x8000 and redirects them to flash.

I wasn't aware of 2-series, it looks like they lose some peripherals for a better ADC.

[DiTBho]

address magic in the hardware

if you implement a RISC softcore in FPGA you basically have the same problems and you will probably end up with the same solutions:

• "software bridge" dedicated instructions
• "hardware bridge" dedicated circuits, usually achieved with a dual port mapping

I call them "bridge" because they somehow connect two different spaces: code-space <==> data-space

Personally I prefer the first approach for safety reasons. In my case, all the "software bridge" instructions that need to write something are privileged, hence only the kernel can use them to reprogram (e.g. the bootloader), while reading a constant is always allowed.

I could achieve the same goal with the other approach, but it would require extra hardware like a circuit that checks if an address belongs to the allowed range for a given operating mode.

E.g. don't allow a task in user-space to write things in the range 0xE000.0000..0xf000.000, because there it's mapped the code-space, and only the kernel can write there.

It's doable, just it costs more hardware and effort.

[Nominal Animal]

Yep.  A simplified summary about address spaces is that in C, both gcc and clang handle them just fine; and in C++, clang++ handles them just fine, but g++ does not.

And by "handle correctly", I mean the compiler will generate correct access instructions and place the variables and objects in the correct sections without having to use helper functions or macros for the accesses, as long as either the variable or the type includes the address space attribute.

It's not just older AVRs that have multiple address spaces, like DiTBho mentioned (about same in FPGA implementations); OpenCL and even x86-64 have them, too.
On x86-64, named address spaces are used to indicate specific segment register must be used; clang uses address_space(256) for GS, address_space(257) for FS, and address_space(258) for SS, and has clear rules on how these work through casts etc.  (gcc end uses __seg_fs and __seg_gs; g++ does not support them.)
The special segments are used for example for thread-specific data, as well as some very nifty kernel-userspace transparent data stuff.

Version 9 clang/LLVM language extensions lists these.  It looks like OpenCL drove the need for named address space support in clang; hopefully, GNU folks will change their mind and come along.  This is definitely useful.

[netmonk]

So yesterday i was reading gcc documentation after digging for named address space. And i was trying to focus on how to build a .bin.
Well i guess gnu doc repository is not the place for such information.

For exemple, when i was playing with my riscV gd32 i found this blog post https://vivonomicon.com/2020/02/11/bare-metal-risc-v-development-with-the-gd32vf103cb/ very interesting but sadly im not familiar with 99% of the content:
- linker script
- vector table
- boot

My thinking is "ok let's learn C++ freestanding" but it's like when i read the K&R long time ago, beside playing with language concept, it was difficult to apply to real programming (back in the time i was trying to code a demo on msdos) K&R is unable to teach you how to play with cpu/vga card/sound blaster card/ and so on.

I feel pretty the same now if i want to go the hard way. I can setup an esp-idf framework ready to go, but lots of work and pitfall are hidden. It's like with Arduino, it's very fast to code a temp sensor pushing data over wifi in few line of code. But when you want to play with a peripherial at full extent, you feel rapidly limited with such env.

So where is it possible to find the fundamentals of MCU development from how to start with an empty host os (which tool to install) an empty directory (which file are mandatory and for what) to produce a working firmware ?

[Nominal Animal]

I can setup an esp-idf framework ready to go, but lots of work and pitfall are hidden. It's like with Arduino, it's very fast to code a temp sensor pushing data over wifi in few line of code. But when you want to play with a peripherial at full extent, you feel rapidly limited with such env.

Very true.

That's why I've been mentioning the ATmega32u4 so often.  On one hand, it is simple and one can basically grasp all of it.  On the other hand, because it does have native USB support (12 Mbit/s), it can appear as whatever USB device you want, including USB HID devices like keyboards and joysticks and gamepads, as an USB Serial device, or even some native USB device (like slower USB audio devices – note the 12 Mbit/s limitation).

Starting from those (well, actually AT90USB1280 aka Teensy 2.0++, then 2.0, then to Teensy 3.0, a diversion back to ATmega32u4 via cheap Pro Micro clones using Arduino Leonardo bootloader, and so on), I could grow my understanding and capabilities step by step, without encountering a sheer vertical wall at any point.

You can even look at the intermediate files the Arduino environment generates, and explore the C runtime (crtmcu.o object files linked with Arduino code for each MCU), to see exactly how stuff gets implemented.

Most of the technical details I've mentioned in this thread are advanced stuff, in the hopes that you and others reading this thread will someday reach the point where they can make informed choices, and remember this thread and my points; and make stuff that Is Just Better than what we have now.
It is perfectly acceptable – and personally recommended by me – to start with the "easy" environments like Arduino, as getting stuff working gives that delicious boost of success and keeps ones motivation up; and while the environments are less than perfect, they are still quite usable, and very good for learning (as one quickly gets visible feedback, instead of just a laconic "Ok, that worked", and can play with stuff).  Just keep in mind that it is just a step along the ladder – optional, though; not everybody needs to create stuff for others, it is perfectly okay to just play with stuff for fun – towards better control and mastery of the subject domain.

So where is it possible to find the fundamentals of MCU development from how to start with an empty host os (which tool to install) an empty directory (which file are mandatory and for what) to produce a working firmware ?

I wish I had a real answer to that!

I think –– but I seriously hope others will pipe in, because I'm not at all sure about this –– that the best approach is to start in the Arduino or PlatformIO environment with preferably a simple microcontroller, and learn to write code in this funky environment (while being aware of what are quirks of a particular environment, what is hardware-specific, and what is C or C++), and when you have something working (even a blinky one), in parallel start to examine the build process and intermediate files part by part.  If the microcontroller is suitable for bare metal development, then the "advanced" step would be to recreate something you did in the Arduino/PlatformIO environment in "bare metal".

I do definitely recommend getting development experience in Arduino/PlatformIO first with different microcontrollers and hardware types.  Not only is it different, but you'll find you yourself will shift paradigms based on the task at hand.  Something like an USB arcade controller with joystick and buttons is nice to do in freestanding C, but a complex configurable robot firmware with different compile-time selectable modules can be much easier to write using the C++ subset.  It is not just the language, but also how you go about looking for a way to implement something.

The best example I can think of is how the different display module libraries work.  Some of them have a full frame buffer, others only have a partial buffer, and you need to repeat the drawing commands (they only capture/rasterize to the current buffer slice) until the entire buffer has been generated and sent to the device.  So, for small memory devices you prefer to deal with the drawing primitives, and for larger memory devices you can operate on the entire frame buffer and can track changes (so that unnecessary parts are not updated, and the overall updates are faster – especially useful with eInk/e-paper displays).  It does not matter what language you use, but for efficient implementation, your approach will differ based on whether you expect to have SRAM for a full framebuffer or only a part of one.

Even though I think Arduino could be better, I still think it is a very good environment to learn, and to do interesting stuff and gadgets in.
For example, I designed a Pro Micro Gamepad so that you could solder a Pro Micro clone on top of the (bottom board) upside-down, add an 128×32 (0.91") I2C OLED display, and program the entire thing in the Arduino environment (treating it as an Arduino Leonardo there, but using the Pro Micro pinout to see how the hardware pins are actually connected to the gamepads).
There is absolutely no need to do this in bare metal; the Arduino version will work just fine.
(The idea is that the gamepad is actually a gamepad + keyboard combination, and generates keypresses when the buttons are pressed, so that one can play Online games with it using standard keyboard controls.  The two smaller tactile buttons are intended to switch between mappings, with the OLED display showing the currently selected mapping for a few seconds after the most recent small tactile button press.  It's a laughably simple design, so I designed another based on CH551g, but haven't built that yet either.)

linker script

Linker script is a linker configuration file.  You can find these in various projects in a ldscripts directory.  The suffix determines the set of linker parameters used;

For both gcc and clang, you can specify a custom linker script using -Wl,-T -Wl,filename .

The default ones for avr-binutils are created by a shell script named avr.sc in the binutils sources.

You can find a quick synopsis at OSDev Wiki, and full details at the binutils ld scripts documentation.

This is what determines where code starts, where the fuse bits are, where data will be put, and so on.

vector table

Vector table is either an array of code addresses, or an array of jump instructions to code addresses.  When the microcontroller starts up, or a hardware interrupt occurs, it will use a specific entry in the vector table.

For ATmega32u4 (avr5), the vector table is in Flash, four bytes per entry (0x0c 0x94 0xLL 0xHH for a jump to address 0xHHLL), with the first (zeroth) one being the init/reset vector, that the MCU starts with when it powers on or resets.

boot

When a microcontroller starts up or resets, it is in a very hardware specific state.  To make firmware upgrading easy, the very first code it executes is a bootloader.  Essentially, its purpose is to decide whether the microcontroller should expect a new firmware to be uploaded, or just jump to currently existing firmware.

Arduino uses its own bootloaders, so do Teensies.  There is an USB standard, Device Firmware Upgrade or DFU, that microcontrollers with native USB interfaces can support for updating the device firmware via USB.  In many cases, the bootloader will not replace itself, only the rest of the Flash contents, and to replace the bootloader, one needs to use the in-system programming interface (JTAG, for example) appropriate for the hardware.  While the bootloader is running, it can use the entire SRAM as it wishes, since once it hands control over to the existing firmware, the firmware will be in full control.  So, the only limited resource bootloaders consume, really is just the Flash they take up.

Therefore, "booting" in this context means the phase in microcontroller startup and reset sequences, where the "bootloader" code checks whether a new firmware should be uploaded to that microcontroller, or whether it should just hand over the reigns to the currently installed firmware.

Interestingly, the proprietary bootloader for Teensies, Halfkay, uses the HID protocol (and therefore needs no special OS drivers!), and the 8-bit AVR implementation (for Teensy 2.0 and 2.0++) takes less than half a kilobyte of Flash.  This made early Teensies so much nicer to develop on than e.g. Arduinos.

[DiTBho]

I wish I had a real answer to that!

This also depends on the tool-chain. For example, a couple of commercial tool-chains I need to use are for Windows and they just need to click "setup.exe" and follow the video instructions.

examples:
Analog Devices DSP Visual Studio ++
CodeWarrior Studio

(I don't like Windows, but .... I have to say, the instructions you have to follow on Windows XP are the same you have to follow on Windows 10, and the development tool is exactly the same, that's a good point!)

In these examples, it's all included, and it also includes the support for the debugger, as well as templates and "get started" examples for the bootstrap code.

Linux and FreeBSD have a different approach with G-CC or Clang, and it's a rather "rolling" approach, if you read an "how to" article today, the given instructions may become out of date the next day due to the volatile nature of the tools involved and you need more specific knowledge and skills to manage them as well as the things required by "ecosystem".

So I think there is no simple answer, and only practical experience can tell and teach 

[westfw]

I wasn't aware that e.g. TinyAVR 0, 1, and 2 all have an unified address space!

Also the "mega0" line (ATmega4809) manages to map a full 48k of Flash space into the 64k unified address space, and the newer AVRmmDxpp can map 32k at a time (of their up-to-128k) into unified addresses via yet another bank switching scheme.  (I'm not sure that's good for C in general, beyond pretending that you have an aways-there 32k, but...)

["xtiny"] aren't fully compatible with traditional AVR, hence somewhat obscure. They are 8 bit, but peripherals have changed.

Yup, two years ago, I abandoned them for exactly this reason.

You should probably reconsider, IMO.  The changes in peripherals are "compatible with modern practices", the new features are useful, Microchip has dontinued the line enough that I'm pretty confident it's their vision of the future, and they have significantly more RAM/flash/perhipherals than the previous generations at a lower cost.  (Still no USB or 8pin/8kB chips, though.  The 32u4 and tiny85 must be real cash cows.)

im not familiar with 99% of the content:
- linker script
- vector table
- boot

Yes, there's quite a bit of effort that goes on in the process of putting "generic" output from a compiler into an address space (or two) that isn't flat, contiguous, isn't read/write everywhere, and has "special addresses" containing "magic stuff."  Fortunately, compiler writers have kept up pretty well (I can remember when "produces ROMable code" was a selling point for 8088 compilers), and the "usual cases" don't require manual intervention.  It's worth learning about, at least "some."   (it's vaguely similar to the shock that new assembly language programmers go through when they are faced with not having initialized RAM.)