EEVblog Electronics Community Forum

Products => Computers => Programming => Topic started by: peter-h on July 02, 2023, 01:27:39 pm

Title: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: peter-h on July 02, 2023, 01:27:39 pm

Context:
https://www.eevblog.com/forum/microcontrollers/a-question-on-gcc-ld-linker-script-syntax/msg4938223/#msg4938223 (https://www.eevblog.com/forum/microcontrollers/a-question-on-gcc-ld-linker-script-syntax/msg4938223/#msg4938223)

Some of the directives in the generic LD manual do not work in this version (supplied with ST Cube IDE).

This invocation is one of many formats I tried

Code: [Select]

/* This collects all other stuff, which gets loaded into FLASH */
   	.code_constants_etc :
  	{
  	
  	    *(EXCLUDE_FILE(*b_loader.o) .text .text* .rodata .rodata* .data .data* )
  	    	
    	. = ALIGN(4);
    	*(.text)           /* .text sections (code) */
    	*(.text*)          /* .text* sections (code) */
    	*(.rodata)         /* .rodata sections (constants, strings, etc.) */
    	*(.rodata*)        /* .rodata* sections (constants, strings, etc.) */
    	*(.glue_7)         /* glue arm to thumb code */
    	*(.glue_7t)        /* glue thumb to arm code */
		*(.eh_frame)

    	KEEP (*(.init))
    	KEEP (*(.fini))
    	
    	. = ALIGN(4);
    	_e_code_constants_etc = .;        /* define a global symbol at end of code */
	} >FLASH_APP

I've solved it by re-ordering stuff in the linkfile which based on the addresses in .map seems to have worked but there are side effects which I am trying to track down... spent hours on this so far.

The problem is that the above section is collecting everything (as it should) but I need it to not collect b_loader.o stuff.

Code: [Select]

.b_loader_all :	
  	{
    	. = ALIGN(4);
    	_loader_ram_start = .;
    	KEEP(*(.b_loader))
    	*b_loader.o (.text .text* .rodata .rodata* .data .data*)
     	. = ALIGN(4);
     	_loader_ram_end = .;
  	} >RAM AT>FLASH_BOOT
  	
  	_loader_flash_start = LOADADDR(.b_loader_all);

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: Nominal Animal on July 02, 2023, 02:35:18 pm

Quote from: peter-h on July 02, 2023, 01:27:39 pm

Code: [Select]

/* This collects all other stuff, which gets loaded into FLASH */
   	.code_constants_etc :
  	{
  	
  	    *(EXCLUDE_FILE(*b_loader.o) .text .text* .rodata .rodata* .data .data* )
    	. = ALIGN(4);

This includes .text*, .rodata*, and .data* from all files except *b_loader.o, and then aligns to a 32-byte boundary.  Then,

Quote from: peter-h on July 02, 2023, 01:27:39 pm

Code: [Select]

    	*(.text)           /* .text sections (code) */
    	*(.text*)          /* .text* sections (code) */
    	*(.rodata)         /* .rodata sections (constants, strings, etc.) */
    	*(.rodata*)        /* .rodata* sections (constants, strings, etc.) */

includes the same from the excluded *b_loader.o.

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: peter-h on July 02, 2023, 03:19:18 pm

Right; I get that, but I tried the exclude after the other stuff too.

As I've posted in the other thread, I've given up on this and got it working by a re-ordering of the various blocks. I understand the basic ideas of collecting symbols (beeing doing embedded with text, data, bss, etc for 40 years) but these subtleties are way above my head :)

And the documentation on LD is absolute total shit. The whole language is just circular. It's the worst ever, as evidenced by the vast number of posts online with questions on it.

/DISCARD/ should also be a way of doing it, but I found it dumps the specified stuff permanently; it cannot be picked up by a later section.

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: Nominal Animal on July 02, 2023, 03:56:19 pm

I fear you are brute-forcing something that would be much easier to construct analytically.

First step is realizing you need a paradigm switch.  LD scripts are not configuration files, they are actual scripts the linker runs to generate an ELF file, written in the linker command language.  While the syntax is funky, the underlying logic is quite straightforward, and very much reminds me of awk (https://www.gnu.org/software/gawk/manual/gawk.html#History), the record-and-field -based filtering and conversion tool and language.

Second step is realizing you always need a plan before writing a script.  You need to know what it does, or you'll end up adding features on top of features, making it into an unmaintainable spaghetti mess that has to be brute-forced to get to work as desired.

What I recommend, is using a spreadsheet (like LibreOffice Calc, but anything goes) to create the actual memory map you want to achieve.  I prefer to start with zero at the top, increasing addresses downwards.  In separate columns, put

• Memory area name, like RAM, FLASH, BOOTFLASH.
• Output section (with descriptive names), to order things within each memory region on the device.
• Linker-defined symbol names (to identify specific addresses)
• Address, if fixed
• Excluded (or included) file name(s)
• Input section(s)

With the complexity you have, you'll probably have a couple of hundred rows in that spreadsheet, but that's perfectly okay.  Just make sure you never make a change to the linker script that deviates from the spreadsheet; always edit the spreadsheet first.
A plain text file (in UTF-8 with Unicode Box Drawing (https://en.wikipedia.org/wiki/Box_Drawing) characters for prettification) has the benefit that when using source control like git, even the changes are human-readable.

Next, you read chapter 3. Linker Scripts (https://sourceware.org/binutils/docs/ld/Scripts.html) in the binutils LD manual.  (You can check which version you actually use by running the linker command with --version as the parameter; you should find the corresponding manual at https://sourceware.org/binutils/docs-M.N/ld/, where M is the major version (2), and N is the minor version (latest being 40, as of 2023-07-02).

Your linker script should probably start with the MEMORY (https://sourceware.org/binutils/docs/ld/MEMORY.html) command.  Its contents are determined by memory area name (1. column) and address (4. column) in the spreadsheet above.

It is common to next define any linker symbols that depend only on the memory areas, like the start and end of RAM, start and end of FLASH, and so on.  You can add and subtract addresses to calculate lengths.  You often use the ORIGIN(memory_area) (https://sourceware.org/binutils/docs/ld/Builtin-Functions.html#index-ORIGIN_0028memory_0029) and LENGTH(memory_area) (https://sourceware.org/binutils/docs/ld/Builtin-Functions.html#index-LENGTH_0028memory_0029) for defining these symbols.  Remember the syntax is
    symbol_name = expression ;
here.

Next, you should use the SECTIONS (https://sourceware.org/binutils/docs/ld/SECTIONS.html) command.  I'll expand on its contents below the dotted line.

Finally, you add the ENTRY(symbol) (https://sourceware.org/binutils/docs/ld/Entry-Point.html#index-ENTRY_0028symbol_0029) to set the ELF file entry address.  You can have it at either the root level, or within the SECTIONS command.  If the hardware has it hardwired, use that address.  It is mostly a nicety, but it might be useful in some cases – if not for anything else, then for us humans to see without having to consult the microcontroller programming manual.
 _ _ _ _ _

Sections:

This is the part that most reminds me of processing data with awk.  Each output section (https://sourceware.org/binutils/docs/ld/Output-Section-Description.html) command defines one output section, its contents, and which memory area it is placed in.  The contents are specified by one or more input section descriptions (https://sourceware.org/binutils/docs/ld/Input-Section.html).  You can also define symbols in each output section, but note that by default, the addresses (including .) is relative to the section, so you might wish to use ABSOLUTE(.) (https://sourceware.org/binutils/docs/ld/Builtin-Functions.html#index-ABSOLUTE_0028exp_0029) to obtain the non-relocatable absolute address, or one of the other built-in functions (https://sourceware.org/binutils/docs/ld/Builtin-Functions.html), to manipulate the symbol values.

You can also define overlays (https://sourceware.org/binutils/docs/ld/Overlay-Description.html).  You can use it for your code run in RAM during bootup before firmware/application is loaded to RAM, to ensure the linker uses correct addresses, or you can use position independent code.
 _ _ _ _ _

If you find yourself lost in the various terms, create your own crib sheet or example, with specific names so you can see the cross references.  For example,
   
    MEMORY {
        FLASH (rx): ORIGIN = 0x08000000, LENGTH = 1024K
        RAM  (rwx): ORIGIN = 0x20000000, LENGTH = 128K
        CCM  (rwx): ORIGIN = 0x10000000, LENGTH = 64K
    }
   
    SECTIONS {
        . = 0;                        /* No offset */
       
        .text: {
            KEEP(*(.startup))         /* Startup code */
            *(.text*)                 /* Program code */
            KEEP(*(.rodata*))         /* Read only data */
        } >FLASH
       
        .data: ALIGN(16) {
            *(.data*)                 /* Initialized data */
        } >RAM AT>FLASH
    }

Note how I put all names in italics, and used a different font for the comments?  These are the cues I use to remind myself; use whatever works for you.
For example, just looking at the .data output section definition, I remind myself that >region specifies the run-time region, but AT>sregion where the contents are stored, and that my bootup needs to copy from sregion to region for this to work.

Instead of making just one example to contain everything needed (like adding the symbols and expressions needed for the aforementioned copy), make separate examples or crib sheets, to reduce your cognitive load.  Perhaps label them according to their use.  You won't have that many of them, less than a dozen, anyway; linker scripts aren't that complicated, once you grok how they work.

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: Nominal Animal on July 02, 2023, 04:08:17 pm

Quote from: peter-h on July 02, 2023, 03:19:18 pm

Right; I get that, but I tried the exclude after the other stuff too.

Thing is, you only need the exclude.  For example (https://sourceware.org/binutils/docs/ld/Input-Section-Basics.html#index-EXCLUDE_005fFILE),
    EXCLUDE_FILE(*foo.o *bar.o) *(.text .rodata)
includes all .text and .rodata sections in all files excluding *foo.o and *bar.o.

The syntax is precise; for example,
    EXCLUDE_FILE(*foo.o *bar.o) .text .rodata
includes all .rodata sections in all files, and all .text sections in all files excluding *foo.o and *bar.o.

In other words,
    EXCLUDE_FILE(*foo.o *bar.o) *(.text .rodata)
and
    EXCLUDE_FILE(*foo.o *bar.o) .text EXCLUDE_FILE(*foo.o *bar.o) .rodata
are equivalent, including .text and .rodata sections from all files except for *foo.o and *bar.o.

In summary, the syntax is
    EXCLUDE_FILE(exclude-glob-pattern(s)) input-section
    EXCLUDE_FILE(exclude-glob-pattern(s)) include-glob-pattern:input-section
    EXCLUDE_FILE(exclude-glob-pattern(s)) include-glob-pattern:(input-section(s))
where input-section is one of
    section-name
    include-glob-pattern:section-name
    include-glob-pattern:(section-name(s))
where (s) indicates plural, i.e. one or more, whitespace-separated.
If include-glob-pattern is not specified, it defaults to * (all files).

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: peter-h on July 02, 2023, 06:31:20 pm

Thank you so much for that amazing explanation!

Some of that is just bizarre but I guess it has its uses. I just can't imagine getting back to it a year later and understanding it :) It explains why the various examples online didn't work; people posted them without explanation of the very subtle syntax.

My project has been as simple as I could get it:

1) boot block - 32k size - asm init code does the data copy, and bss zeroing (this was the original ex ST MX demo project), stack top of 128k RAM (even if 32F437 used which has 192k RAM)

2) boot loader* - 1.5k size - data is copied with the code; to somewhere high up in RAM, to an address after bss of 3), loader uses no bss, stack hard-coded in the CCM

3) main code - from base+32k to the rest of the 1MB (this can also be an overlay supplied by someone else (I have a special Cube kit for building that), data copy and bss zeroing is done by a "main stub" (with C)

* This was tricky to link because it is stored in the boot block and copied to RAM, so the code needs to be located to run from RAM, and for a long time the only way I could solve the linker complaints about section overlap was to lose that much from the usable RAM. I know one can also suppress the linker warnings. It isn't much RAM though. But then I had some problems and decided to clean up a lot of stuff, and placed the loader execution address after main code bss, and there it doesn't interfere (it is heap space which the linker doesn't know/care about).

Other constraints are that 2) can call functions in 1), but 1) and 3) are totally separated at the base+32k boundary (main code is entered with an asm jmp to base+32k).


Also the loader needs to be in RAM because I want to retain the option of updating the boot block too. Yes, there is a way whereby a 32F4 can program its own flash (it generates loads of wait states) but it can't program the addresses of the programming code, so one needs to switch to another bit of programming code then. But I never saw an example of that so probably it doesn't actually work. Another approach is to get the compiler to generate relocatable code but again I never found an example.

If it wasn't for the loader, it would be trivial :) But I need remote firmware upload, over http/https.

Back to basics, why do C compilers generate sensible error messages? Real men need just "syntax error" and possibly, at a stretch, a line number. They go down with the ship while saluting the queen. This LD thing was written for real men, and probably the reason for the totally crap usability is because most people set it up once and never touch it again. And avoid touching it at all costs.

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: DiTBho on July 02, 2023, 08:39:47 pm

Quote from: peter-h on July 02, 2023, 06:31:20 pm

This LD thing was written for real men, and probably the reason for the totally crap usability is because most people set it up once and never touch it again. And avoid touching it at all costs.

Compared to the my-c project linker script, which has been inspired by Avoget and Green Hills stuff, my personal opinion is that the GNU linker script is not very friendly because it is not well designed for the common user as it requires a lot of specific knowledge and competence, and I address the defect to the fact that designing things that look easy to be used requires more effort, which is not a GNU priority.

Title: Re: Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version)
Post by: Nominal Animal on July 02, 2023, 08:42:18 pm

Quote from: peter-h on July 02, 2023, 06:31:20 pm

Some of that is just bizarre but I guess it has its uses.

Domain-specific languages with a tiny set of users do tend to be, yeah.

Quote from: peter-h on July 02, 2023, 06:31:20 pm

My project has been as simple as I could get it:

I can't parse your description well enough to arrive to an actual memory map.  You need that spreadsheet or a diagram!

For example, starting with just the memory regions and output sections, ignoring what files define what, fill up the table of where things should end up at:

Memory region	Output section	Overlay	Linker symbol	Address	Description
FLASH			__flash_start	0x08000000
	.boot_init				Executable in place, VMA=LMA
	.boot_loader_src				LMA, VMA=.boot_loader
	.fastrun_src				LMA, VMA=.fastrun, for RAM code
	.text				Executable in place, VMA=LMA
RAM			__ram_start	0x02000000
	.fastrun	RAM1			LMA, VMA=.fastrun_src, copied by foo() in .boot_init
	.boot_loader	RAM1			VMA, LMA=.boot_loader_src, copied by bar() in .boot_init
CCM			__ccm_start	0x01000000

Ignore/clear the Overlay column first.  When you have every output section you need, you combine RAM and CCM output sections that can occupy the same address ranges since they never exist at the same time, by giving them a shared (but unique) overlay name.  Above, I used RAM1 for .fastrun and .boot_loader, since you mentioned it has to be copied to RAM, but is not accessible to .text or .fastrun.  For non-overlapped sections, just leave it empty.

Then, and only then, will you add the column for input sections, and two columns for files: one used for when only specific files' sections are included, and the other when all except specific files' sections are included.  When all files are included, both are empty.  When only specific files' sections are included, only the first file column is used.  When all but specific files' sections are included, only the second file column is used.  I'd call the input section column "Input section", and the two file columns "in only files" and "in all files except", I think: it has to be crystal clear, we don't want some notation we forget in a month.

With such a table or spreadsheet, constructing a linker script that does exactly what is wanted, is quite straightforward.
If you fill the above, I'm pretty sure we can construct one that does exactly what the table or spreadsheet specifies.

It is absolutely critical to keep the table or spreadsheet the master, and the linker script as the slave, because it can often be less work to rewrite the linker script based on the table or spreadsheet, than modify the existing script.  However, with both, you can also re-learn what you have forgotten, by comparing the existing table or spreadsheet to the linker script itself, and check how each part has been configured.

(While I believe it would be possible to write a Python script to parse CSV-formatted table with fixed-name fields into a linker script, it would go too far into code generation land for my taste.  I do mention it, because the table or spreadsheet must contain all information, and therefore a translation is possible.  For example, you might wish to add a column that states the required alignment for the output section or linker-defined symbol.  And you probably want the linker to calculate the sizes for you (since it can), so you don't have to do the calculations at runtime, saving a few instructions here and there; I'd list these in the spreadsheet, before the actual table.)

Quote from: peter-h on July 02, 2023, 06:31:20 pm

Back to basics, why do C compilers generate sensible error messages? Real men need just "syntax error" and possibly, at a stretch, a line number. They go down with the ship while saluting the queen. This LD thing was written for real men, and probably the reason for the totally crap usability is because most people set it up once and never touch it again. And avoid touching it at all costs.

It's not that bad.

Many of those porting a build toolchain to new hardware (i.e., most of those who ever write linker scripts in the first place) tend to have experience with weird domain-specific languages and tools, so they don't actually mind much.  I've seen worse.  The procedure I outlined earlier, and the experimentation with a trivial example source tree that defines at least one symbol for each section, is what most such people tend to end up with (because with weird tools, it is efficient way to truly find out).  If they then look at ELF file capabilities, and start working on a better linker script command language, they discover they need the same features currently provided, with just a better syntax.  That set of people, being used to all sorts of syntaxes, just shrug and deal with it, because replacement is just not worth the effort to any of them.

Is it a crappy language?  I don't think so.  The documentation is horrible, because it does not use terms consistently.  When it talks about addresses, they can be load addresses (addresses in the image), virtual memory addresses (where the code or data will reside at runtime), or relative to the beginning of the current output section.  Even the exact grammar isn't specified; that alone in aBNF (https://en.wikipedia.org/wiki/Augmented_Backus–Naur_form) format would help a lot, because that way the sub-command sequences like the way input sections can be listed, would be clear (and is just sequential stuff; just like "a" varname "b" in Awk constructs a string from the contents of variable varname, prefixed with a and suffixed with b).
And the index doesn't even mention that linker script command language expressions can use the ternary operator, i.e. ((expression) ? if-true-or-nonzero : if-false-or-zero), it's only mentioned in Operators list (https://sourceware.org/binutils/docs/ld/Operators.html) and in the description of the DEFINED(symbol) built-in (https://sourceware.org/binutils/docs/ld/Builtin-Functions.html#index-DEFINED_0028symbol_0029)!

Quote from: DiTBho on July 02, 2023, 08:39:47 pm

my personal opinion is that the GNU linker script is not very friendly because it is not well designed for the common user as it requires a lot of specific knowledge and competence, and I address the defect to the fact that designing things that look easy to be used requires more effort, which is not a GNU priority.

Yes, but it's not just GNU, it is in all free/open source software, because designing things for others to use is much harder than designing things for oneself to use.  In general, user-friendly design takes a lot of effort.

Just compare how far KiCAD user interface has come in the last couple of major versions!  I like version 7 a lot.