how to include a binary with C

[DiTBho]

How to include a binary into a C project? 

There are several ways ...

Today I had to include 8x16 fonts and 32x32 icons in an application that runs on ROM, and I thought it would be nice to post here how I did it

Code: [Select]

        .globl  suppa
        .size   suppa, 4
        .globl  suppa_size
        .size   suppa_size, 4

        .align 1
suppa:
        .byte  0xde,0xad,0xbe,0xaf
        .ascii "hAllo World!"
suppa_end:
        .align 4
suppa_size:
        .long  (suppa_end - suppa)
(assembly file, asm/data.s)

Code: [Select]

/*
 * ... "public" is the same as "extern" in c89
 */

public uint8_t  suppa[];
public uint32_t suppa_size;

    my_size = sizeof(suppa);

    for (i0 = 0; i0 < my_size; i0++)
    {
        ... suppa[i0]
    }

    for (i0 = 0; i0 < suppa_size; i0++) /* that's how it works too, shouldn't it? */
    {
        ... suppa[i0]
    }
(c source, distilled example)

Code: [Select]

all: data ...
        gcc  obj/data.o ... obj/app.o -o app

data: asm/data.s
        as asm/data.s -o obj/data.o
(Makefile, "as" is "GNU/AS")


To fill the asm/data.s file I used a tool that read the wanted binary file and outputs hex - byte per byte -

[bingo600]

I'd prob. just have read the binary , converted every byte into a 0x??, notation , and slapped a bin_data[] = {......} around the output.

/Bingo

[magic]

To fill the asm/data.s file I used a tool that read the wanted binary file and outputs hex - byte per byte -

You aren't following the coolest thread on this subforum, are you?
https://www.eevblog.com/forum/programming/a-new-hardware-oriented-programming-language/msg4637599/#msg4637599

[DC1MC]

There are a lot of quick and friendly methods of linking binary files into a C program, I strongly recommend having a look here:
http://elm-chan.org/junk/32bit/binclude.html

[Siwastaja]

Yeah, I used this method: https://www.eevblog.com/forum/programming/a-new-hardware-oriented-programming-language/msg4637629/#msg4637629 the very next day. A real timesaver compared to generating a .c file with an array, although it's not that much work. But I think the linking idea is more elegant, as well.

[langwadt]

I'd prob. just have read the binary , converted every byte into a 0x??, notation , and slapped a bin_data[] = {......} around the output.

/Bingo

xxd -i  will do that for you

or use .incbin or the linker file to directly add the binary

[DiTBho]

I'd prob. just have read the binary , converted every byte into a 0x??, notation , and slapped a bin_data[] = {......} around the output.

Yup, sometimes it's even better to entirely stay with C files.

I like this other way because I can reuse some bin-file.s with assembly routines for hc11 without the need to look at other .C files, which would then have to be converted to assembly (uint8_t [..] into ".byte" field ....); I mean no problem, it's even script-able (python, lua ...), but it's just more work.

Crazy that I will recycle the same fonts with hc11 (isn't that too slow with a graphic LCD?  ) ... but that's another story

[DiTBho]

xxd -i  will do that for you

Code: [Select]

overlay:dev-util/xxd


Code: [Select]

# echo "hAllo world" > hAllo.txt
# xxd -i hAllo.txt
unsigned char hAllo_txt[] =
{
  0x68, 0x41, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x0a
};
unsigned int hAllo_txt_len = 12;

nice tricks 

[DiTBho]

Code: [Select]

app-editors/vim-core-8.2.0360

updating vim-core (with USE=!minimal), adds /usr/bin/xxd
very nice 

[T3sl4co1l]

I'm fond of the header format, too.

Rundown of options:
https://tratt.net/laurie/blog/2022/whats_the_most_portable_way_to_include_binary_blobs_in_an_executable.html
Binary can be linked directly as an object, it just needs to be named and tagged appropriately.  Although the commands shown here seem to be missing some things... not sure how the linker is supposed to know what to do with the object without a name, or can it default to its filename as the symbol?..  Anyway, you probably know more about linkers than I do, heh.

There's also an #embed (finally!):
https://thephd.dev/finally-embed-in-c23
I'd suggest updating your toolchain, but, uh... with the artisinal bespoke toolchain you have, I guess you can implement that at your convenience, or stick with the old ways?

Tim

[SiliconWizard]

Didn't we go exactly over that a few days ago?

[Sherlock Holmes]

Didn't we go exactly over that a few days ago?

Yes we did, where it was discussed how convenient it would be if we could initialize static entities directly from a raw file using some new compiler directive, some compile time executable operation.

[DiTBho]

Didn't we go exactly over that a few days ago?

nobody mentioned the assembly method 

and the guys here mentioned some pretty new ways, indeed I just found i needed to update my vim-core!!! and prepare a new overlay!!!

(thanks for that comment!)

[SiliconWizard]

As I said in the other thread, to me, generating object files from binary files is the simplest, most straightforward way of doing it, and can be accessed from pretty much any language.
Doing that is dead-easy if you have binutils, but even if you don't, as long as you know the object file format for a given environment, writing the conversion tool is an easy task.
Just my 2 cents.

[brucehoult]

It's pretty easy to convert a binary file to either an ASM file with .byte etc declarations, or a C .inc file with char or int etc values.

An advantage of a text format is that is may make life easier if you want to put it into a source code control system such as git.

Contrary to some opinion out there, git itself works just fine with binary files, and only storing deltas for them, etc. But many tools around git, such as "git diff" or the github web interface etc.

[Sherlock Holmes]

As I said in the other thread, to me, generating object files from binary files is the simplest, most straightforward way of doing it, and can be accessed from pretty much any language.
Doing that is dead-easy if you have binutils, but even if you don't, as long as you know the object file format for a given environment, writing the conversion tool is an easy task.
Just my 2 cents.

How can that be simpler though? you need the compiler for the source code and another tool for data file, that's two distinct tools, or one must write a conversion tool. Surely being able to do that programmatically from within the source code is even easier? There are likely umpteen ways one can express the idea but having the compiler do it all for you has to be better - IMHO.

[DiTBho]

@SiliconWizard
Oh, well ... I have to do something similar with the DDE of my RISCOS v4.39 adjust 

There is no binutils(1) for RISCOS, there is no g-as, so ... you really like to only have { %.s , %.c , %.h } files.

DDE, what? It's the native development toolchain&ide of RISCOS. It's a commercial product and it's very powerful (you can even recompile some RISCOS modules with that) but it can appear limited if you try to compare with what you use for GNU/LINUX or *-BSD.

Why limited? Well, ... on DDE you have a C compiler(3) and an assembly (ARM-classic) compiler, and a linker. Plus other tools, which are for GUI and other specific RISCOS stuff. EndOfList. So, lot of tricks you use on GNU/Linux, *-BSD (including vim-core(2)) are not ... practical. Especially objdump and similar, and playing with native linker script file is not as flexible as you can do with GNU-gd.

So, you really like to have something you can select with DDE as "project files", and then click on "build it".

(1) actually, you can cross-compile { "make", "bash", "coreutils", "binutils", "gcc" } for RISCOS classic using a lib-unix compatibility layer on a special sandbox that handles files(5).

I did it, as well as I cross-compiled cross-compilers for RISCOS, and somehow - don't ask my how - that Frankenstein-stuff works, but it's very very slow and inconvenient, also due to the different filesystem.

In fact, on RISC files can't have an extension, so... if you have "hallo.c" you need to create a folder "c" and put "hallo" in that folder; thanks god, there is a (commercial) virtual filesystem module that you can invoke on the fly to adapt the RISCOS filename to the UNIX file name convention.

e.g.
unix app needs to operate with "hello.c" --> file-adapter  --> native RISCOS filesystem operates with "c/hello"
unix app needs to operate with "hello.c" --> file-adapter  --> native RISCOS filesystem operates with "h/hello"

Crazy? Insanely crazy if you consider that I paid 50 UKP for that (why?!?  "because science" ).

I think, "money well spent" if you consider that at least it also works over the network (4); just imagine how many things you have to change to get those UNIX-things to work.

That's why you don't want to use binutils tricks on RISCOS 

edit:
(2) vim, nano, joe ... this stuff needs an extra level of compatibility because there are no tty on RISCOS.
(3) there is also a C++ compiler, but it doesn't work, forget it!
(4) exported as NFS-v2, you no longer have to move files with a iomega ZIP/SCSI 50Mbyte cartridge.
(5) especially Bash needs to live into a sandbox in order to mimic stdin/out/err and redirection.

[DiTBho]

An advantage of a test format is that is may make life easier if you want to put it into a source code control system such as git.

yup, to make life easier is *THE* reason we do that with Doors at work 
Doors is a commercial tool, I think the same concept applies to Git/SVN/...

[Sherlock Holmes]

@SiliconWizard
Oh, well ... I have to do something similar with the DDE of my RISCOS v4.39 adjust 

There is no binutils(1) for RISCOS, there is no g-as, so ... you really like to only have { %.s , %.c , %.h } files.

So consider writing a tool that can consume an arbitrary file, with some additional arguments and do as some have mentioned here, generate a C initialization source?

One could tell the tool that you're creating an array initializer for some kind of type:

Code: [Select]

convert_to_c <somepath> "int[4][3]" "arr" <Enter>
[/quote]

to get an output file that contains:

[code]
/* This text was generated by a tool on <date> at <time> */
/* do not modify this text.                              */

int arr[4][3] =
{
 { 2, 3, 1 },
 { 19, 12, 7 },
 { 10, 9, 8 },
 { 3, 11, 5 }
};

As some say out in these parts "Go get her writ!".

Or just

https://gist.github.com/albertz/1551304

https://github.com/TheLivingOne/bin2array/blob/master/bin2array.c

[Whales]

There's also an #embed (finally!):
https://thephd.dev/finally-embed-in-c23

Took me ages to find this link (was it #incbin, #binary or something?) only to discover you beat me to it. 

It's an interesting read.

[Sherlock Holmes]

There's also an #embed (finally!):
https://thephd.dev/finally-embed-in-c23

Took me ages to find this link (was it #incbin, #binary or something?) only to discover you beat me to it. 

It's an interesting read.

Well said, I agree with this myself:

You would think that we would have figured out a decent, cross-platform way to put data in executables after 60 years of filesystem research and advances and 40 years of C and C++ implementations.

Sound just like something I would write! anyway...

So much energy went into convincing people that yes, we should have this, no, we can’t “just parse better” our way out of it, no, the smartest people in the last 30 years across all major compilers were literally not capable of doing a better job at this! I actually wondered for a brief moment if I was not a proper user of C and C++ but an alien who existed in a planet that got the cursed or dark version of C and C++ and my interactions with other human beings were actually a brief window into an alternative universe where Everything Is Just Fine, Actually.

Buddy, I know how you feel, changing things is so much easier than changing attitudes.

Yes a very, very interesting read.

[DiTBho]

So consider writing a tool that

that works like xxd (now part of vim-core tools), just generating an assembly file like the one in my first post as further option.

xxd is already a great tool, with that option it would be more than great!

Why? Well, here ... I have to say, because with an assembly file you can easily tell the linker in which section you want it to put stuff.

.rodata?
.debug?
.custom?

no problem! no need to do weird stuff in C, no need to pass weird flags.
and Git will like it too!!!

It's what I am writing right now

[Sherlock Holmes]

So consider writing a tool that

that works like xxd (now part of vim-core tools), just generating an assembly file like the one in my first post as further option.

xxd is already a great tool, with that option it would be more than great!

Why? Well, here ... I have to say, because with an assembly file you can easily tell the linker in which section you want it to put stuff.

.rodata?
.debug?
.custom?

no problem! no need to do weird stuff in C, no need to pass weird flags.
and Git will like it too!!!

It's what I am writing right now

All I can say is that there's really nothing weird about using the same language to do both the code and the import of static data held in external files. Compilers read files for a living!

Read the article about #embed above.

[retiredfeline]

One tool I didn't see mentioned above, or in a link I didn't follow is https://srecord.sourceforge.net/. I use it for other purposes, but one of its capabilities is a C array declaration as an output format. The man page srec_cat details the relevant options.

[SiliconWizard]

@SiliconWizard
Oh, well ... I have to do something similar with the DDE of my RISCOS v4.39 adjust 

There is no binutils(1) for RISCOS, there is no g-as, so ... you really like to only have { %.s , %.c , %.h } files.

There must be a linker that can handle object files? If so just figure out the object file format and generate object files.
(Sure you can generate C files instead, your pick, but I like the option of generating object files directly - certainly more efficient for storing large binary blobs. And I'd be pretty intrigued if the object file format for your toolchain on RISCOS was not documented.)