Low-footprint printf replacement for embedded dev

[brucehoult]

I like building the strings in reverse when severely constrained.

Yes, that's an option of course, although reversing the string at the end of the conversion only takes a loop with half its length iterations and one byte of temporary storage (which should fit in a register unless you use a very, very small/odd target with only a handful of registers like, say, a Z80... or an 8-bit PIC!)

The smallest PICs have 32 registers. And no RAM.

Z80 isn't too hard, since BC, DE, and HL can all be used to load/store A, and can all do 16 bit inc/dec.

Code: [Select]

    # BC = count, DE = loPtr, HL = hiPtr
loop:
    ld a,(de)  #7
    ld b,(hl)  #7
    ld (hl),a  #7
    ld a,b     #4
    ld (de),a  #7
    inc de     #6
    dec hl     #6
    dec bc     #6
    ld a,b     #4
    or c       #4
    jp nz,loop #10

So 13 bytes of code, 68 cycles per byte pair. You could make it a little bit quicker if count is 256 or less (so actually 512 byte string) as the dec bc;ld;or can simply be replaced by a 4 cycle dec b. So 58 cycles per byte pair.

Or course that's much worse than a modern ISA e.g.

Code: [Select]

    # a0 = loPtr, a1 = hiPtr
loop:
    lb a2,(a0)
    addi a0,a0,1
    lb a3,(a1)
    addi a1,a1,-1
    sb a2,-1(a1)
    sb a3,-1(a0)
    blt a0,a1,loop

16 bytes of code, quite likely 4 cycles per byte pair on a dual-issue in-order core.

Z80 needs a lot of instructions, but does have the benefit that most instructions are only 1 byte.

[westfw]

The smallest PICs have 32 registers. And no RAM.

That's misleading.
The smallest PICs have an accumulator, and 25 data storage locations that behave somewhere between "registers" and "RAM" (they call it "Data Memory.") (and then an additional 7 peripheral registers, plus some additional memory ("call stack", Data direction registers) that is not directly accessible by the normal data instructions.)

In general, IMO, the 8bit PIC data memory behaves more like RAM than "registers", with most of the instructions operating between a data memory location and the accumulator (some instructions operate directly on the data memory (increment, decrement, bitwise operators), which I guess makes them somewhat "register-like", but pre-formalized-RISC there were a lot of architectures that would do some operations directly on memory.)

I think you have to go to the 16bit PICs before you get "real RAM", but the PIC16 and PIC18 can have up to 4k "registers" (in banks.  With weird addressing.)


---
I'm not sure I understand how building your strings backwards saves much code space.  The string reversal you'd normally get formatting integers isn't very much code...

[brucehoult]

The smallest PICs have 32 registers. And no RAM.

That's misleading.
The smallest PICs have an accumulator, and 25 data storage locations that behave somewhere between "registers" and "RAM" (they call it "Data Memory.") (and then an additional 7 peripheral registers, plus some additional memory ("call stack", Data direction registers) that is not directly accessible by the normal data instructions.)

I don't think it's at all misleading, especially when comparing against the Z80/8080.

W on the PIC is just like A on the Z80.  You can move things there, all 8 bit arithmetic is between another register and A/W. PIC is slightly more powerful in that you can choose between e.g. W <- W+r7 or r7 <- W+r7 using an extra bit in the instruction, while Z80 always puts the result in A.

Z80 has a 3 bit field in the instruction to choose which register to move/add/or/and etc to A, letting you choose any of A,B,C,D,E,H,L or (HL) (which 8080 calls "M") as the other operand source.

The smallest PICs have a 5 bit field which lets you choose any of the 32 registers to move to/from or do arithmetic with W.

It's very much the same thing.

Yes, only some of the PIC registers (25 you say, I haven't counted) are general purpose and the others (7 as you say) have special purposes. But they're all registers, and you can move stuff between them and W. One of the registers gets ALU result flags ... zero, negative, carry etc (though you can conditionally skip based on any bit of any of the 32 registers, not just the flags register). There's a FSR register to hold a RAM address, if you've got RAM other than the 32 registers), and an INDF register that then serves as an alias for the selected RAM address (8051 has similar special registers). And yeah some of the registers are GPIO ports.

FSR does allow you to point to any of the 32 registers, as well as to RAM (up to 224 bytes, depending which chip ... some have 0), which goes a bit against modern ISAs. But AVRs are the same (except the very newest I think), with the lowest 32 RAM addresses aliasing the 32 registers.

That's just the kind of thing people used to do in low end ISAs that were never intended to get superscalar OoO implementations.

You're not going to tell me that AVR doesn't really have registers are you?

[brucehoult]

I'm not sure I understand how building your strings backwards saves much code space.  The string reversal you'd normally get formatting integers isn't very much code...

Also it's trivial to use the stack to temporarily store the least significant digits, with little code size or stack usage (with or without return addresses interspersed).

[Nominal Animal]

I'm not sure I understand how building your strings backwards saves much code space.  The string reversal you'd normally get formatting integers isn't very much code...

No, but it all does add up.

All I can say is try it for yourself for your favourite architectures, and see if it makes a big enough difference for you.

The interface I suggest you use defines the buffer as for example
    char         buf[SIZE];   // SIZE being 127 or less
    int_fast8_t  len = sizeof buf;
with conversion functions returning the updated len:
    int_fast8_t  prepend_TYPE(int_fast8_t len, char buf[len], TYPE value);
with each function immediately returning len if it is already negative.

After all conversions are done, one need only check if len is negative.  If it is, you ran out of buffer space.  If it is zero or positive, it is also the offset to the beginning of the string, with ((sizeof buf) - len) characters in the buffer.

One case where you don't want to do this, is when you have an interface where the printf() style function can directly emit the data to a peripheral device, blocking until it has been completely sent.  Then, you only need enough buffer space for the largest dynamically constructed field.

The reason this yields smaller code is the compactness of typical accesses using this approach.  len is both the index to the start of the string, but also the number of characters available in the buffer.  It is very cheap to check if len is negative; much more code is needed to check if it exceeds a separately passed limit or constant.  If you have hardware division or modulo operation, then conversion of integers in any base is trivial.  (On AVR and similar, I do use repeated subtraction, with an array of ones in each decimal digit position, as that tends to yield very efficient code, much moreso than the alternatives.)

[peter-h]

Surely the bottom line is that any "small" printf will not be standard compliant.

So what you are really getting is something that parses some subset of the format specifiers (%u etc) and outputs the data, and again with limitations. So you are probably getting only a bit more than a DIY job done with itoa, ltoa, ftoa, etc, and since almost nobody needs scientific notation output, the foregoing will do most jobs where you really want it tiny.

A standard compliant printf is pretty big. Look at the newlib sources (plenty of threads on newlib printf). The code to handle all the weird stuff is probably 90% of the total size. Some reading around here
https://www.eevblog.com/forum/microcontrollers/is-st-cube-ide-a-piece-of-buggy-crap/msg4577884/#msg4577884

I would also suggest that someone doing a very compact project is not really after a full printf implementation, by the very nature of what he's doing.

[rstofer]

I just rip off the conversion functions from "The C Programming Language" - Kernighan & Ritchie and add some hex output functions for nibble, byte and word widths.

Step 1 - get the UART working
Step 2 - include string output function.  Print "Hello World\n"
Step 3 - include conversion functions and hex output functions
.. now start work on project.

No library functions are used and especially no heap use.

[AVI-crak]

I would also suggest that someone doing a very compact project is not really after a full printf implementation, by the very nature of what he's doing.

Reasonable idea -> cut as much as possible, but in a way that works.
I cut formatting completely. More than 93% of the source code was cut, the remaining 7% was rewritten differently.
Don't try to understand how it works.

[Siwastaja]

Surely the bottom line is that any "small" printf will not be standard compliant.

Yeah, that's an interesting balance. You can cut quite many features out of printf() and 99% of developers still feel it's like the real thing, giving them the advantage of familiarity (which really is the only reason to include printf implementation at all). For example, writeback and exponential (scientific) format are rarely used. (But I realize when I say that, soon someone replies that they use these features all the time. Fair enough.)

But if you cut a tad too much, then it's too far from standard printf() and the familiarity is gone. It is then worse: developers waste time debugging the printf and wondering "why it doesn't work"*. Any custom solution with totally different interface would be easier, then.

*) and if they are competent and interested enough to immediately understand what is going on, evaluate different printf implementations, configure them and so on, I wonder why they are not rolling their own, better logging system which suits exactly to their needs and is likely much more efficient?

[AVI-crak]

There are many printf() implementations, all contain a format field, all without exception. It is impossible to defeat the legacy, 99.99% of the code has classic printf(). Whether you want to or not, the library will be attached to the project.
When old solutions stop working, topics like this one appear.

[westfw]

We had a custom printf() that originally supported stuff like “%i” for an Internet address, and I think eventually let code dynamically add additional format specifiers.  Fun stuff.
(“Small” wasn’t so much a goal, although it might have turned out that way.  Had some “issues” when gcc started checking arguments against the format string…)

[Nominal Animal]

When you drop printf compatibility requirements, you can consider some really funky alternatives.  As I mentioned, in very constrained situations I create strings bass-ackwards, using separate function calls for each part or field.

In network services, it is common to have a relatively slow pipe, and possibly many concurrent clients.  Instead of reserving memory for each client, one can create an array or linked list of structures formatting the response, and only reserve enough memory for describing dynamic state per client.  The idea is that given a state and a read-only formatting structure chain, a function computes the next 1 to N bytes of the response.  The function will always compute at least 1 byte, unless it is complete, and is called whenever the client output pipe has room for writing (and it is time to output data to that client).  Dynamic fields are computed on the fly if possible, otherwise at most one field in advance.  In essence, this is an async printf.

The downside of this approach is that it too easily balloons into a templating "language".  The main upside is that it allows very long and very complex outputs with relatively little RAM use (a little bit more than the longest dynamically constructed field needs).

[brucehoult]

When you drop printf compatibility requirements, you can consider some really funky alternatives.  As I mentioned, in very constrained situations I create strings bass-ackwards, using separate function calls for each part or field.

People just seem to like the convenience of writing a single literal string with places to insert other data marked out. A ton of other languages have copied that basic idea from C, but not included the formats in the string, just the locations e.g. with "%=" or "{3} {1} {2}" or something to indicate where to insert data, and the toString() or whatever function is automatically called on each argument.

If you don't mind giving that up then the C++ overloaded operator<< convention is one of the lowest visual and typing overhead ways to do it. And has the big benefit (as with any function-call-per-object aproach) of making it easy to not link in unneeded functionality.

[Siwastaja]

If you don't mind giving that up then the C++ overloaded operator<< convention is one of the lowest visual and typing overhead ways to do it. And has the big benefit (as with any function-call-per-object aproach) of making it easy to not link in unneeded functionality.

"Many" people are fine with the C++ way. In C operator overloading is not available, but nothing prevents you from using macros for syntactic sugar, or put multiple statements on a single line. I have been doing stuff like this for years:

Code: [Select]

print("Value of x is "); print_i32(x); print("\n");

It's not that different from:

Code: [Select]

std::cout << "Value of x is " << x << std::endl;

Both are more tedious to write in some specific cases with many values and separators mixed together where printf-type format strings shine. But how much printing do we actually have in microcontroller projects, does it matter at all if those few lines of code that print "a lot" require 30 more keystrokes?

Meanwhile, I think I have saved time and typing by a PR_VAR(x) macro which #stringifies the variable name and then prints " = " and value and linefeed. Easy to sprinkle around in code, less typing than printf("you_need_to_type_this_twice = %d\n", you_need_to_type_this_twice);

[brucehoult]

"Many" people are fine with the C++ way. In C operator overloading is not available, but nothing prevents you from using macros for syntactic sugar, or put multiple statements on a single line.

Nothing prevents you using the C++ compiler instead of the C compiler.

The Arduino library is C++ -- or at least requires the C++ compiler. Some would argue it doesn't use much of C++. But it does use it.

No doubt there is some overhead, but I've happily used Arduino C++ code to write programs for microcontrollers with 512 bytes of RAM (ATTiny85) and didn't have any RAM or flash size issues with the kind of simple things you do with an 8 pin package.

The ATTiny25 -- 128 bytes RAM, 2k flash -- is probably too small for C++. I haven't tried. I do know the Arduino IDE doesn't offer the '25 as a target option, but does offer the '45 (256 bytes RAM, 4k flash).

[peter-h]

We had a custom printf() that originally supported stuff like “%i” for an Internet address,

Presumably IPV4 (uint32_t) not IPV6 (some struct thing)

I like that - very useful!

[westfw]

Presumably IPV4 (uint32_t)

Yep.  Also "%e" for ethernet addresses.
Compared the C++ way, I find printf formats to be a nice compromise in being able to visualize what the output will actually look like, not quite so explicit as fortran FORMAT or COBOL PICTUREs (?), but much better than individual function calls.

[Nominal Animal]

If you have separate print functions for each type you use, then in C11 and later one can use preprocessor macros and _Generic to implement a variadic print() macro that expands to individual function calls; i.e.
    print("x = ", x, ", y = ", y, "\n");
expanding to say
    print_cs("x = ");
    print_int(x);
    print_cs(", y = ");
    print_float(y);
    print_cs("\n");
I personally do not bother, because the multi-function sequence is just as readable to me as the single-function one.

For embedded uses, I like to use an ELF section containing a description of the variables, i.e.

Code: [Select]

struct var_desc {
    void *const ref;
    const char *const name;
    int_fast8_t (*const get)(uint_fast8_t maxlen, char buf[maxlen], void *ref);
    int_fast8_t (*const set)(uint_fast8_t maxlen, char buf[maxlen], void *ref);
};

// Getters convert the referred to variable to a string into buf
extern int_fast8_t  vardesc_get_int(uint_fast8_t maxlen, char buf[maxlen], void *ref);
extern int_fast8_t  vardesc_get_float(uint_fast8_t maxlen, char buf[maxlen], void *ref);

// Setters convert the string to the referred to variable
extern int_fast8_t  vardesc_set_int(uint_fast8_t maxlen, char buf[maxlen], void *ref);
extern int_fast8_t  vardesc_set_float(uint_fast8_t maxlen, char buf[maxlen], void *ref);

#define  MERGE4_(a,b,c,d)   a ## b ## c ## d
#define  MERGE4(a,b,c,d)    MERGE4_(a,b,c,d)
#define  VAR_DESC_NAME(prefix)   MERGE4(__, prefix, _, __LINE__)
#define  DECLARE_VAR(_var, _name, _getter, _setter) \
    static const struct var_desc  VAR_DESC_NAME(vardesc) \
    __attribute__((section ("vardesc"), used)) = { \
        .ref = &(_var), \
        .name = _name, \
        .get = _getter, \
        .set = _setter, \
    }
// Note: We could use _Generic for autoselection of .get and .set functions (based on _var),
//       and stringify variable name.  Sometimes, the name will differ, for example with a subsystem prefix.

extern const struct var_desc  __start_vardesc[];
extern const struct var_desc  __stop_vardesc[];
#define  __num_vardesc  (uintptr_t)(__stop_vardesc - __start_vardesc)
so that in the final linked binary, the vardesc section will collect all such declarations from all object files into a single contiguous array.  A simple text interface can then query and set any of these variables.  The above assumes that the linker script exports __start_vardesc at the beginning of the section, and __stop_vardesc at the address just past the section, as is usual; and this section can/should reside in Flash.

For example, you might have
    static int  step_count;
    DECLARE_VAR(step_count, "step_count", vardesc_get_int, vardesc_set_int);
either in a global scope, or in a function scope.  Note that the variable itself does not have global linkage, nor does the __vardesc_N static structure the declaration generates.

If one omits the setter, and adds say a debug level value for each, then this can easily be used for state dumping.  You might also use a char array instead of a pointer for the .name member so that the strings will also be included in the same section.

(The order of these entries will vary depending on compiler and linker version, and although it is possible to sort these so that lookup is O(log₂N) instead of linear, sorting them is a bit annoying to do as it requires direct ELF object file modification.  My approach is to read the final linked ELF object file, then generate a new C source file with the actual array, that when compiled, reproduces the section, with a different section name.  The final linking is then redone, dropping the original section, and including the recompiled section.  This approach is very robust, and pretty portable across architectures.)

This is particularly useful when you have a build system that conditionally compiles and/or links in objects depending on build configuration, because this way the debug interface does not pull in all code, and does not need to know the build configuration.

I obviously use the same for command-response interfaces, where each command is described using a similar structure.  That way, commands do not need to be listed and edited in a central array or list, and instead can be described in their source files.  The actual descriptor array is then constructed at link time by the linker, with the array stored in Flash, nor RAM.

[SiliconWizard]

If you have separate print functions for each type you use, then in C11 and later one can use preprocessor macros and _Generic to implement a variadic print() macro that expands to individual function calls;

Yes, I'm actually writing a replacement for printf functions, my approach is more with passing the various elements via an array to a single function call. I also use macros and _Generic to make defining this array easier.
I wish we had a slightly more elaborate preprocessor though. Variadic macros are pretty limited and "iterating" through a variadic macro arguments requires ugly "hacks".

[cv007]

If using C++, you can write a cout 'style' replacement without much trouble. This example lives in a single header-

https://godbolt.org/z/rfrfYrsvz

In the above case, the online compiler used is x86 just so can view stdout and can develop/test in the online compiler and move to your pc when done. Any mcu with a c++ compiler will do, such as avr, any cortex-m, etc. The example uses string_view from the c++ standard library which the avr doe not have, but can easily change to eliminate string_view if not available.

I have used this in various cortex-m as the newlib stdio code is too complex to understand (if you care about that). Whether is is better/worse than printf, not sure, but you do get any problems sorted at compile time rather than waiting to find out at runtime. Another advantage is it becomes easy to get formatted output of your own types, and is easy to 'attach' to any class which simply has to produce a single char output (whatever that may be- hardware, a buffer, doesn't matter).

It probably looks ugly if you cannot read C++, but there is really not much to it. There are 3 main functions that do the work (string output, number to string, double to string), and everything else is dealing with options and getting C++ to 'direct' incoming data to the right place. The example could be stripped of ansi things to make it smaller, but I left them in place as ansi output becomes easy (and the online compiler now seems to handle ansi codes). The 'double to string' conversion function is probably not perfect, but for my mcu use it seems to be fine. Also not going to win speed contests, but I do not enter any contests and just want reliable formatted output I can drag from mcu to mcu without change. You would think the code size could be a problem, but have found it makes little difference vs printf- even with extensive usage.

If I find any mistakes or changes I need/want, I have a single page of code to deal with and its not complex. Also note how many defines and macros are needed.

[i509VCB]

Maybe a little unrelated, but if your only use for printf is going to be debugging info and it the result is not shown to the user (i.e. serial console) then something like porting defmt (https://defmt.ferrous-systems.com/introduction) to act like printf from C could be very interesting. The gist of defmt is that at compile time the string format is converted to an ID. The "printf" in this case will just write the message ID and any arguments via whatever output you use (RTT for SWD debug, UART, etc). The host PC for debugging since it has the binary running on the target can interpret the ID and arguments and do the actual expensive printf formatting.

[brucehoult]

Maybe a little unrelated, but if your only use for printf is going to be debugging info and it the result is not shown to the user (i.e. serial console) then something like porting defmt (https://defmt.ferrous-systems.com/introduction) to act like printf from C could be very interesting. The gist of defmt is that at compile time the string format is converted to an ID. The "printf" in this case will just write the message ID and any arguments via whatever output you use (RTT for SWD debug, UART, etc). The host PC for debugging since it has the binary running on the target can interpret the ID and arguments and do the actual expensive printf formatting.

I already suggested doing this kind of thing in https://www.eevblog.com/forum/microcontrollers/low-footprint-printf-replacement-for-embedded-dev/msg5742613/#msg5742613

Has the benefit of not having to change the C source code. Just some tricks with the .o files, or maybe compiling via .s.

I've done it a few times in the past.

[Nominal Animal]

One important point to consider is the difference between snprintf() and printf()/fprintf().

In embedded environments in C, I typically need the former.  In C++, the latter is usually an interface on top of a communications stream, implemented on top of a common base written=write(buffer,length) functionality.

The main difference is the buffer handling and partial writes.

For snprintf() and similar interfaces, the caller owns and is responsible for the buffer.  If the buffer has insufficient space, the caller can easily observe it from the result value.

For printf()-type interfaces, the underlying communications stream owns the buffer, and the call cannot return before all of the string has been buffered.

Neither is well suited for asynchronous interfaces where the call begins the buffering of the resulting string, with a separate interface provides information on the progress of the buffering/sending/storage.  Current C interfaces are particularly poorly suited for such asynchronous interfaces (and coroutines, which would make such async interfaces much easier to implement, by generating the result characters on an as-needed basis; noting, however, that the final reversal of the output string would not be feasible then).

(Such async printing interfaces are much more common in systems programming and "templated" responses from service daemons; that is where I encountered them first a couple of decades ago.  As IoT use increases, I suspect the usefulness of such interfaces will increase even in the microcontroller/small-embedded domains.)

[SiliconWizard]

Neither is well suited for asynchronous interfaces where the call begins the buffering of the resulting string, with a separate interface provides information on the progress of the buffering/sending/storage.  Current C interfaces are particularly poorly suited for such asynchronous interfaces (and coroutines, which would make such async interfaces much easier to implement, by generating the result characters on an as-needed basis; noting, however, that the final reversal of the output string would not be feasible then).

Yes, what you mean is that the printf functions are all "blocking" - they can only run to completion, even if said completion is a truncated string (in the case of the s*prinff functions).
For "async" behavior, one needs to retain the state. That would require passing a "state" variable (typically a structure). It does make things more complex to implement obviously.

For my replacement, I'm thinking of adding something like this. To make things simpler and more efficient, my thought was to handle the number conversions strictly synchronously and the overall string formatting asynchronous. Meaning if there are numbers to format, numbers would need to be each entirely converted, which would just require a relatively small minimum buffer (numbers will typically require only a few bytes as a string, maybe a couple tens of bytes for the very longest ones). If you see what I mean.

[Nominal Animal]

Nothing prevents you using the C++ compiler instead of the C compiler.

Actually, there is one thing: older AVR (and other Harvard architecture MCUs) and named address spaces, on plain hardware, outside Arduino core libraries.

For very specific but oddball reasons, g++ does not support named address spaces nearly as well as gcc does.  Clang has no issues in C or in C++, though, but unless I'm mistaken, clang's/llvm's AVR support is not as complete as gcc's.

The benefit is being able to differentiate between character data in RAM and character data in Flash at the type level; including _Generic() macros.  This is not really possible in Arduino (using g++) right now, at least the last time I checked.

For my replacement, I'm thinking of adding something like this. To make things simpler and more efficient, my thought was to handle the number conversions strictly synchronously and the overall string formatting asynchronous. Meaning if there are numbers to format, numbers would need to be each entirely converted, which would just require a relatively small minimum buffer (numbers will typically require only a few bytes as a string, maybe a couple tens of bytes for the very longest ones). If you see what I mean.

Yes, I do believe I do.  For standard printf-type formatting strings, it means converting the format string to the stateful structure(s) synchronously, plus supplying RAM for the temporary buffer needed for longest dynamically generated field.  It is "wasteful" in the sense that it would be possible to generate these structures at compile time, for lower run-time cost.

In fact, I have done something similar, except without the printf-like part: users supply an array of fields, temporary one-field conversion storage, plus the pointers to the format specifications or formatters for each field (these being in Flash, either read-only or code).  This worked well for both embedded and systems programming under a fully featured OS, so I'd say your main hurdle will be the string-to-structure conversion, and using compact and efficient structures.  (I found that separating the varying state kept in RAM, and the formatting specs kept in Flash, was quite important.)