[Solved] Saving "printf" arguments for later? Real-time on slow processor.

[tggzzz]

P.S. Is there any way to upgrade your CPU, or add a "printf coprocessor" to the system?

With xC+xCORE, I've used one core for each input, one for central controller, one for front panel, and the USB comms library used 8 cores (I think). The RTOS is implemented in hardware

Guaranteed by design hard realtime processing fully occupied the two cores processing the inputs (<2% spare cycles), at the same time as bidirectional comms with an application running on the PC.

The first time I tried using xC+xCORE it was so painless that I had the first very simple version doing all that within a day of receiving the hardware. That made hard realtime processing fun again.

[ejeffrey]

For example, if it might be possible to copy a chunk of the printf stack? Having to parse the format string (to get the number and type of arguments) and copy the variadic arguments one by one is a step that I would not mind pushing off to a background thread. But I'm not sure if it is even possible.

You can't do that in portable C as the number and size of arguments is not stored explicitly.  The only portable way to operate on va_list is with the va_start, va_arg, va_copy, and va_end macros, or to pass them to another function that does the same.

Parsing the format string and finding the arguments is a bit of work to code but should be pretty fast.  So I would try that approach first.  The main overhead of printf is formatting the outputs.

If you are OK with non portable code it's likely possible to examine the stack frame to find the beginning and end of the varargs block.  How to do this depends on your platform ABI and likely requires some assembly.

[nctnico]

With a library that is so brittle in a realtime environment, you need to ditch that library.

If you don't, you will be playing whack-a-mole in the field.

I have no choice in the mater. The library, the processor, and the realtime-requirements are not changable in this application.

One option is to print the floating point numbers as hex and mark these in the output for processing later on (like prefixing these numbers with a special character). Or go a step further and print every numerical argument as hexadecimal. You can't avoid going through the formatting string because you need to determine the length of the argument given (a double is 8 bytes for example) but you can skip expensive conversions. If you have enough flash or ram, byte to ASCII conversion can be done through a look-up table.

[Nominal Animal]

In debug mode it easily generates several kilobytes of formatted number containing text per minute. And, what the library does is so complicated that we really need the debug output to be generated all of the time (both in development and production).

Unfortunately, It turns out that printf is so slow that it causes the system to fail in a variety of subtle yet completely catastrophic ways. (due to it missing real time deadlines and not being able to detect that it has missed them)

Has anyone dealt with this before?

Not the exact situation, but I've worked a lot on high-performance computing, especially on classical-type molecular dynamics simulators, which generate a LOT of numerical data.  For example, a colleague ran a large simulation with short snapshot time intervals which produced some 4 Tb of data, basically in Protein Data Bank file format –– essentially formatted text with many numeric fields.

I've specialized in simulator development.  Currently, the main problem is very similar to what you see: instead of interleaving communications and state snapshotting with computation, they compute, then communicate/snapshot, then compute, and so on, wasting a significant fraction of available CPU time.  (Some people have objected to my complaint by saying that if they fully utilized the CPUs in the clusters, it "might overwhelm their cooling capability", which to me is an asinine cop-out.)

The solution here is obvious: instead of formatting the output immediately, you need to store the output data, and optionally format it at leisure.

Because printf() format string parsing takes a surprising amount of unnecessary CPU time, and you have to do that in order to determine the number and type of each parameter, I would recommend reimplementing the output using individual functions per parameter type.  That is, you convert
    printf("Foo %d bar %.4f baz\n", x, y);
into something like
    // Converted from printf("Foo %d bar %.4f baz\n", x, y);
    emit_const("Foo ");
    emit_int(x);
    emit_const(" bar ");
    emit_float(y, 4);
    emit_const(" baz");
    emit_newline();
Note that I'm assuming you use C and not C++ here.  With C++, overloading << is a better approach.

The emit_() family of functions (or the overloaded operator in C++) appends the type of the parameter and the parameter value to a circular buffer.  This takes minimal time.

When the processor is idle, it converts one parameter from the circular buffer to output and returns, minimizing the latency/duration to one single conversion at a time.  You can then choose whether you allow the buffer to overrun, or whether you convert from the circular buffer before adding more to it.  It is even possible to flush this buffer to storage (say microSD card or similar).

In C, I would separate the parameter type into its own circular buffer, and the parameter data to its own fixed-size (32- or 64-bit/4- or 8-byte) records:

Code: [Select]

#define  LOG_SIZE  29127 // Number of elements in the circular buffer; this is 262143 == 256k - 1

union log_item_union {      /* Type, example values only */
                            //      0      reserved for unused item
    char            c[8];   //   1 .. 8,   length
    uint8_t         u8[8];  //   9 .. 16,  count+8
    int8_t          i8[8];  //  17 .. 24,  count+16
    uint16_t        u16[4]; //  25 .. 28,  count+24
    int16_t         i16[4]; //  29 .. 32,  count+28
    uint32_t        u32[2]; //  33 .. 34,  count+32
    int32_t         i32[2]; //  35 .. 36,  count+34
    uint64_t        u64[1]; //     37
    int64_t         i64[1]; //     38
    float           f32[2]; //  39 .. 40,  count+38
    double          f64[1]; //     41
    const char     *cc;     // 42 .. 255, length+41
};

uint8_t                 log_type[LOG_SIZE];
union log_item_union    log_item[LOG_SIZE];
uint_fast16_t           log_head = 0;   // Next free item in buffer
uint_fast16_t           log_tail = 0;   // Oldest item in buffer, unless equal to head
This uses 9 bytes of RAM per log item.  The type specifies the item type in the 64-bit union, and supports various numeric types and vectors (whatever fits into 64 bits), various constant strings stored in Flash, plus short strings of up to 8 bytes/chars long.  The reason for not interleaving the type and the item data is hardware alignment requirements; typically, larger than byte sized items either require an aligned address, or are slower to access from an unaligned address, depending on the hardware architecture used.

Note that the above format would also allow emit_2float(f1,f2), emit_i16(i1), emit_2i16(i1,i2), emit_3i16(i1,i2,i3), emit_4i16(i1,i2,i3,i4), and so on.

Note that you can then choose whether you convert the circular buffer contents to strings at your leisure, or whether you simply write them to some serial stream.  The key is to do it when your CPU is otherwise idle, and keep each conversion/write operation short enough to not block so long as to affect the main computation.  Writing the data synchronously just will not work; the circular buffer here is what makes the prints asynchronous, and lets you handle the actual operations later on in small enough chunks to not tie up your resources for too long.  For floating point types, using a fixed number of decimals and a limited range (say, -99999999.99999999 to +99999999.99999999) means you can speed up the conversion significantly compared to printf() et al.; it is supporting values very close to zero or very large in magnitude that makes generic precise conversion to string slow.

[magic]

If the library only uses integer numeric formats (%d, %u, %x, etc) and the format strings are always statically allocated then it should be simply a matter of saving all arguments to printf to a buffer (as raw 32/64 bit words) and calling real printf on them at a later time.

Not sure if it would work with floats, I'm not familiar with typical calling conventions.

%s could be a problem if the strings passed to the format may be deallocated or overwritten by the library, then your fake print must make a copy.


If you find that some problematic cases exist, you may write your own lightweight format parser which only decides how to copy and store each argument. Maybe you could extract all format strings from the library ahead of time and generate optimized functions for each, then use a hash table to dispatch the functions based on format string pointer (this again assumes that format strings are statically allocated and immutable, which is the usual case).

Have fun
And, of course, you shouldn't be using crap libraries in the first place.

[radiolistener]

technically you can write your own printf, which will use linked list to store passed arguments (format string and values), then you can process it from separate background thread - format string and write it to slow output. Note, that you're needs to copy string buffer with format string and arguments, because they can be changed when printf call returns control to the caller.

The only issue here is to detect size of passed argument. Usually I'm using such approach on managed languages, where you can check actual data type at runtime, I'm not sure if there is fast and simple alternative for C. I think there is possible problem to detect argument size for %s format, because it may have more than standard 4/8 bytes, like int/float/double

More reliable way is just to store data in queue and then process it in separate thread. But it will be less comfortable than usual printf. This is how I do such tasks in C.

[magic]

Don't bother with lists. If the machine has "lots of free RAM" then simply use an array of arrays, each long enough to store the worst case number of parameters.

[ejeffrey]

If the library only uses integer numeric formats (%d, %u, %x, etc) and the format strings are always statically allocated then it should be simply a matter of saving all arguments to printf to a buffer (as raw 32/64 bit words) and calling real printf on them at a later time.

"Saving all arguments" is the entire hard part.  The way varargs work in C makes this difficult to do.

However if you can enumerate the possible format strings at compile time, you could build a hash table of format strings -> precomputed # of arguments.  That would save parsing the format strings at runtime. 

[Nominal Animal]

If one uses an ELF-based toolchain, then you can do a "trick" at build time to record all the printf() formats your code uses.

With GCC and clang-based toolchains, you can do e.g.
    #define  logprintf(fmt, ...) \
             { static const char format[] __attribute__((section ("printformats"))) = fmt; } \
             printf(fmt, __VA_ARGS__)
and then compile your firmware; then run
    objcopy -O binary -j printformats input.o output.bin
to get all the format strings in input.o with at least one nul (\0) in between in output.bin.  I like to use something like the following C99 or later program to extract them in their original C string format with the number of bytes it takes:

Code: [Select]

// SPDX-License-Identifier: CC0-1.0
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

const char special[9][4] = {
    [0] = "\\a",
    [1] = "\\b",
    [2] = "\\t",
    [3] = "\\n",
    [4] = "\\v",
    [5] = "\\f",
    [6] = "\\r",
    [7] = "\\\"",
    [8] = "\\\\",
};

void read_from(FILE *in, FILE *out) {
    while (1) {
        size_t  n = 0;
        int     c = getc(in);

        while (c == 0)
            c = getc(in);

        if (c == EOF)
            return;

        fputc('"', out);
        while (c != EOF && c != 0) {
            n++;
            if (c >= 7 && c <= 13) {
                fputs(special[c - 7], out);
                c = getc(in);
            } else
            if (c == 34) { // '"'
                fputs(special[7], out);
                c = getc(in);
            } else
            if (c == 92) { // '\\'
                fputs(special[8], out);
                c = getc(in);
            } else
            if (c >= 32 && c <= 126) {
                fputc(c, out);
                c = getc(in);
            } else {
                fputc('\\', out);
                fputc('0' + ((c / 64) & 3), out);
                fputc('0' + ((c / 8)  & 7), out);
                fputc('0' + ( c       & 7), out);
                c = getc(in);
            }
        }
        fprintf(out, "\" (%zu bytes)\n", n);
    }
}

static int  stdin_dumped = 0;

static int  dump_stdin(int status) {
    if (stdin_dumped)
        return status;

    stdin_dumped = 1;

    read_from(stdin, stdout);
    fflush(stdout);
    if (ferror(stdin)) {
        fprintf(stderr, "Error reading from standard input.\n");
        status = EXIT_FAILURE;
    }
    if (ferror(stdout)) {
        fprintf(stderr, "Error writing to standard output.\n");
        status = EXIT_FAILURE;
    }

    return status;
}

int main(int argc, char *argv[]) {
    int  status = EXIT_SUCCESS;

    if (argc > 1 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))) {
        const char *arg0 = (argc > 0 && argv != NULL && argv[0] != NULL && argv[0][0] != '\0') ? argv[0] : "(this)";

        fprintf(stderr, "\n");
        fprintf(stderr, "Usage: %s -h | --help\n", arg0);
        fprintf(stderr, "       %s [ INPUT-FILE ... ]\n", arg0);
        fprintf(stderr, "\n");
        fprintf(stderr, "This program will output all null-terminated C strings in C notation\n");
        fprintf(stderr, "from the standard input (\"-\") or specified file(s).\n");
        fprintf(stderr, "\n");

        return EXIT_SUCCESS;
    }

    if (argc > 1) {
        for (int arg = 1; arg < argc; arg++) {
            if (!argv[arg] || argv[arg][0] == '\0' || !strcmp(argv[arg], "-")) {
                status = dump_stdin(status);
            } else {
                FILE *in = fopen(argv[arg], "rb");
                if (!in) {
                    fprintf(stderr, "%s: %s.\n", argv[arg], strerror(errno));
                    status = EXIT_FAILURE;
                    break;
                }
                read_from(in, stdout);
                if (ferror(in)) {
                    fprintf(stderr, "%s: Read error.\n", argv[arg]);
                    status = EXIT_FAILURE;
                    fclose(in);
                } else
                if (fclose(in)) {
                    fprintf(stderr, "%s: %s.\n", argv[arg], strerror(errno));
                    status = EXIT_FAILURE;
                }
                if (ferror(stdout)) {
                    fprintf(stderr, "Error writing to standard output.\n");
                    status = EXIT_FAILURE;
                }
            }
            if (status != EXIT_SUCCESS)
                break;
        }
    } else {
        status = dump_stdin(status);
    }

    return status;
}In Linux, you can do the above for all object files in your build directory, and filter the output through | sort | uniq, and you get only the unique format strings used:
    find . -name '*.o' -exec objcopy -j printformats -O binary '{}' /dev/stdout ';' | cstrings | sort | uniq

I love the ELF shenanigans I can do in my build scripts and C/C++ sources!

[edavid]

However if you can enumerate the possible format strings at compile time, you could build a hash table of format strings -> precomputed # of arguments.  That would save parsing the format strings at runtime.

OP said he doesn't have source to the library.  This is kind of obvious since if he did, he could replace the printf calls with some lighter weight tracing, or just debug the library so he wouldn't need the debug output.

If one uses an ELF-based toolchain, then you can do a "trick" at build time to record all the printf() formats your code uses.

Doesn't anyone read the thread before posting 

[ejeffrey]

However if you can enumerate the possible format strings at compile time, you could build a hash table of format strings -> precomputed # of arguments.  That would save parsing the format strings at runtime.

OP said he doesn't have source to the library.  This is kind of obvious since if he did, he could replace the printf calls with some lighter weight tracing, or just debug the library so he wouldn't need the debug output.

That doesn't mean you can't extract the format strings, although it does make it more difficult.  The OP already said that the format strings were all just formatted numbers and short literals.  At the very worst you can make a printf that only emits the format strings, and run through enough tests to generate all important format strings.

[edavid]

However if you can enumerate the possible format strings at compile time, you could build a hash table of format strings -> precomputed # of arguments.  That would save parsing the format strings at runtime.

OP said he doesn't have source to the library.  This is kind of obvious since if he did, he could replace the printf calls with some lighter weight tracing, or just debug the library so he wouldn't need the debug output.

That doesn't mean you can't extract the format strings, although it does make it more difficult.  The OP already said that the format strings were all just formatted numbers and short literals.  At the very worst you can make a printf that only emits the format strings, and run through enough tests to generate all important format strings.

Yes, I suggested just that.  What you can't do, is do it at compile time.

(Except sort of, by running strings on the library object file.)

[ejeffrey]

Sorry I didn't mean "use the compile to extract the list" I meant get the list ahead of time so you can build a compiled lookup table.  It seems the OP likely already and the list.

[tggzzz]

Doesn't anyone read the thread before posting 

Given the number of times the same partial solution has been suggested (starting in the first reply!), the answer is a resounding no.

To me it looks like the system+library as described is not "fit for purpose". If the management can't work that out and/or refute it, their customers will.

Place your bets

[incf]

If one uses an ELF-based toolchain, then you can do a "trick" at build time to record all the printf() formats your code uses.

With GCC and clang-based toolchains, you can do e.g.
    #define  logprintf(fmt, ...) \
             { static const char format[] __attribute__((section ("printformats"))) = fmt; } \
             printf(fmt, __VA_ARGS__)
and then compile your firmware; then run
    objcopy -O binary -j printformats input.o output.bin
to get all the format strings in input.o with at least one nul (\0) in between in output.bin.  I like to use something like the following C99 or later program to extract them in their original C string format with the number of bytes it takes:

...

I love the ELF shenanigans I can do in my build scripts and C/C++ sources!

That is a fantastic trick. Not sure I can use it on this issue, but I can easily see myself doing something like that in the future.

I'm fairly convinced that I'll end up doing some assembly to copy the stack containing the variadic arguments list and format string into a buffer, and then "reconstitute" it as a call to snprintf later.

[DavidAlfa]

Do you really need printf?
For simple strings, use your own function.
If only printing integers, you may use itoa() instead...

va_copy() macro looks intesting, though I've never made variadic functions and I don't really know how they work under the hood.

https://learn.microsoft.com/en-en/cpp/c-runtime-library/reference/va-arg-va-copy-va-end-va-start?view=msvc-170.

[incf]

Do you really need printf?
For simple strings, use your own function.
If only printing integers, you may use itoa() instead...

va_copy() macro looks intesting, though I've never made variadic functions and I don't really know how they work under the hood.

https://learn.microsoft.com/en-en/cpp/c-runtime-library/reference/va-arg-va-copy-va-end-va-start?view=msvc-170.

Yes, I really do need something fairly close to printf to execute eventually. And even a minimal implementation would be too slow to run in real time.

I was under the impression that I can't really store va_list due to it having an unknown size at runtime (correct me if I am wrong).
I believe I have to write some assembly instead in order to read the total size of the parameters list (from a combination of the stack pointer and R7 - although, I am not 100% certain)

[SiliconWizard]

But again, as edavid said, the OP's problem is that this is done in a third-party library the source of which he doesn't have access to, if I got it right. So I guess we should stop suggesting replacing function calls on a source code level, like using macros.

After that, you could resort to doing it on a link level, but that becomes a mess quickly: replacing all references to printf to a different, own symbol for a custom function which has the exact same interface. And, that assumes that the black box library code only uses the "printf" function itself and never any of its variants (printf is a whole family of functions), which is possibly a dead-end.

If most of the time spent in those "printf" calls is not in the formatting code itself, but in sending it to whatever it is redirected to (like an UART?), then you could, much easier, just re-implement the '_write' function and buffer whatever is passed to it in a RAM buffer rather than send it directly to a peripheral.

[incf]

But again, as edavid said, the OP's problem is that this is done in a third-party library the source of which he doesn't have access to, if I got it right. So I guess we should stop suggesting replacing function calls on a source code level, like using macros.

After that, you could resort to doing it on a link level, but that becomes a mess quickly: replacing all references to printf to a different, own symbol for a custom function which has the exact same interface. And, that assumes that the black box library code only uses the "printf" function itself and never any of its variants (printf is a whole family of functions), which is possibly a dead-end.

If most of the time spent in those "printf" calls is not in the formatting code itself, but in sending it to whatever it is redirected to (like an UART?), then you could, much easier, just re-implement the '_write' function and buffer whatever is passed to it in a RAM buffer rather than send it directly to a peripheral.

I'm already doing two layers of buffering. IO is not an issue, it's a straightforward lack of CPU power doing string formatting operations as far as I can tell with my testing.

Hence the desire to use assembly to save the printf variable arguments list into a buffer for later processing. (Which is unfamiliar territory for me, and probably most people here - variadic arguments + C ABI for ARM cortex M0 + assembly, it's probably more than just architecture specific it is likely compiler specific - I'm using gcc)

[mwb1100]

va_copy() macro looks intesting, though I've never made variadic functions and I don't really know how they work under the hood.

I was under the impression that I can't really store va_list due to it having an unknown size at runtime (correct me if I am wrong).

You can't store the data in the va_list using only va_copy - va_copy stores the current state of the source va_list into the destination va_list.  It allows you to access the variables in the va_list again after having modified the original va_list (for example in order to make two passes over the variable arguments).

Copying the data in the va_list would require some parsing of the printf() format string to identify the number of arguments and their size. However if a format string contains a specification that passes data via a pointer, such as "%s", then your wrapper would need to copy the buffer that the pointer points to instead of just the pointer (unless you perhaps determine that the pointer points into flash).  The copy operation wouldn't need to do any actual formatting though, so it's possible that it could be fast enough for your purposes. 

If it isn't fast enough, then I think you will probably need to perform the build time (or pre-build) analysis of format strings extracted from the library mentioned previously.  Keep in mind that format strings can be generated at runtime so you might need to handle that corner case.  But I suspect that for logging it's likely that all format strings you'd see in this application are literals.

[Nominal Animal]

I'm fairly convinced that I'll end up doing some assembly to copy the stack containing the variadic arguments list and format string into a buffer, and then "reconstitute" it as a call to snprintf later.

That will not work, because the size of each argument varies, and <stdarg.h> does not provide a way to copy the parameters as a "context" for later reuse.  Even va_copy() simply allows you to traverse the argument list more than once in the same block.  This is a fundamental limitation in C, because the callee simply cannot know how many parameters the caller provided; it can only assume the caller provided all that were specified in the prototype.  For variadic arguments, there is absolutely no way in C for the callee to find out how many the caller supplied.  It is easy to work around at the source code level, but not for already compiled code.

What you can do is implement your own printf() that does the conversion I described in my first suggestion internally, via say something like

Code: [Select]

#define  _POSIX_C_SOURCE  200809L
#define  _GNU_SOURCE                // For ssize_t
#include <stdlib.h>
#include <stddef.h>
#include <stdint.h>
#include <wchar.h>      // Only if %lc and %ls are supported (wint_t and wchar_t)
#include <stdarg.h>

int printf(const char *format, ...) {
    va_list     args;

    if (!format || format[0] == '\0')
        return 0;

    va_start(args, format);

    while (*format) {
        if (*format == '%' && format[1] != '%' && format[1] != '\0') {
            // TODO: This simplified implementation ignores position and formatting, and accepts invalid formats!
            //       It does not support variable precision via %* or %*.* either.

            unsigned char  size = 0;
            while (*format != '\0') {
                if (*format == 'h') {
                    size |= ((size & 1) << 1) | 1;
                } else
                if (*format == 'l') {
                    size |= ((size & 4) << 1) | 4;
                } else
                if (*format == 'L') {
                    size |= 12;
                } else
                if (*format == 'z') {
                    size |= 16;
                } else
                if (*format == 'j') {
                    size |= 32;
                } else
                if (*format == 't') {
                    size |= 64;
                } else
                if (*format == 'd' || *format == 'i') {
                    if (size == 0) {
                        int  v = va_arg(args, int);
                        // TODO: Store integer 'v'
                    } else
                    if (size == 1) {
                        short  v = (short)va_arg(args, int);
                        // TODO: Store short 'v'
                    } else
                    if (size == 3) {
                        signed char  v = (signed char)va_arg(args, int);
                        // TODO: Store signed char 'v'
                    } else
                    if (size == 4) {
                        long  v = va_arg(args, long);
                        // TODO: Store long 'v'
                    } else
                    if (size == 12) {
                        long long  v = va_arg(args, long long);
                        // TODO: Store long long 'v'
                    } else
                    if (size == 16) {
                        ssize_t  v = va_arg(args, ssize_t);
                        // TODO: Store ssize_t 'v'
                    } else
                    if (size == 32) {
                        intmax_t  v = va_arg(args, intmax_t);
                        // TODO: Store intmax_t 'v'
                    } else
                    if (size == 64) {
                        ptrdiff_t  v = va_arg(args, ptrdiff_t);
                        // TODO: Store ptrdiff_t 'v'
                    } else {
                        // TODO: Unsupported conversion!
                    }
                    format++;
                    break;
                } else
                if (*format == 'o' || *format == 'u' || *format == 'x' || *format == 'X') {
                    if (size == 0) {
                        unsigned int  v = va_arg(args, unsigned int);
                        // TODO: Store unsigned integer 'v'
                    } else
                    if (size == 1) {
                        unsigned short  v = (unsigned short)va_arg(args, unsigned int);
                        // TODO: Store unsigned short 'v'
                    } else
                    if (size == 3) {
                        unsigned char  v = (unsigned char)va_arg(args, unsigned int);
                        // TODO: Store unsigned char 'v'
                    } else
                    if (size == 4) {
                        unsigned long  v = va_arg(args, unsigned long);
                        // TODO: Store unsigned long 'v'
                    } else
                    if (size == 12) {
                        unsigned long long  v = va_arg(args, unsigned long long);
                        // TODO: Store unsigned long long 'v'
                    } else
                    if (size == 16) {
                        size_t  v = va_arg(args, size_t);
                        // TODO: Store size_t 'v'
                    } else
                    if (size == 32) {
                        uintmax_t  v = va_arg(args, uintmax_t);
                        // TODO: Store uintmax_t 'v'
                    } else {
                        // TODO: Unsupported conversion!
                    }
                    format++;
                    break;
                } else
                if (*format == 'e' || *format == 'E' || *format == 'f' || *format == 'F' ||
                    *format == 'g' || *format == 'G' || *format == 'a' || *format == 'A') {
                    double  v = va_arg(args, double);
                    // TODO: Store double 'v'
                    format++;
                    break;
                } else
                if (*format == 'c') {
                    if (size == 0) {
                        unsigned char  v = (unsigned char)va_arg(args, int);
                        // TODO: Store single character 'v'
                    } else
                    if (size == 4 || size == 12) {
                        wint_t  v = va_arg(args, wint_t);
                        // TODO: Store wide character 'v'
                    } else {
                        // TODO: Unsupported conversion!
                    }
                    format++;
                    break;
                } else
                if (*format == 's') {
                    if (size == 0) {
                        const char  *p = va_arg(args, char *);
                        // TODO: Store string at 'p'
                    } else
                    if (size == 4 || size == 12) {
                        const wchar_t  *p = va_arg(args, wchar_t *);
                        // TODO: Store wide string at 'p'
                    } else {
                        // TODO: Unsupported conversion!
                    }
                    format++;
                    break;
                } else
                if (*format == 'p') {
                    uintptr_t  p = (uintptr_t)va_arg(args, void *);
                    // TODO: Store the address in 'p' (output would be in hexadecimal).
                    format++;
                    break;
                } else
                if (*format == 'n') {
                    // TODO: %n is unsupported!
                    format++;
                    break;
                }

                format++;
            }

        } else {
            const char *ends = format;

            while (*ends != '\0') {
                if (*ends == '%' && ends[1] != '%' && ends[1] != '\0')
                    break;
                else
                    ends++;
            }

            // NOTE: This keeps %% for simplicity.
            // TODO: Store (size_t)(ends - format) -character string starting at format.

            format = ends;
        }
    }

    va_end(args);

    // Fake return value, because we do not actually know the length of the string yet.
    return 0;
}Essentially, you need to parse the format string, and all format specifications.  The above does not support the %N... format, where N is the parameter index to be converted, nor does it support %n, but something along these lines might suffice for your needs.  Since you store the exact value to be printed, I don't see any particular need to remember how it should be formatted, either.

If you can check all the formatting strings the closed-source code does (you can find them using objdump/objcopy rather easily, since you should have the object files or the library in linkable form; ELF file.a being just a collection of ELF object file.o files; see your toolchain ar documentation), you can probably omit quite a few of the cases.  In particular, %lc and %ls are exceedingly rare in embedded environments, so you can probably omit <wchar.h>, wint_t, and wchar_t * type support.  You may not need double support at all.  And so on.

My own next step in your shoes would be to locate and inspect all the formatting strings used.  I would go as far as decompiling the closed-source file around each call to printf(), examining what the value of the first parameter (register or stack depending on function call ABI used) is, to check if any are dynamically generated, and to ensure I find all possible formatting strings I need to support.

[incf]

I'm fairly convinced that I'll end up doing some assembly to copy the stack containing the variadic arguments list and format string into a buffer, and then "reconstitute" it as a call to snprintf later.

That will not work, because the size of each argument varies, and <stdarg.h> does not provide a way to copy the parameters as a "context" for later reuse
...

I'm looking at the assembly and I think gcc stores the size of the variable arguments list using SP and R7. Using that size, I might be able to copy it to a buffer someplace. I think.

Although, I'm not sure about the case where there are less than 4 arguments. I guess I just need to spend a bunch of time analyzing the machine code that gcc emits under each possible scenario.

below is some quick and dirty (nonfunctional) test code in godbolt that shows how the printf arguments list get passed around.

R0 is a pointer to the formatting string
R1 is the first argument
R2 is the second
R3 is the third
Then arguments 4 through 8 get put onto the stack
R7 is the stack value that the printf must set its stack to before returning (I think)
SP is the bottom/top of the arguments list.

The stack also seems to contain a copy of what the stack needs set to when the printf-like function returns (I think via R7 - correct me if I am wrong)
That value, minus the stack pointer position when the printf-like function is called should give the size of the variable argument list in bytes.

Then it is just a matter of copying it around, and calling printf with the same register conventions later.

[tggzzz]

The OP hasn't said why that library must be used, nor what the library presumed about the environment in which it is executing.

If the library source is available, tweak that.

If not, talk to the creator and pay them to do a better job.

If not, use a more powerful processor or processors. If the processor is only just fast enough as it is, what happens when a small enhancement is requested. Or the cache misses are pessimal. Or an extra interrupt occurs.

Disassembling the source or what is on the stack seems a very brittle approach. Even if it appears to work now, changes to the library source or compiler or execution environment could subtly break things.

[tggzzz]

Although, I'm not sure about the case where there are less than 4 arguments. I guess I just need to spend a bunch of time analyzing the machine code that gcc emits under each possible scenario.

You mean this version of the compiler with specified flags. Compiler output changes regularly. Ditto the library code and whatever is used to compile the library.

[SiliconWizard]

Disassembling the source or what is on the stack seems a very brittle approach. Even if it appears to work now, changes to the library source or compiler or execution environment could subtly break things.

Yes. As I mentioned, printf is a whole family of functions and the library may use a whole bunch of different variants for its printf-formatting needs. Catching all of them looks, as I said, like a dead-end to me, but hey. Good luck.