stdlib + MCU haters: std itoa() uses less resources!

[Siwastaja]

That is text to binary, which is a different problem.

What do you mean? What is itoa() if not text-to-binary? EDIT: Oh, I got it, you are just talking about the conversion direction.

[Nominal Animal]

That is text to binary, which is a different problem.

Haven't you seen how similar they are?

Besides, that entire thread I linked to is exactly about converting numeric data to strings.  You just commented on what I described where I initially encountered the problem, years before getting my hands on my first microcontroller (AT90USB1286, Teensy 2++).

Even then, like I said, I had to export the numerical data (from binary) to text formats anyway, to use standard tools.  So it was definitely the exact same problem on that part.

As is floating point.

I said, that was the context I first encountered the problem: standard library functions revolving around string generation and parsing being slow, as they are designed for correctness for all possible inputs/outputs, and not speed, efficiency, or size.

Consider it a note how the problem is not just on tiny machines, but also when dealing with massive amounts of data interchange.

While one might think floating-point is magically different, the coordinate system in molecular dynamics simulations tends to be in Ångströms, with at most eight decimals in the typical decimal formats.  That is, 99.9% of numbers one encounter are of format [ '-' ] ['0'-'9']{0,8} [ '.' ['0'-'9']{1,8} ].  It differs from integer parsing only in that exact rounding has to be maintained; there are quite a few different ways to do that, even though arbitrary-precision arithmetic is the currently most used / accepted method.

[Nominal Animal]

That is text to binary, which is a different problem.

What do you mean? What is itoa() if not text-to-binary?

You're thinking of the old standard atoi()/atol() functions.

itoa(value,buffer,radix) is non-standard binary-to-decimal-text conversion found in avr-libc and newlibc, among some other C standard library implementations.

[brucehoult]

That is text to binary, which is a different problem.

Haven't you seen how similar they are?

In integers, yes, but only if you regard multiplication as similar to division, which it isn't, especially on a machine such as the Cortex-M0+ which has a 1-cycle multiply but doesn't have a divide instruction at all. That makes parsing numbers from text much much faster than the inverse.

In floating point the operations are even more different. While the exact decimal expansion of a binary floating point value can get very long -- one significant decimal digit for every bit after the binary decimal point [1] -- the number of significant decimal digits you need to output in order to distinguish one floating point number from the adjacent ones (and thus be able to convert the text back to the exact same binary value later) is strictly limited.

I've written before about parsing arbitrary floating point numbers, using big (~144 bytes IIRC) but not infinite integers. In extreme cases. But in practice most numbers encountered can be correctly handled by parsing a 53 bit integer and multiplying by a power of ten from a table.

[1]  2^-1022 (about 2.22e-308) has 307 0's after the decimal, followed by 1022 non-0 digits ending with a "5".

[MK14]

I doubt that speed of converting integers to strings is very often a concern in embedded uses, even on a 1 MHz 6502 let alone a 24 or 48 or more MHz Cortex-M0. It's probably only updating a small display a few times a second (no point in doing it more often than a human can observe and react to), or writing to a log file where you don't want to be spewing out MBs of text every second. Something like 223 cycles (maybe 5-10 µs) doesn't really need to be reduced, and if it was 10x more probably would not matter.

The code size has got to be much more important in most cases.

Not necessarily, let me give you an example.

[Hypothetically]
There is some kind of sensing unit (like smart meters (gas/electricity etc), hand-held meters, etc).  With an MCU solution, running from batteries, on some kind of low cost MCU.

Although it might be able to run at 48 MHz or even 72 MHz etc, it may well be running at 500kHz, or 32,768Hz, maybe even only briefly turning on to make measurements, then going back to sleep mode.  To minimise power/battery consumption.

Let's say the LCD (for very low power consumption) display is updated every 1 to 5, times a second.

If during development, the LCD display sometimes reportedly shows weird results, perhaps every few hours, so maybe around every 10,000 readings.  Indicating a possible software (much less likely hardware) bug(s).  The developer(s), probably don't want to watch it and wait around for several hours, when the glitch(s) occur.

So they may want to use a very high (for battery powered MCUs) speed port (such as the SPI or QSPI), at a greatly speeded up rate (maybe 100,000 times a second, just for the diagnostic debugging test work), to help them spot/diagnose and hence hopefully fix the (otherwise) very rarely occurring glitch.

Or they may want to send hundreds of thousands of ADC readings per second (if the hardware is up to it), to a PC, to try and see the glitch occurring.

As you stated (in another post), it might have a very small RAM and flash, making direct logging on the MCU infeasible for any length of time.

You could argue, that they should use binary to avoid the processing delay.  But text is a lot friendlier, and easier to search for faulty/glitchy patterns.  Especially if there are separate people/departments/organisations at either end (e.g. An internal software department and external testing and verification services company).

So, a text based communication, is much less likely to cause confusion, and tends to be easier to fix and diagnose, if people get things wrong.

Therefore the conversion speed, of integer to text (string) format, may need to be as fast as possible.

[AVI-crak]

So they may want to use a very high (for battery powered MCUs) speed port (such as the SPI or QSPI), at a greatly speeded up rate (maybe 100,000 times a second, just for the diagnostic debugging test work), to help them spot/diagnose and hence hopefully fix the (otherwise) very rarely occurring glitch.

Well-written code with explicit indication of variable types is able to work on any device and in any executable environment.
In this way it is possible to carry out tests of individual functions on a large machine (PC).

[Nominal Animal]

That is text to binary, which is a different problem.

Haven't you seen how similar they are?

In integers, yes, but only if you regard multiplication as similar to division

Only if you consider multiplication/division-with-modulus the only way to implement them.
I do not, nor does my example in the other thread, which uses subtraction for string construction (and parsing uses addition).  I do consider addition and subtraction with integers "similar".

I do feel you are nitpicking here.  You only pick one small part from my answer, and reply to that, but not to my point at all.

In my first message, you replied to just my side note how I first encountered the common greater problem, disputing it as "similar".  In my second message you replied to only the first sentence, completely ignoring the note how I did have to do integer-to-string conversion anyway.

What gives?

[AVI-crak]

For very flawed microcontrollers (no multiplication, no division) - there is an algorithm for dividing by 10 by shifts and addition. The general meaning is very simple: first line "multiply" by 0.75, and then slowly "multiply" to the value of 0.8. However, when "multiplying" the least significant bits are discarded at the very beginning of the algorithm, so the result is sometimes less than one.
The algorithm works on cortex-m0 faster than all other options, the size is also small.

Code: [Select]

#include "stdint.h"

char* u32_char (char* tail_txt, uint32_t value){
    *tail_txt = 0;
    uint32_t res, rem;
    do{
        res = value >> 1;
        res += res >> 1;
        res += res >> 4;
        res += res >> 8;
        res += res >> 16;
        rem = value + '0';
        value = res >> 3;
        res = value + (value << 2);
        rem -= res << 1;
        if (rem > ('9')){value++; rem -= 10;};
        *(--tail_txt) = rem;
    }while (value > 0);
    return tail_txt;
};bug fixed

[cv007]

If during development, the LCD display sometimes reportedly shows weird results, perhaps every few hours, so maybe around every 10,000 readings

The less optimized conversion may be costing an extra half hour of 'energy'/cpu cycles per year for the normal use.

For the high speed diagnostics, it now takes you an extra fraction of a second to get to the 10k reading point.

Obviously having real numbers to work with would make for better/concrete answers, but there really is not much there to get excited about for both infrequent and frequent uses of an itoa. Any optimizations made are not bad, it just may not be worth the extra time involved to get there if in the end it makes little to no difference. The readability of code should also enter the equation- having simple/obvious code will have a benefit as opposed to a code base full of special optimized code where the why/how is long forgotten when it needs revisiting in the future.

The same can be true for size- if an optimized version of some library function results in a savings of 100 bytes, it probably makes no difference in the end for the 16k mcu.

there is an algorithm for dividing by 10 by shifts and addition

Except it may be hard to get right. You showed an earlier version that is different, and now this version does not seem to work-
https://godbolt.org/z/sYn95fhPz

So saving some clock cycles on each conversion may not be worth the effort if one has to keep tweaking the code, or if it is easy to get wrong. The simpler version using div/mod, or using something from the library like itoa/snprintf may be worth the cost in terms of speed or size.

[MK14]

If during development, the LCD display sometimes reportedly shows weird results, perhaps every few hours, so maybe around every 10,000 readings

The less optimized conversion may be costing an extra half hour of 'energy'/cpu cycles per year for the normal use.

For the high speed diagnostics, it now takes you an extra fraction of a second to get to the 10k reading point.

Obviously having real numbers to work with would make for better/concrete answers, but there really is not much there to get excited about for both infrequent and frequent uses of an itoa. Any optimizations made are not bad, it just may not be worth the extra time involved to get there if in the end it makes little to no difference. The readability of code should also enter the equation- having simple/obvious code will have a benefit as opposed to a code base full of special optimized code where the why/how is long forgotten when it needs revisiting in the future.

The same can be true for size- if an optimized version of some library function results in a savings of 100 bytes, it probably makes no difference in the end for the 16k mcu.

I agree with you, here.  Optimising itoa, is just one, of hundreds or even thousands of other optimisations, that the language compiler (C/C++ etc), needs to make, to achieve leading optimisation results, (especially) at high optimisation (flag) settings.

So, as long as itoa, is as optimised as all the other operations, or less so, if it is considered a lower priority than some other, much more frequently (in general) used, things.  I guess all is good.

But (in theory), there always can be special, rare circumstances, where the execution time of something, happens to be especially critical, for various reasons.  On the other hand, there are often many ways of fixing such situations.  Which in the case of itoa, many alternative solutions, are probably perfectly viable, in most or all situations, depending on the exact application/requirements/specifications/budget etc.

[AVI-crak]

You showed an earlier version that is different, and now this version does not seem to work-

You are right, there really was a mistake.
And now I will tell you how to get such an error without sms and registration. It is enough to install the Russian keyboard layout, and a new embitz.
While the project is open in ide embitz2,5 - everything works fine. But when the ide is closed, or the project is completed, the project is automatically saved with the juggling of variables. The IDE literally rearranges variables within functions, not all - but only a small part.
Without version control, it's very difficult to get projects back up and running. This particular project was considered temporary, with no version control.

[westfw]

Almost no one objects to "stdlib" because itoa() supports bases up to 35 and is therefore somewhat bigger than a more optimized version that only handles the bases you're interested in.
They object when using sprintf() to concatenate some strings sucks in 30k of code because the library thinks printf is part of stdio and a string is just a special case of a file, so it needs all the rest of stdio as well. :-(  (not to mention maybe floating point libs...)

[AVI-crak]

Almost no one objects to "stdlib" because itoa() supports bases up to 35

Where is base 3 or 7 applied? I'm just curious.

[Nominal Animal]

Almost no one objects to "stdlib" because itoa() supports bases up to 35

Where is base 3 or 7 applied? I'm just curious.

The only use case for me has been specifying sets of states related to combinatorics (to show to an user, or to parse from user input).

Base 3 is useful for specifying sets of tristate logic pins.
For a set of five tristate pins, there are 100000₃ possible combinations, i.e. 3⁵ = 243.
Any combination in base 3 is easy for humans to read and write, for example using 0 for High-Z, 1 for Low, and 2 for High.  Say, 02111₃: leftmost pin is high-z, second pin from left is high, and the three rightmost pins low.

An example of why such tristate logic combinatorics problems can be useful, is Charlieplexing LEDs.  Construct all base-3 numbers with as many digits as you have outputs.  Each lit LED is between a high and a low pin, in that order (anode-cathode).  Only the combinations where at most one LED is lit, are used (so that current, and thus brightness, is constant).

The reason for using itoa() or similar within the combinatorics problems is that converting to a string makes it easier/faster to check individual digits.  If you check all digits at least once, and the string fits in the fastest cache, it is faster to convert it only once.

[cv007]

when using sprintf() to concatenate some strings sucks in 30k of code

I don't see the size as a problem, as it seems to be less than 3k for any basic usage. The bigger problem for me is the use of heap which I like to avoid, and the fact that the stdio source code is pretty much unreadable so cannot easily understand what its doing or how it works (such as what is going on with FILE, which is a huge struct).

I use C++, and add 'LDFLAGS += -Wl,-wrap=_malloc_r' to my makefile so I do not get any heap usage without my knowledge (the specs file in use actually prevents it first, as there is no _sbrk). Since iostream is basically too large for about any mcu, I created my own format class which works about the same as the typical C++ cout usage. It is also much easier to code than something like a printf (its about 400 lines in a header, and only a couple functions do any amount of real work).

I use this when I want to mostly do my own thing, but will still use snprintf for something like an efr32 where I am using their ide/drivers/ble stack/etc. and I have already given up much control and am mostly in a C world (I still create the app code in a C++ source file, and wrap the driver/stack C code for easier use if it makes sense to do). This format code will also work fine for an avr, but the avr printf system is already small, uses no heap, is easy to understand and is easy to hook into to get the output directed anywhere you want, so the bigger advantage in that case is eliminating the use of % syntax which always seems to require an investigation for anything other than the simple/common things.

Not a big fan of the cout style, but with some helper functions it ends up being nicer than I first thought. Just like printf use can sometimes require some code formatting to make it readable (lots of vars to handle, takes multiple lines to write), this style also can require a little work in making multiple lines readable. It is also nice not having to match up left side format with right side vars.

Code: [Select]

//stm32g031k8
// ~3k (empty main is 0.5k)
#include <stdio.h>
int main(){
    char buf[64];
    snprintf( buf, 64, "%s %s 0x%08X%c", "Hello", "World", 123456, '\n' );
    //to make sure compiler does not optimize something away
    auto p = buf;
    volatile char c;
    while( *p ) c = *p++;
    (void) c;
    while(1){}
}


//stm32g031k8
// ~1.3k (empty main is 0.5k) (exact same makefile as above)
#include "Print.hpp"
//trying to duplicate snprintf example so is a fairer comparison
template<int N>
class Astring : public FMT::Print {
    char buf_[N]; //uninitialized
    int idx_{0};
    bool write(const char c) override {
        if( idx_ >= (N-1) ) return false;
        buf_[idx_++] = c;
        buf_[idx_] = 0;
        return true;
    }
    public:
    Astring(){ clear(); }
    void clear(){ idx_ = 0; buf_[0] = 0; }
    char* operator()(){ return buf_; }
};
int main(){
    Astring<64> str;
    using namespace FMT;
    //'normal' cout style, which ca get quite verbose (this code will work)
    // test << "Hello " << "World " << showbase << hex << uppercase << setw(8) << setfill('0') << 123456 << endl;
    //but some helper functions can make it more pleasant to use (but still doing the same thing as previous line)
    str, "Hello ", "World ", Hex0x(8,123456), endl; //I added comma operator also, which I prefer
   
    //to make sure compiler does not optimize something away
    auto p = str();
    volatile char c;
    while( *p ) c = *p++;
    (void) c;
    while(1){}
}

[westfw]

[printf] seems to be less than 3k for any basic usage.

Well, it'll depend on exactly which library setup you're using.  A useless error message printf() in the Arduino Due (SAM3x ARMCM3 with newlib, IIRC) added about 12k: https://github.com/arduino/ArduinoCore-sam/issues/47

[cv007]

They object when using sprintf() to concatenate some strings sucks in 30k of code

I assumed you were talking about sprintf. I have never used printf for 32bit as I will use snprintf and hook up (the resulting buffer) to a device on my own, as that is more straight-forward for me. I now tried printf instead of snprintf in my example, with no underlying mechanism to put it to use, compiles to about 5k (0.5k baseline). Not hard to imagine the library pulling in things left and right, so I guess also not hard to imagine the size can explode and the reasons for it may require some digging.

The makefile is nothing special, and using whatever newlib (nano) version is included with the compiler in use-
https://github.com/cv007/NUCLEO32_G031K8_chrono/blob/main/Makefile
but gcc 11.3 changed the specs files in some way, and I now just use nano.specs.I think the earlier version(s) required both nano.specs and nosys.specs to skip the syscalls.

[nctnico]

[printf] seems to be less than 3k for any basic usage.

Well, it'll depend on exactly which library setup you're using.  A useless error message printf() in the Arduino Due (SAM3x ARMCM3 with newlib, IIRC) added about 12k: https://github.com/arduino/ArduinoCore-sam/issues/47

There are smaller versions of printf available. There really is no reason to avoid using printf and it's cousins. Just use the right one for the job.

[SiliconWizard]

[printf] seems to be less than 3k for any basic usage.

Well, it'll depend on exactly which library setup you're using.  A useless error message printf() in the Arduino Due (SAM3x ARMCM3 with newlib, IIRC) added about 12k: https://github.com/arduino/ArduinoCore-sam/issues/47

There are smaller versions of printf available. There really is no reason to avoid using printf and it's cousins. Just use the right one for the job.

Yes there are, and if you're using printf() a lot in your embedded application or using it in critical parts, then sure I also highly suggest using an alternative printf() implementation.

Now, unless you already have such a function in your arsenal (tested and proven) and the only thing you need is to convert an integer to its decimal representation in ASCII, writing the function yourself will only require a few lines of code and will take a lot less time than finding one alternative printf() that can be trusted.

All obvious, but people spending hours or days looking for existing solutions to a simple problem they could have solved in a few minutes is all too common.
And most often, "reinventing" a small wheel is a lot more effective.
Just saying.

[nctnico]

Re-inventing a wheel is never good. I prefer somebody spending time on finding a good existing solution / design pattern rather than trying to come up with something 'quickly' by themselve. In the end: if it takes somebody long to find a good solution, then it is likely the solution they come with by themselves is going to be worse. The first post in this thread is a good example. Better make sure to have a whole bunch of wheels on hand so you are prepared for every situation. Otherwise you end up with a mine field of self induced bugs.

I've seen my fair share of printf workarounds that where just steaming turds. Adhering to Posix is a good idea in general. A lot of thought, effort, trial & error, etc went into the function calling conventions.

[Siwastaja]

Re-inventing a wheel is never good. I prefer somebody spending time on finding a good existing solution / design pattern rather than trying to come up with something 'quickly' by themselve.

Do you realize this is a textbook example of cargo cult engineering?

Adhering to Posix is a good idea in general

Completely impractical on microcontrollers, except for the few largest ones. And supporting 0.01% of POSIX does not add any value from being POSIX-y itself.

[nctnico]

Re-inventing a wheel is never good. I prefer somebody spending time on finding a good existing solution / design pattern rather than trying to come up with something 'quickly' by themselve.

Do you realize this is a textbook example of cargo cult engineering?

No, not at all. It is all about avoiding NIH syndrome and doing a poor job. If you want to know/learn something new, are you going to read a book about the subject or start from scratch and make a crap load of mistakes somebody else has already made (and thus could be avoided)?

Cargo cult engineering is just incompetent people never becoming competent at their job and incompetent managers leaving them muddling on. Sure that is a problem but far from a valid reason not to re-use existing code and apply well established design patterns. Competent engineers know their limits and thus when to do research / re-use existing work but also know to evaluate existing solutions / patterns and apply these to their problem at hand.

Adhering to Posix is a good idea in general

Completely impractical on microcontrollers, except for the few largest ones. And supporting 0.01% of POSIX does not add any value from being POSIX-y itself.

No. I'm not saying to implement everything from Posix but if you do use a function / piece of functionality that exists under Posix, use the Posix API. It saves you from documenting your own code, stepping into pitfalls others have stepped into before and makes code more portable.

[Siwastaja]

But those "small version of printf" you suggest are not POSIX compliant and thus require extra documentation what is actually supported and what is not, and more often than not this documentation is not properly available. The only thing you have here going is the feeling of using something standardized. Your argument is thus 100% invalid.

[wek]

IMO it's the other way round: if you find printf()/scanf() adequate to your work, you are probably not programming microcontrollers (both 'micro' and 'controllers' is meaningful here).

JW

[nctnico]

But those "small version of printf" you suggest are not POSIX compliant and thus require extra documentation what is actually supported and what is not, and more often than not this documentation is not properly available. The only thing you have here going is the feeling of using something standardized. Your argument is thus 100% invalid.

Sorry but this just sounds like yet another attempt to justify re-inventing the wheel. I wrote to use the Posix API, not implement all off the functionality! As an example: smaller printf implementations often allow to enable / disable features based on the requirements. A competent engineer will see the limitations and act accordingly.