how do you debug the code?

[Dadu@]

I don't understand what is the best way to debug c code. What is your general approach for any compiler to c debug code ?

Should I set a breakpoint after pressing the debug button or set a break point first and then press the debug button?

There is step in and step out button while debugging how to use it? should i stop the program then continue press step in button

[Ian.M]

That's going to be IDE, toolchain, hardware debugger and device dependent.   As you haven't told us what you are using the only applicable answer is "Do what works!" 

If you want a more general answer, how about:

42

[brucehoult]

I never use interactive debuggers.

It wasn't a good technique 40 years ago when CPUs ran 250 thousand instructions a second and I could press the "step" button realistically once or twice a second.  It's a MUCH WORSE technique today when desktop CPUs run in excess of 10 billion instructions a second, per core, and interact with the real world (other machines) in very timing-dependent ways.

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

The only reason to ever use an interactive debugger is perhaps if your problem is not that your program is misbehaving but that you simply don't understand how a programming language works.

[MK14]

I don't understand what is the best way to debug c code. What is your general approach for any compiler to c debug code ?

Should I set a breakpoint after pressing the debug button or set a break point first and then press the debug button?

There is step in and step out button while debugging how to use it? should i stop the program then continue press step in button

Ignoring the politics, of if and when, debuggers should be used, here is an example of how to use a debugger.

Let's say you want to produce a list of prime numbers, and at some point, it starts outputting obviously wrong, negative numbers.  (Assuming your debugger has this feature), you can set the breakpoint, to be a conditional breakpoint, set to activate when prime < 0.

Run it, until the conditional breakpoint stops the code, then hopefully, by looking around the code (and variables values), can help you see what has gone wrong.  If not.  You can look at what the loop counter was set to when the prime variable became negative.  Let's say it had reached 10,001 when the apparent bug, occurs and primes suddenly became negative.

You can now set the conditional breakpoint, to be loop_counter >= 9,998.  Restart form the beginning and let it run, then use the single step features, to watch the prime variable become negative, and then hopefully see what has gone wrong.

It can get very fiddly and time-consuming, continually pressing the single-step (into) feature.  So, if you see it call a function, which you think is 100% perfect, and completely unrelated to the likely bug you're trying to find.  You can use the step-over feature, to keep in the same place, within the source file.  I.e. it will 'instantly' run the function, and return to the next line in the source code, without you needing to press the single-step (into), button/feature, perhaps hundreds of times.

N.B. If we really are talking about microcontrollers (the opening post seems to be rather short on specific details), then as previously suggested, printf, can be a better approach.
As well as printf, turning on or off, specific ports, possibly also connected to indicator LEDs.  Can also be helpful/necessary, especially if complicated peripheral setup conditions are really the issue.  It also means you can check the timings with a scope, as necessary.

As a rule of thumb, it might be best to think of using a debugger, as a tool of last resort.  Unless you are specifically trying to learn how to use them, or other possible reasons.

[T3sl4co1l]

I never use interactive debuggers.

It wasn't a good technique 40 years ago when CPUs ran 250 thousand instructions a second and I could press the "step" button realistically once or twice a second.  It's a MUCH WORSE technique today when desktop CPUs run in excess of 10 billion instructions a second, per core, and interact with the real world (other machines) in very timing-dependent ways.

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

The only reason to ever use an interactive debugger is perhaps if your problem is not that your program is misbehaving but that you simply don't understand how a programming language works.

Indeed there's even merit in just logging everything you can get your fingers on.  This isn't so feasible with limited RAM or slow comms on a real time embedded system say, but when you are able to collect a wide swath of things, take advantage of your other tools -- dump it into a database, or run filters or regex or whatever on it -- look for outliers, race conditions, faulty calculations, etc.  Examples: function inputs/parameters, steps of calculations, interrupts, main() loop stuff, etc.  Perhaps including timestamps, if you have a reasonably useful timer/clock to use that way.

With enough bandwidth, you could log some megabytes of data very quickly (seconds)... which is a terrifying amount for just a little MCU or something, but a pitiful embarrassment to any PC in the last couple decades.

Compare with some of the debugging/monitoring tools on PCs themselves -- Windows procmon for example, hooks all system calls -- thousands per second, and who cares, you've got GB of RAM to hold onto all that!

Tim

[nctnico]

I never use interactive debuggers.

It wasn't a good technique 40 years ago when CPUs ran 250 thousand instructions a second and I could press the "step" button realistically once or twice a second.  It's a MUCH WORSE technique today when desktop CPUs run in excess of 10 billion instructions a second, per core, and interact with the real world (other machines) in very timing-dependent ways.

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

The truth is in the middle. Printfs are definitely handy for printing status information. All my programs have those (and sometimes with the option to enable more debugging). Very valuable to get a good understanding what a program is doing in realtime.

However, interactive debugging is a good tool for finding elusive bugs like a pointer failure or a variable that somehow gets the wrong value. Being able to trace function calls back to their origin is very handy. Another good thing about interactive debugging is to verify code actually does what it is supposed to do. I use interactive debugging exclusively on a PC though when developing code that is supposed to run on a microcontroller later on.

[coppice]

I never use interactive debuggers.

I never use interactive debuggers for debugging desktop or server software. I do use them for small MCUs. There is so little potential for instrumenting the code in a very small machine, with limited I/O, that interactive debuggers can really help. Especially when the machine has some kind of build in logic analyser, so you can let a real time app run up an interesting point, and get a snapshot.

[IanB]

I never use interactive debuggers.

On the contrary, I do use interactive debuggers.

There are two useful aspects. One is that you can inspect the call stack, and can examine more about the program state than just the values of variables. The second is that depending on what you see, you can examine the contents of variables not only in the current routine, but also in any function higher up the stack.

A third use of debuggers is that you can interactively evaluate expressions in response to what you see, so you can check things out. You can also adjust things before proceeding, including perhaps modifying code ("edit and continue"), to see if your proposed improvement is going to work.

This is not to say that print statements and logging facilities are not also useful. They are. But every tool has a place, and an effective tool users knows when and where to use every tool in the tool kit.

[mikerj]

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

A professional uses whatever method is most appropriate rather than constraining themselves to one specific method.  e.g. printf can be an execution time hog that can change the behaviour of the system on slower micros.

[pcprogrammer]

A professional uses whatever method is most appropriate rather than constraining themselves to one specific method.  e.g. printf can be an execution time hog that can change the behaviour of the system on slower micros.

I second that.

For instance if you need to know timing of an interrupt without loosing a lot of cpu cycles toggle an io pin and hook up an oscilloscope to measure the time taken for the interrupt and the frequency of its occurrence.

When working on a MCU there is not always a serial connection to the PC, so printf won't do you any good. Or you are working bare metal and don't have printf because you don't want to use the libraries. Then Single Wire Debugging can be very helpful.

But you have to learn when to use which tool.

[Dadu@]

That's going to be IDE, toolchain, hardware debugger and device dependent.

Mostly when we start learning C programming, first of all we practice by installing compiler IDE tool on PC. In my case Hardware is PC and IDE is Visual Studio 2022. Visual studio also has a debugger and I can't get the idea how to use this debugger. I can see the variable value but I can't see the exact place where the variable will store. of course i can use print statement but i want to look on debugger window

[IanB]

I can see the variable value but I can't see the exact place where the variable will store.

Do you mean that you want to see the place in the code where the value gets modified? For that, you can put a debug watch on the variable, so that the debugger will stop when any change is made to the value.

Or do you mean you want to see the memory address of the variable? That is easy to to, but why would you want to? How would it be useful?

[Nominal Animal]

I don't understand what is the best way to debug c code. What is your general approach for any compiler to c debug code ?

Eyeball mark 2.  The mark 2 is the version that understands that developer intent and what the code actually does are two independent things, and whenever they diverge, you are likely to end up with a bug.

I've written so much C, and read/gone through even more C code, that I do not usually need to run the binary in a debugger; I can usually detect the bugs with eyeball mark 2 just fine.  For my own code, I do need to sleep between writing the code and going through it with debugging-eyes; otherwise the human visual filtering machinery will intervene making me miss some of the errors.

I do often use both fprintf(stderr, "describe key variable values\n", key variables) at key position in the code, for example at the beginning of a function I'm working on, and things like Graphviz DOT format output for trees and graphs, that I can then easily visualize using e.g. dot (from Graphviz).

When I learned to write unit "tests" first (to examine low-level solutions, for example abstract data type implementations – like disjoint sets – and their accessor functions), and then incorporate the implementations one by one into a working program, testing frequently and fixing all bugs and implementing all error checks before going on to the next step, and compiling at each step with warnings enabled, I suddenly became much more productive with much fewer bugs.

As an example, whenever I start writing a new command-line tool, I start by writing a working skeleton that does nothing:

Code: [Select]

#include <stdlib.h>

int main(int argc, char *argv[])
{
    return EXIT_SUCCESS;
}
Then I add command-line parsing, usually using getopt() (POSIX) or getopt_long() (GNU extension):

Code: [Select]

#define _POSIX_C_SOURCE  200809L
#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <locale.h>
#include <getopt.h>
#include <stdio.h>

static void usage(const char *arg0)
{
    fprintf(stderr, "\n");
    fprintf(stderr, "Usage: %s [ -h | --help ]\n", arg0);
    fprintf(stderr, "       %s FILE...\n", arg0);
    fprintf(stderr, "\n");
}

int main(int argc, char *argv[])
{
    static const struct option  longopts[] = {
        { .name = "help", .has_arg = 0, .flag = NULL, .val = 'h' },
        { 0 }
    };
    const char *arg0 = (argc && argv && argv[0] && argv[0][0]) ? argv[0] : "(this)";
    int  opt;

    if (!setlocale(LC_ALL, ""))
        fprintf(stderr, "Warning: Current locale is not supported.\n");

    while ((opt = getopt_long(argc, argv, "h", longopts, NULL)) != -1) {
        switch (opt) {

            case 'h':
                usage(arg0);
                return EXIT_SUCCESS;

            default: /* '?', and catch-all for unimplemented but specified options */
                usage(arg0);
                return EXIT_FAILURE;
        }
    }

    /* Require at least one non-option argument */
    if (optind >= argc) {
        usage(arg0);
        return EXIT_FAILURE;
    }

    /* argv[optind] through argv[argc-1] are the non-option arguments. */

    /* Debug output: */
    for (int i = optind; i < argc; i++)
        fprintf(stderr, "Argument %d (%d): \"%s\".\n", i - optind + 1, i, argv[i]);

    return EXIT_SUCCESS;
}
Because I like efficiency (being lazy!), I also tend to apply my standard Makefile skeleton.  Let's assume the above is named example.c, and we want to compile it to ./ex.  The needed Makefile is then

Code: [Select]

CC      := gcc
CFLAGS  := -Wall -Wextra -O2
LDFLAGS := -lm
PROGS   := ex

.PHONY: all clean

all: $(PROGS)

clean:
rm -f *.o $(PROGS)

%.o: %.c
$(CC) $(CFLAGS) -c $^

ex: example.o
$(CC) $(CFLAGS) $^ $(LDFLAGS) -o $@
although the indent has to be with Tab characters which this forum converts to spaces, so you need to run sed -e 's|^  *|\t|' -i Makefile to fix the indentation.  (You can run it safely at any point, and I recommend doing so after modifying the Makefile).  Then, to recompile the program from scratch, you only need to run make clean all.  (Plain make only rebuilds modified sources; here, only if example.c has been modified.)

This way, whenever I write additional code, I already know that any compilation error is due to the code I've added since the latest successful build.  Before adding more code, I fix those errors.  It saves a LOT of time.

If I am not sure yet what data structure I should use, I create a separate minimal test program(s), and implement the data structure there.  For trees and graphs, I often generate random data, and output the tree or graph in DOT format, using a safe tree/graph traversal (usually using a flag, "already described" in each node) to avoid getting caught in a traversal loop, and visually examine a few dozen trees/graphs to see it works.  I then create the pathological cases (like what happens if the input is the same, in the worst possible order, and so on), and verify my code won't get confused.

When I've found a thing that works, I then transplant the needed code to my main project, and recompile and check.  Again, the key is implementing and verifying each part, before advancing to the next (writing any additional code), so that the amount of code one needs to check stays minimal.

I can categorically say that writing a full program without compiling it at any point, is the wrong approach for me at least.  It wastes time, because I'm not perfect.  If I leverage the compiler and especially its capability to warn about suspicious code, I don't need to worry much about typos – I do rely on man pages instead of memorizing standard C and POSIX C interfaces –, and concentrate on the more important things like the overall design and algorithms, I am both faster and generate fewer bugs.  At this point, I'm pretty sure I've written over half a million lines of C, although only a fraction of that as paid work.  (But I have worked on and provided bug-fixing patches to quite a few open source projects, including to the Linux kernel, the GNU standard library, and the GCC compiler suite.)

[hans]

Don't feel bad to use a debugger. Printfs are an okay way to move forward.. it works.. having continuous trace logs of your program is useful for postmortems (for example if your program suddenly breaks, or only breaks once a week).

But beyond that scope, if you're using printfs to check where the program currently (still is), what values the local variables have, and then after verifying you remove it straight away.. we have modern GUI's for that, and they are called debuggers. Don't make people convince you it's a weakness to use them. It's an ease of use thing in my opinion. Do whatever works easiest for you. If you want to set breakpoints before launching the program, or after, pick what you like. Maybe some tools have quirks that it only works one way or the other.. then learn that quirk and move on.

Now there are some things printfs and debuggers can't do.

For one, if you hit a debugger breakpoint then any real-time system will crash. Protocols are also real-time, such as USB or TCP/IP, as the other side will eventually hit a timeout and drop the connection. This is less of a problem on PC debugging where you're working in "user land". The OS will, in the background, will service keep-alive stuff on an USB or TCP/IP level, so it's unlikely to time out there. But in embedded, especially with machinery equipment that is continuously processing values to stop a robot arm in time, I wouldn't want my code to hit any breakpoint.

Still that doesn't make a debugger connection useless. A debugger tool is so much more than just a code execution flow visualizer. On a lower level, JTAG/SWD debuggers read/write CPU registers and memory content. Accessing CPU data typically requires it to be halted, so in real-time we don't want that. But memory poking is still possible and can be useful to make low or 'zero' overhead printf tracers or data loggers. For examples you can take a look at e.g. Segger RTT or SystemView. Both allow data transport over the same interface as is used for programming. If you combine this with host-based printf formatting, then you can have rich a printf tracing experience with just a few hundred bytes of library code on the MCU, a few dozen CPU cycles per "printf" (copy straight into SRAM FIFO, where the host software will collect the data)... all running on the same few wires as your programming interface.

Finally, if you can afford to pause your program, then a debugger can also be a cheap profiling tool. If you want to profile a program without bloating the executable with instrumentation code, you can use a sampling based approach. There may be nice automated tools for that, but chances are.. those are desktop based, perhaps clunky to use or paid software. Instead, you can also do the sampling yourself by pausing-resuming the program a couple of times. Eventually you can get a pretty good idea what code is eating most of your CPU time, as you'll have a good chance to hit it most of the time.

I use both printf and debuggers off and on. Both have their strengths, and it's best to use them for that.

[nctnico]

One step further: what is very helpfull is to have counters in software. All my embedded projects have a command line interface that allows to query the status of various modules, do diagnostics (self test) and counters. For example: number of interrupts, number of characters / packets received / transmitted, number of errors. This is extremely handy to do remote diagnostics; having this feature saved huge amounts of time on multiple occasions. Some of my customers even use it to collect data for quality control purposes.

[jpanhalt]

I code in Assembly (MPASM), but my suggestion is not specific for that language.  Microchip gives 2 choices: a software simulation and hardware simulation with a limited number of breakpoints.  Software is good at finding stupid substitutions/typos, e.g., movlw v. movfw instructions.  Hardware is more useful with some peripherals (e.g, LCD's).  I add a 3rd type.  Flash an LED.  I can put a little routine anywhere I want and am basically only limited only by the number of "free" pins I may have available.

[T3sl4co1l]

Some incidental features on newer MCUs are nice.  I did a cascade of PID loops recently and dumped the inner loop's output to a spare DAC channel -- DACs used to be quite rare on MCUs.  I can compensate the loop by updating parameters from debug console, and read the result immediately on the scope.

(I mostly use a debug console with some very rudimentary memory inspection/update commands, and various commands customized for the task, e.g. demoing a new peripheral configuration or algorithm or whatever. See console and commands files here: https://github.com/T3sl4co1l/Reverb )

Tim

[nctnico]

Some incidental features on newer MCUs are nice.  I did a cascade of PID loops recently and dumped the inner loop's output to a spare DAC channel -- DACs used to be quite rare on MCUs.  I can compensate the loop by updating parameters from debug console, and read the result immediately on the scope.

A PWM channel also does fine for that purpose! Having an oscilloscope that can do filtering is a big plus.

[peter-h]

The answer to the OP depends, as some have pointed out.

As nctnico says both breakpoints (with single stepping) and printfs have their uses. Both have limitations.

The former stops your program running (the CPU is basically stopped) but breakpoints are great for checking algorithms etc and basic code function, plus they are great for picking up elusive/rare faults, and then you get a stack trace which is usually useful. But you have to connect the debugger so your board needs to have the connector on it for that.

The latter doesn't stop the program running but will tend to introduce a long delay (printf is a huge chunk of code, even if the entire output string goes quickly into an interrupt driven TX buffer so you are not e.g. serial port baud rate limited; in this context people pay for debuggers which have a fast debugging port feature). And this method can be used without a debugger, via a serial port, USB VCP, ethernet (http server etc).

[brucehoult]

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

A professional uses whatever method is most appropriate rather than constraining themselves to one specific method.  e.g. printf can be an execution time hog that can change the behaviour of the system on slower micros.

If you can't afford the timing changes from a printf() -- and it doesn't have to literally be printf(), it can be as simple as a unique byte put in a buffer or toggling pins on a GPIO port -- then how can you afford a breakpoint?

[brucehoult]

The latter doesn't stop the program running but will tend to introduce a long delay (printf is a huge chunk of code, even if the entire output string goes quickly into an interrupt driven TX buffer so you are not e.g. serial port baud rate limited

You don't have to do the actual formatting on the microcontroller!

The easiest quick&dirty thing is to link in a custom printf() function that doesn't actually format but just writes its arguments in binary to the output buffer, the first of those being the address of the format string.

Host-based software can then easily look up the format string in the ELF file and apply it to the arguments.

The only tricky thing about this is knowing how many arguments there are. You could just always dump 3 or 4 (especially if you write your code to limit the maximum number), or add a sentinel value to each call, or make printf1(), printf(2), printf3() etc functions that write a byte to say how many arguments there are. On host builds they can just map to normal printf().

More complex methods can allocate enums for each format string, either manually or using a source code pre-processing technique -- and not have the strings present in the microcontroller binary at all.

[hans]

I agree that if printf breaks your system, then a breakpoint will do for sure. But with a breakpoint, I've pretty much accepted that this will happen and the system may need a reset. With printf it can be more subtle. For example, let's have a spinlock that's polling a status register very fast. There is some problem with this code, because sometimes this status polling says the DUT reports a reset state when it's between device switching modes (and let's assume that this occurrence puts your driver stack in a fault mode).

 So you add a printf() for the status data. But as you do, you're changing the sampling rate of the status register significantly, and with that also decrease the chance of hitting the problem. Of course there are multiple ways of debugging further. You could only printf changed values. You could add a conditional statement to begin with. You could bitmask a certain state and write it's value onto a GPIO, which could be used for o'scope/LA triggering as well. If you have a good trigger on the LA, maybe you don't even need a printf at all because you directly probe the interface bus? (or spit out status bytes on a max. speed SPI bus, instead of printf tracing them) etc.

Trying to solve problems with conditional breakpoints is actually quite slow. AFAIK the debugger will set a regular hardware breakpoint, and once that's hit will evaluate a certain C expression before it exposes the breakpoint-hit to the user. Under the hood it has to read some CPU registers or data that it's a PC-software breakpoint with long round trip times. Not ideal to debug time sensitive problems..

-----
Concerning host based printf.. that's exactly what I did. I assigned unique file IDs to all source files. I format the 16-bit file ID with a 16-bit line number to create an unique printf ID. This ID is sent to the host, which can then parse it and grab the appropriate format string from the ELF file (you can also store the strings in a debug section) or directly from source.
Sending data arguments quickly is quite easy in C++ with a template. That gives full type information at compile time of the argument chain. It only generates the minimal amount of code needed to send those arguments to the host. The host should consume the incoming data bytes in a correct way, similar to a printf, to not over or underrun the data stream (e.g. don't use %lX for a 16-bit type).

[pcprogrammer]

Trying to solve problems with conditional breakpoints is actually quite slow. AFAIK the debugger will set a regular hardware breakpoint, and once that's hit will evaluate a certain C expression before it exposes the breakpoint-hit to the user. Under the hood it has to read some CPU registers or data that it's a PC-software breakpoint with long round trip times. Not ideal to debug time sensitive problems..

This is certainly the case when working on an USB driver in a MCU. You cant stop it because then the host times out and it does not work anymore

[Psi]

You tend to find engineers fall into two groups. Those who use and trust the debugger and those who don't.

Engineers who don't use/trust the debugger very much tend to send debug messages over UART, or flash an LED, or toggle GPIOs while looking at scope/logic_analyser etc.

It's really up to personal preference and which you find work better for you. Most engineers will use both but favor one much more than the other.

The most annoying thing about using a debugger is those situations where the problem goes away as soon as you enable the debugger and try and catch it. 

When using a debugger there's quite a few traps for young players. Trying to debug with too much optimization enabled etc..
So the debugger does have its own learning curve to it. Also the debugger for every CPU type will have its own quirks and traps that you will discover.
That said, MCU debuggers have gotten a lot better than they used to be.

[T3sl4co1l]

Optimization itself is a matter of preference.  One can argue the validity or merits of the approaches, but it's largely irrelevant these days, what with embedded CPUs of various capability being essentially equivalent in price, and that price itself being minor compared to development time/cost (and project timeline / time to market).

There are two kinds; and probably they correlate some with the debugging dichotomy above?

One is to simply write what seems right, what works.  Keep it at -O0 or at worst -O1, because the compiler ""introduces bugs"" and ""breaks working code"".  And we have plenty of CPU power to deal with it, who cares.  This is more common in "it has to be right", quantitative or safety oriented programming I think.  (Related, there's also hardware approaches, like redundant or lockstep cores -- another method very wasteful of CPU resources, but in a different way, and for different purposes.)

The other is to study and understand the C language itself, in great depth, so as not to be afraid of what the optimizer can do.  The compiler can't discover your intent from what you've written, if you've written it ambiguously with respect to your purpose.  Think of the optimizer as lawyer-gaming your code, trying to find outs that generate minimal machine code (not necessarily code per se, but whatever the optimization goal is set to, usually speed or size).  Think about what you write, in every possible way, game it out yourself and figure if one of those is broken or not.  Probably, this kind of programmer will produce uglier code -- because it's more closely tailored to the ugly intricacies of the C language itself, e.g. using more specifiers like const and volatile -- and probably there's more inspection of machine code going on, so, requiring knowledge of the machine ISA, and some tweaking of statements to try and generate better output in the first place.

The first is likely to be cheaper in development, if perhaps more expensive as a recurring cost (more powerful CPUs); it's also perhaps more portable, using fewer nuances tailored to the platform it was developed on.  (Mind, the latter approach can be just as portable, depending on how one does it -- sticking to strict language features, not platform dependencies or tall stacks of #defines.)

One can also -- and should -- run unit tests, from either class, though perhaps these are more likely in the latter than former?  If it passes the test, and the test has complete coverage, then it doesn't matter how casual and idiomatic, or nuanced and cryptic, the source is.  The challenge of course is recognizing when edge cases are desired normal behavior, irrelevant side products, or program-breaking bugs.  (Often, tests get developed from whole lists of issues, accumulated over years or decades of bug fixes; good tests can be very hard won indeed!)

I don't know that there's much TDD (test driven development) in the embedded space though; it's hard, with so many platform-specific dependencies.  The few things you can, you should though.

Tim

[nigelwright7557]

The best approach is incremental design.
Start off with a small piece of code and convince yourself that is working before adding a little more etc.
Use debugger to view the result of the code.
I usually add an LED to embedded systems to help with debugging.

[nctnico]

Incremental design should go without saying 
You can't make a pie by starting with the toppings.
More often than not you'll find unexpected quirks coming from hardware / sensors that need special attention by the part that implements the logic.

For example: recently I did a project involving radars that can track people. Ofcourse the supplier has very nice videos showing that these radars track persons perfectly but the reality is that the radars do lose a person rather quickly so this needed some additional work in the software to do some matching.

[T3sl4co1l]

Well, both.  You also can't make a pie by starting with fruit, sugar, flour, etc.  Hm... admittedly, you can make a lot of tasty or at least nutritious things with those ingredients in fairly random amounts and order... but, needless to say programming is a bit more complicated (and picky!) than pastry.   So to make a..."pie", you need to start with the notion of making a pie, and then how to assemble it, and so on.  Hence: top-down, bottom-up design.

The solution for the top-level problem needs to be informed by knowledge (or if lacking, research -- early development or PoC) from the bottom -- which peripherals you can use, how much CPU they'll take (ISRs, etc.), how much logic or other things you'll need to fill in the gaps (HW vs. SW support for various functions, housekeeping, etc.; stuff like buffering or DMA), and then overall program code on top (the "draw the rest of the fucking owl" part ).  Do some top level planning, bottom level implementation, and gradually fill in between.

Which is also suggestive of a layered architecture: do drivers, with whatever degree of abstraction you like/need, at the bottom; then layers of "kernel", application, etc. on top of that.

Which, most embedded stuff probably doesn't go very deep, but when it does, you'll likely choose something to help manage that; RTOS or Linux say.  Ingredients in the solution will naturally reflect the structure and scale of the solution.

Tim

[PlainName]

I don't understand what is the best way to debug c code.

Same way you debug anything. The tool you use to zero in on the cause comes later.

What is your general approach for any compiler to c debug code ?

First, you gotta figure out what the problem is you're wanting to solve. Sounds the easy part but typically all you know is that something that's expected to happen doesn't, or something unexpected happens. But what, exactly, is that? Suppose your LCD displays "42" instead of the expected "On" or something - what's actually gone wrong there?

If you have access to a single-stepping debugger the temptation is to set up a breakpoint just before the print function and then, when the code halts, check the variables or strings. Maybe if you don't have that kind of debugger you'll just shove a load of printf statements around to try and capture the thing going wrong.

I would suggest those are the wrong approaches at this point. Instead, you should think about what the fault could be. Does it happen all the time or just when certain values are displayed? Maybe when a certain GPIO pin is toggled? A timer rolls over? I would usually look closely at what else is happening and just collect a lot of data. There might be other things that aren't actually wrong but, you know, not quite right. Perhaps a glitch just before something is displayed, or something taking longer (or shorter) than it should. Forget about whether it's related to the fault you can see and just collect info.

Then you play with that in your mind and figure out what kind of thing could affect all of that info you have. Clearly, the more info you have the better chance of it all pointing to something common, or a likely suspect. Now you've figured the likely cause you work out how you can prove that using the tools at your disposal. Clearly, some tools are easier to use than others, but a simple LED can be as informative as a breakpoint or printf if you know what you want it to say.

Sounds a bit long-winded, but the technique is pretty simple, reasonably quick and should apply to most fault-finding scenarios.

[As an example, our hot water overflow is currently overflowing dramatically and the cause could be one of two issue - a cheap one replacing the thermostat or an expensive one replacing the tank. I figure if it's the expensive one (heating coil corroded leading to boiler circuit leaking into hot water tank) then using the immersion heater instead of the gas boiler will not provoke the issue, and sure enough it doesn't . There, collecting info and applying a simple test shows the cause where the equivalent of single-stepping would be to drain and dismantle the tank, or change the thermostat just to see.]

[RJSV]

Yeah mk14 has it right, (back a couple of posts). I would add to that you could set a breakpoint on some error catching code you've included; that is somewhat equiv to using printf(). But the context, in real time interactions, with serial ports, etc. might preclude just general use.
   I think strong point, here, is when you can get a really specific case, where some time-out or other process isn't behaving as predicted.  THEN it's a good time to get onto post your very specific confusion / unexpected result.  I think, rather than a general 'this before that' question.  Those, you could try guess which is best order.  Try both ways.  (But my guess is you would enter your debugger first, then set breakpoints).  Usually it's not a general thing, but one specific breakpoint, as you focus on one specific area of problem.  If it's 3 different breakpoints that's maybe too 'general' an approach...if that makes sense.

[Siwastaja]

1)
Read and UNDERSTAND the code, in your head. If you can't understand the code you wrote yourself, re-write it until you can. This reduces number of bugs to manageable level. This is important because each bug might take hours to find and fix, on average.

2)
Add instrumentation. Simplest is so-called printf debugging, or more properly, if(logging_level > 42) fprintf(log_file, "we are at x, i=%d and asdf=%08x", i, asdf);
For more timing critical thing, you can do like
typedef struct
{
    int16_t current:
    uint16_t voltage;
    uint8_t return_code;
    uint8_t next_state;
} log_elem_t;

log_elem_t log[LOG_LEN];

Just fill the log somewhere (where you cannot print due to timing etc) and dump it out later. This continues working in production, unlike attaching JTAG based debugger on lab table!

Especially useful: if you have a state machine: in a function which changes state, log previous and next state along with some other variables. You can also record some free-running counter to get idea of time.

3) Add error checks, range checks, sanity checks - basically asserts. Even in embedded:
if(x < 3 || x > 7) error(6); // x should be within this range because of asdfg, and latter code assumes so.

This works in production, too! Make the LED blink 6 times and ask the customer to count the blinks. Catching wrong function argument by a clear error message only takes a minute; having to debug some overindexing issue or similar can take days.

All of this is not that much work, and once you spend the few hours initially, it sticks to your codebase and continues being helpful when you least expect having any problem.

[Picuino]

I never use interactive debuggers.

Me neither.
With microcontrollers I use a combination of:

   * printf()

   * switch_output_pin() + oscilloscope or multimeter

The second method is better for processes that are fast and may be affected by printf delays.
For example you can switch the output of a pin on the input and on the output of an interrupt and thus clearly see the execution time of the interrupt and the total time it takes up.

[Siwastaja]

   * switch_output_pin() + oscilloscope or multimeter

Also, SPARE_UART->TXDR = 0x57;

This gives you both timing (nearly if not as accurate as GPIO), and any 8-bit tag or other data attached to it, which you can see on the oscilloscope decode function, in sync with any actual analog measurements you are doing off the same system. And it's still just a few lines of code and one-two register write(s) which is unlikely to mess up any critical timing.

If I have a spare LED and some complicated ISR, I routinely just enable the LED at the beginning and disable at exit. This way, it's obvious to the eye if the ISR ever gets stuck (during early development), and you can eyeball the % of CPU time spent in ISR from the LED brightness. Then you can just probe the LED pin with oscilloscope probe tip to get exact timing without having to remember where you did route the signal, because it's just the LED.

I like simply but powerful small tricks like this. Variations are endless, all you need is a bit of creativity.

[DiTBho]

The truth is in the middle. Printfs are definitely handy for printing status information.

Yup. I don't need printfs, my ICE uses communication channels and inspection points.

[Siwastaja]

+ do unit testing of state machines and algorithms on PC, instead of just writing long pieces of complex code for MCU target only to be tested on that tiny CPU. On PC, you can just loop through all interesting combinations and log the output and decision process to a csv file, for example.

[rstofer]

I never use interactive debuggers.

It wasn't a good technique 40 years ago when CPUs ran 250 thousand instructions a second and I could press the "step" button realistically once or twice a second.  It's a MUCH WORSE technique today when desktop CPUs run in excess of 10 billion instructions a second, per core, and interact with the real world (other machines) in very timing-dependent ways.

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

The only reason to ever use an interactive debugger is perhaps if your problem is not that your program is misbehaving but that you simply don't understand how a programming language works.

I remember using Digital Research's DDT debugger way back around '78.  I sure won't be doing it again.  As you said, use printf() for tracing program flow.

When I start a uC project, the very first thing I do is get a UART running.  That is always Step 1.  Then I go to "The C Programming Language, First Edition" (Kernighan and Ritchie) and rip off the conversion functions like itoa().  These are simple functions and don't require a heap.  I have a collection of these functions for displaying nibbles, bytes, shorts and words with and without the leading 0x for hex values.  Combined with a puts() function, I can now create all the debug messages I need and I don't need to use the C library and its string functions which likely require a heap.  I avoid a heap at all costs.  Note that I don't actually have, or use, a real printf() function.  I just build strings per K&R and use puts() to display them..

I can surround the debug print statements with #ifdef DEBUG ... #endif to disable the printing while leaving the code intact.  Or I can remove it but sure enough, I'll find I still need it! 

[PlainName]

To answer the specific debugger use questions (which are generic answers, modified by your specific tools):

Should I set a breakpoint after pressing the debug button or set a break point first and then press the debug button?

Typically you will know where you want to start stepping through your code, or where you want to stop to examine, say, variables. So you would set your breakpoint and then start the debugger. The downside to this is where you want to stop on some commonly called function, which may be called 200 times but different processes before the time when you actually want to debug it. In this case you would run the debugger up until close to the right time (or set a breakpoint in the process you know will call your suspect function just before) and the halt, set the breakpoint, continue.

There is step in and step out button while debugging how to use it? should i stop the program then continue press step in button

Step in typically enters the function about to be called and then continues stepping through that function. There should be a step over buttong too, which just runs until the called function returns (that is, it steps through the code at the current level and doesn't step through any called functions).

Step out is used when you're stepping through a function and decide you don't want to any more (that is, go back to whatever called this one). You could keep stepping until the current function returns, but if there are a lot of steps to go then step out shortcuts them all and skips to just after the return.

[peter-h]

Quite often, the only way to solve something is to waggle a GPIO pin and watch it with a scope. In fact I have a scope (Lecroy 3034) permanently wired to my target for this purpose. There is an LED on that GPIO but if it lights up only for 5us, you won't see it

I remember using Digital Research's DDT debugger way back around '78.

I implemented CP/M 2.2 on several targets and never used that horrid tool Back then, it was usually a debug string sent to a serial port. The really key thing was having a decent size tx buffer on it so the debug statement didn't hold up the target.

[T3sl4co1l]

Oh and not to be exclusive -- there's always both.

This's been alluded to, in earlier replies I think, but yet another example is adding check code in your program, which normally falls through so has very little effect on program execution (even only a small cost to execution speed), but within which the code can set a breakpoint so not only can you "printf()" the condition but you can sniff around with the debugger at the same time.  Maybe a combination of approaches might be used.

Example, say you have a nested loop that's iterating over data, and in the output you can clearly see it's made an erroneous translation or whatever, but you can't really be quite sure where/when the error occurred.  So you log output from earlier and earlier steps in the computation, add asserts, range checks, etc., anything to help figure out where the error occurred.  You start with a big heap of data, you might produce even more data as you're collecting additional evidence at first; gradually, you close in on the problem.  Then you are able to reduce the pile of evidence to the smoking gun, one single statement, or index or pass or whatever, which was meant to perform one operation but did something else (overflow or rounding, say); or, say a function got sent erroneous data, or ran out of data, and sent back an -- entirely reasonable return value, but one you merely didn't anticipate encountering in the process, so you forgot to handle that edge case.  All sorts of things.

Well, you might approach such a problem by short-circuiting part of the process, tracing it up until you find the first inconsistency.  You're momentarily making the output worse, less correct, outputting intermediate steps -- but as long as you know what the data are supposed to be, you can find the discrepancy.  This might be by printf() logging, might be dumping to an array, might be stepping a debugger -- maybe all the above.

Case in point, while I don't use debuggers for embedded -- mainly because I don't have any for my current platform -- I try to make good use of them elsewhere, like, I'm fond of using JS for side tools -- calculators, image processing, code generation, that sort of thing.  And often I make use of all the above.  Values can be logged to console; arrays can be queried with various sorting, regex, etc. functions; objects can be passed through partial steps by copy-pasting code into the interpreter (immediate) window, etc.  And object views are all handy, see all the properties, arrays, etc. of variables in current scope.  Best of all, you literally don't have to go anywhere to do this: JS is all right there in your browser, just hit F12, it's literally right here right now!

Tim

[Nominal Animal]

To be clear, I agree with others above: just because I don't often use debuggers, does not mean they aren't useful.

If one uses gdb, then one can even write Python pretty-printers (example includes a Python function to use Graphviz to display an in-memory binary tree at debug time), which for sure can be handy.

It's just that there are many, many different ways to debug code, and they all start at understanding the intent of the developer separately from what the code actually does.  This is why I personally so emphasize Eyeballs mark II.  (And, also the reason why I tell new programmers to learn how to write comments that describe their intent, instead of what the code actually does.  I only wish I had learned to do so too when I learned to write code; it is darned difficult and slow to learn to do it afterwards... just like security, it just isn't something you can bolt on afterwards; it has to be designed in.)

Personally, I consider basically all of the above as useful suggestions.  As each human is subtly different, the "optimum" debugging tool varies from person to person –– and really, from target to target and project to project! ––, so try several, and find whatever seems to work best for you.  Just don't get stuck on one, and periodically, try the different ones to see if they then work better.  (That's what I did at the above GDB link for myself, some eight years ago.)

[rstofer]

For uC and especially for FPGA work, wiggle a gaggle of pins and catch them with a Logic Analyzer.

For one FPGA project, I threaded a 32 bit debug bus through most of the components and commented it out where is wasn't used.  Of course I was still limited to a single driving signal but that worked out fine.

It is, however, problematic to use 1-hot encoding of an FSM with over 100 states and try to capture state information.  Instead, just set the debug bus to some unique value in each state (or even a subset of states).

One trick:  When you write the C code, be thinking about how you plan to debug it.  Build the debug capability in from the start.

[Nominal Animal]

For uC and especially for FPGA work, wiggle a gaggle of pins and catch them with a Logic Analyzer.

As an aside, the 8-channel 24MHz logic analyzers (Saleae logic clones) you can get for ~ $10/10€ off eBay, work really well with PulseView from the sigrok suite (free and open source, binaries available for Windows, Mac OS, and Linux; although I recommend Linux users to use the one in their standard repositories if possible).  The logic analyzer itself is a direct implementation of the Cypress FX2LP USB chip datasheet, and PulseView/Sigrok uses its own FX2LP firmware, so if you rip out any sticker referring to Saleae Logic, you will not even in principle hurt anyones intellectual property.  Pulseview has many decoders built-in, and you can do your own if you want.  It is limited in its sampling frequency, but if your signal clock is not more than a couple of MHz, these cheap devices will serve you well.

[gnuarm]

I don't understand what is the best way to debug c code. What is your general approach for any compiler to c debug code ?

Should I set a breakpoint after pressing the debug button or set a break point first and then press the debug button?

There is step in and step out button while debugging how to use it? should i stop the program then continue press step in button

The way I debug C code is to write Forth code.  Forth is good for breaking your code into small pieces which can be tested interactively and separately.  So you make building blocks, each of which is tested, and construct larger and larger tested blocks until you've created your application. 

This is much harder to do in many languages, like C.  In C, people typically write a large chunk of code, then look for tools to make it easier to deal with the complexity. 

Forth is also very good for small MCUs, because the code tends to be more compact from code factoring and reuse within the application.

I've never needed a debugger to deal with my Forth programs.  A logic analyzer can come in handy when dealing with the hardware. 

[Bassman59]

For uC and especially for FPGA work, wiggle a gaggle of pins and catch them with a Logic Analyzer.

As an aside, the 8-channel 24MHz logic analyzers (Saleae logic clones) you can get for ~ $10/10€ off eBay, work really well with PulseView from the sigrok suite (free and open source, binaries available for Windows, Mac OS, and Linux; although I recommend Linux users to use the one in their standard repositories if possible).  The logic analyzer itself is a direct implementation of the Cypress FX2LP USB chip datasheet, and PulseView/Sigrok uses its own FX2LP firmware, so if you rip out any sticker referring to Saleae Logic, you will not even in principle hurt anyones intellectual property.  Pulseview has many decoders built-in, and you can do your own if you want.  It is limited in its sampling frequency, but if your signal clock is not more than a couple of MHz, these cheap devices will serve you well.

In the FPGA, attach an embedded logic analyzer to the signals in question. There's no need to bring them out to pins. Well, unless you're in a small FPGA and you've used up all of your block RAM, but ... then you've probably already used up all of your pins.

And all that said, any unused pin in the design really should be accessible on the board in some manner for use as a debug point, or at least a place to solder on a bodge wire because you missed something.

[rstofer]

In the FPGA, attach an embedded logic analyzer to the signals in question. There's no need to bring them out to pins. Well, unless you're in a small FPGA and you've used up all of your block RAM, but ... then you've probably already used up all of your pins.

And all that said, any unused pin in the design really should be accessible on the board in some manner for use as a debug point, or at least a place to solder on a bodge wire because you missed something.

That Integrated Logic Analyzer is a neat feature of Xilinx Vivado that wasn't available with ISE.  It's great and, as you point out, it doesn't require hardware resources (pins).  I need to play with it some more...

[Mechatrommer]

The only reason to ever use an interactive debugger is perhaps if your problem is not that your program is misbehaving but that you simply don't understand how a programming language works.

how is that possible? you make a conclusion about what an interactive debugger is if you....

never use interactive debuggers.

? i lived with VB6's (and to some extend MSVC++) interactive debugger for decades, that makes me still dont understand how a programming language works? please explain why i still dont understand? (i may not know what i dont yet know) but in embedded field i have to improvise my debugging technique because the IDE's for embedded MCU's dont provide me interactive debugging as good as VB6, not even close, in ALL of them.

[brucehoult]

The only reason to ever use an interactive debugger is perhaps if your problem is not that your program is misbehaving but that you simply don't understand how a programming language works.

how is that possible? you make a conclusion about what an interactive debugger is if you....

never use interactive debuggers.

? i lived with VB6's (and to some extend MSVC++) interactive debugger for decades, that makes me still dont understand how a programming language works? but in embedded field i have to improvise my debugging technique and the IDE's for embedded MCU's dont provide me interactive debugging as good as VB6, not even close.

Almost never.

I used them in the 80s when I was a n00b. I occasionally use one when learning a new language that is quite different to ones I already know. I use them when a customer sends a core file from a crashed program. Sometimes I use one to binary patch a running program.

[nctnico]

In the FPGA, attach an embedded logic analyzer to the signals in question. There's no need to bring them out to pins. Well, unless you're in a small FPGA and you've used up all of your block RAM, but ... then you've probably already used up all of your pins.

And all that said, any unused pin in the design really should be accessible on the board in some manner for use as a debug point, or at least a place to solder on a bodge wire because you missed something.

That Integrated Logic Analyzer is a neat feature of Xilinx Vivado that wasn't available with ISE.  It's great and, as you point out, it doesn't require hardware resources (pins).  I need to play with it some more...

Xilinx' internal logic analyser has been around for decades. It is available to use with ISE but I don't recall whether it is a paid option or not. The biggest downside is that the internal logic analyser eats away precious internal memory and there is little of it so the trace isn't very deep. In general I try to bring a few pins to the outside to attach an external logic analyser. Many of my more complex FPGA designs have additional logic to bring debugging information (status of pins, statemachine states) to the outside by using a mux that can de set to bring out a specific set of signals. By having no timing constraints on these signals (it is easy to see what is what), this method doesn't consume much in terms of scarse resources.

[Mechatrommer]

Almost never...

so either:
1) your programs are too simple to not require in depth debugging...
2) or you are genius your programs just work the first time you code them however long they are.
3) or you are diligent type (like most people in pre 70's before computer exist) who do the mental program flow in your head by reading your code line by line, write side notes if neccesary, correct them and redo, until it works?

[Siwastaja]

1) your programs are too simple to not require in depth debugging...

In-depth debugging of complex programs is real PITA with general purpose debuggers (single-stepping, breakpoints, watch lists).

Complex programs have complex interactions and state history, which absolutely requires built-in instrumentation and logging - unless you are of course perfect and never make any mistakes*.

Single-stepping in a debugger is 99% used by beginners who totally struggle understanding basic concepts of language and resort to this slow and painful process. Then they come on EEVBlog forum and claim to be world-class experts because they have 2 years of experience on working with STM32 or whatever.

Obviously, sometimes there are cases where adding a breakpoint is just the right thing to do, though.

*) which is a good target even if unachievable; Nominal A's remark of Eyeball mark II is excellent. By just spending 5 more minutes looking at some weird loop and unrolling it in your head, you can easily save hours of painful troubleshooting by never introducing the bug in the first place.

[brucehoult]

Almost never...

so either:
1) your programs are too simple to not require in depth debugging...
2) or you are genius your programs just work the first time you code them however long they are.
3) or you are diligent type (like most people in pre 70's before computer exist) who do the mental program flow in your head by reading your code line by line, write side notes if neccesary, correct them and redo, until it works?

You forgot

4) implement one tiny feature at a time, and then test it to make sure it works (and didn't break anything else) before moving on to the next tiny feature. Have a running program at all times, starting from HelloWorld.

[Mechatrommer]

man i feel bad that i just jumped in without reading the 2nd page

4) implement one tiny feature at a time, and then test it to make sure it works (and didn't break anything else) before moving on to the next tiny feature. Have a running program at all times, starting from HelloWorld.

a very slow process imho... i prefer code first and debug later. of course experience needed, that i already have few working modules/functions to reuse in a specific program that modules/functions will interact differently specific to the program needs. that "specific" part is what needs debugging. but then... "testing" a small unit/feature in earlier stage will requires "debugging" anyway. but granted, using different strategy other than interactive debugging.

[Mechatrommer]

Single-stepping in a debugger is 99% used by beginners who totally struggle understanding basic concepts of language and resort to this slow and painful process.

we single step (breakpoint, watch variables) at where the program is misbehaving, that we cant make out the reason why by simply thinking about it alone, not single step everything.

It's really up to personal preference and which you find work better for you. Most engineers will use both but favor one much more than the other.
The most annoying thing about using a debugger is those situations where the problem goes away as soon as you enable the debugger and try and catch it. 

this is what i figured when programming mcu world. it opened a new field of headacheness especially the... interrupt routines, and the ever changing registers state based on external conditions (pin change etc), not anything from the code, making interactive debugging technique being rendered almost useless in these systems. anyway, i dont usually use printf/uart technique because mostly i program tiny chip tiny flash, cant afford space to put printf/uart in there. i use sort of simple serial binary GPIO on off spitting technique to debug registers/variables values etc, but for interrupt routines, i dont find any debugging tools in IDE can solve anything about most of it, i have to imagine some "axioms" and device a code to trap something to prove or disprove the axioms ymmv.

[MK14]

You forgot

4) implement one tiny feature at a time, and then test it to make sure it works (and didn't break anything else) before moving on to the next tiny feature. Have a running program at all times, starting from HelloWorld.

There isn't one, perfectly right or wrong (one size fits all) way of sorting out debugging issues, with software development.  There are all sorts of ways, different developers, at different times, solve their issues.

It's like trying to argue, that your favorite brand/type of Coffee/Tea/Whatever, is the best one, and everyone should drink it.

Sometimes the test target hardware isn't available for a period of time, so (unless you want to add the time consuming step of creating emulators or simulators), most of the testing, can't occur, until long after the software has been written.

In some cases, you can knock together a development board and some bits and pieces, to try out the software early.  But a decision may be made, to just press on, and finish the software and leave testing until when the new populated PCBs, are ready.

[PlainName]

Single-stepping in a debugger is 99% used by beginners who totally struggle understanding basic concepts of language and resort to this slow and painful process.

I strongly disagree with that. If an ICE is available I would probably use that to examine state or data at a suspect point because it's quicker and simpler than inserting printf's and other similar debugging tools. Also doesn't affect the code - adding more strings and affecting the stack/heap isn't always innocuous.

Further, if I knew the exact problem I wouldn't need to debug, so a printf will either confirm or point to something else. Another round of inserting data and code, but with the ICE and I can check it out right there and then. And not have to do another recompile because I forgot the damn % in the printf. And then, after all that, remove the just inserted debugging code.

[free_electron]

in order of preference :
- a language that lets me interactively view and edit code / data without having to recompile or restart or dataloss , let's me alter the instruction pointer and add /remove / rerun lines of code. Something like visual basic.
- Hardware emulator with full control of all memory and registers that can trace at source level (we're talking bond-out processors that can go full speed and have dual ported memory so you do not stop or slow down the thing while running.
- an ICE that can do similar to a hardware emulator. Something like Realview ICE or Dstream on Arm. uses dedicate trace port on processor
- dualport ram with a backdoor
- Jlink or jlink style device that lets me explore all memory so i can prod around in there.
- scope / logic analyser , prefer a MSO (analog+digital)
- wiggling pins

printf ? i'll resign ! if you can't afford any / mix / all of the above and i have to waste my time with printf .. i'd rather be unemployed

[Siwastaja]

Single-stepping in a debugger is 99% used by beginners who totally struggle understanding basic concepts of language and resort to this slow and painful process.

I strongly disagree with that. If an ICE is available I would probably use that to examine state or data at a suspect point because it's quicker and simpler than inserting printf's and other similar debugging tools. Also doesn't affect the code - adding more strings and affecting the stack/heap isn't always innocuous.

Further, if I knew the exact problem I wouldn't need to debug, so a printf will either confirm or point to something else. Another round of inserting data and code, but with the ICE and I can check it out right there and then. And not have to do another recompile because I forgot the damn % in the printf. And then, after all that, remove the just inserted debugging code.

There is no need to disagree because you confirm what I say. Quick and easy checks - i.e., something "needed" most by beginners. For someone more experienced, number of these quick and easy checks go down (not to zero of course), and problems become more complex and more difficult, mostly.

Obviously "printf debugging" does not refer to printf itself. I use it in MCUs very little. If you don't like %formats, do not use it.

And "doesn't affect code" is 110% bullshit beyond being funny. Technically might be correct - does not affect code, given that you compile with same settings. Practically wrong because you don't do that - usually we compile with different settings (e.g., -O2 vs. -O1 -g). Also fundamentally wrong - affects peripheral state in difficult to predict ways and causes endless misery among beginners. Also affects timing. Instrumentation of the code itself also affects the code but in much more controllable and predictable way: for example, store a copy of UART data register in one place, instead of letting your ISR and debugger to race whoever reads it first (spuriously clearing flags as side effect).

[SiliconWizard]

Looks like one of those topics that are highly sensitive and triggers very opinionated reactions bordering flame wars. (Apparently, programming is an ideal target for those topics! I've rarely seen "general" programming topics not ending with fights.  Let's be happy though when there's no personal attacks or name calling. )

This is also a frequently discussed topic, so we're all likely to repeat ourselves a good deal.

I personally agree with Siwastaja and Bruce here. I haven't used an interactive debugger on embedded devices for literally ages, and use one very infrequently for "desktop" programming (like maybe a couple times a... year.)

Debugging in general, and not just for software, is all about 1/ observation and 2/ being methodical. There are tons of ways you can "observe" the behavior of running code. The "right" one to use all depends on the context. As to being methodical, interactive debugging rarely promotes it. But if or when a debugger is appropriate for observing a particular behavior that you can't otherwise pinpoint, sure, use one! I just tend to use those as a last resort rather than as first intention.

The (almost) only case in which I'll fire up a debugger like gdb is when the code is crashing. The debugger will locate the line where it happens which will save you time. Once it's located, you may or may not step through the code - depends on the context - but usually, just the backtrace and looking at variables/parameters is enough to proceed, and then you can quit the debugger and go from there.

Of course, some development methods make it less likely you'll run into intractable bugs requiring stepping frantically through code in a debbuger, such as testing relatively small parts of your code as you write it, and proceed incrementally, as Bruce suggested. It again can/has to be adapted to the particular project, your coding abilities and the complexity of the code. Unit testing at too fine a grain, or on trivial functions, is useless and silly: use common sense.

Another development "method" I routinely use (and which I regularly "promote") is parameter validation. Validate functions arguments. It's a tremendous help. Here again, use common sense though.

Tracing using some logging/printf/whatever is of course also useful and usually more productive than manually stepping through code.

While overly generalizing is rarely relevant, resorting to an interactive debugger instead of using one of the above methods (or others of your choice, by no means exhaustive) *can* be a sign that you favor "instant gratification", while using other methods often takes a bit more time upfront, but once done, that can (and usually is) a big time saver at the end of the day. Again not to generalize, but I would say that it is a bit akin to people who have a tendency to copy/paste code repeatedly to achieve a given task rather than pause for a moment and factor code so that the end result is much more maintainable, a lot less prone to errors and will save you a lot of time down the line. Yet some people, from my experience, tend to see this as a waste of time.

So, yeah. My 2 cents.

[nctnico]

I'd like to highlight 'Validate functions arguments'; (very good that SiliconWizard brought it up  ).
I'm doing this in my code at module interfaces (APIs or public members). This creates compartments where errors can not escape from. It makes software fail in a predictable way instead of having a huge oil stain caused by a domino effect from faulty results that travel through a whole bunch of software layers.

Another good tool to know is valgrind. It can do a detailed analysis of memory allocations & frees to check whether all the memory that gets allocated also gets freed. It is a massive help to find memory leaks.

[Bassman59]

In the FPGA, attach an embedded logic analyzer to the signals in question. There's no need to bring them out to pins. Well, unless you're in a small FPGA and you've used up all of your block RAM, but ... then you've probably already used up all of your pins.

And all that said, any unused pin in the design really should be accessible on the board in some manner for use as a debug point, or at least a place to solder on a bodge wire because you missed something.

That Integrated Logic Analyzer is a neat feature of Xilinx Vivado that wasn't available with ISE.  It's great and, as you point out, it doesn't require hardware resources (pins).  I need to play with it some more...

In ISE it was called ChipScope, and seriously it was the best bit of software Xilinx ever implemented. It worked as advertised, no muss, no fuss. Just run the Core Inserter, pick the signals to analyzer, let the fitter run, and off you go.

[PlainName]

There is no need to disagree because you confirm what I say. Quick and easy checks - i.e., something "needed" most by beginners.

OK, perhaps I misunderstood. The drift I got was that only beginners would use those tools; real programmers would pass 'em up (maybe they're too easy). I guess a bit like beginners would use VB whereas serious programmers wouldn't.

[Although, having said that, I once used a fantastic program written in VB. Unfortunately the developer saw the light and decided to rewrite it 'properly' in C#, and it's been pants every since ]

[PlainName]

Another development "method" I routinely use (and which I regularly "promote") is parameter validation. Validate functions arguments.

Yes, all that kind of stuff is good. But that isn't debugging as I gathered the OP wanted to know about - it's preventing the need to debug. I understood the question to mean: it's broken, what do I do now?

[nctnico]

Another development "method" I routinely use (and which I regularly "promote") is parameter validation. Validate functions arguments.

Yes, all that kind of stuff is good. But that isn't debugging as I gathered the OP wanted to know about - it's preventing the need to debug. I understood the question to mean: it's broken, what do I do now?

Learn 'design for testability' 

[MK14]

Another development "method" I routinely use (and which I regularly "promote") is parameter validation. Validate functions arguments.

Yes, all that kind of stuff is good. But that isn't debugging as I gathered the OP wanted to know about - it's preventing the need to debug. I understood the question to mean: it's broken, what do I do now?

No, it sure is part of testing, debugging and software quality assurance/validation.  Otherwise, the software could be silently broken in some places, which can bite you in the foot, if just left there, undiscovered.

[tellurium]

I'd like to highlight 'Validate functions arguments'; (very good that SiliconWizard brought it up  ).
I'm doing this in my code at module interfaces (APIs or public members). This creates compartments where errors can not escape from. It makes software fail in a predictable way instead of having a huge oil stain caused by a domino effect from faulty results that travel through a whole bunch of software layers.

Another good tool to know is valgrind. It can do a detailed analysis of memory allocations & frees to check whether all the memory that gets allocated also gets freed. It is a massive help to find memory leaks.

Params checking, valgrind, and testing are invaluable tools.
Another one worth mentioning, is  cheap to use but very effective: compilation flags. This article has a good overview and explanations: https://interrupt.memfault.com/blog/best-and-worst-gcc-clang-compiler-flags

For example, an (arguably) rarely used -g3 GCC/Clang flag pulls enum / macro definitions into the debugger interactive session.

[SiliconWizard]

Another development "method" I routinely use (and which I regularly "promote") is parameter validation. Validate functions arguments.

Yes, all that kind of stuff is good. But that isn't debugging as I gathered the OP wanted to know about - it's preventing the need to debug. I understood the question to mean: it's broken, what do I do now?

No, it sure is part of testing, debugging and software quality assurance/validation.  Otherwise, the software could be silently broken in some places, which can bite you in the foot, if just left there, undiscovered.

Yup. Some people even only do parameter validation for testing/debugging purposes instead of doing it at all times even in release mode. Often through the use of assert().

I personally do prefer to leave parameter validation everywhere it makes sense, at all times, and not just in "debug" mode. Hoare wrote:

"What would we think of a sailing enthusiast who wears his lifejacket when training on dry land, but takes it off as soon as he goes to sea?"

[free_electron]

so how do you debug a system that does not have a communication interface ? or when you are trying to make it work ?
you get a new processor and there are no libraries. Here is the register map and datasheet ...
your code compiles but all there is is silence. nothing comes out of the uart or usb or whatever...
what do you do then ? printf into the stetoscope? warm finger test ?

[MK14]

so how do you debug a system that does not have a communication interface ? or when you are trying to make it work ?
you get a new processor and there are no libraries. Here is the register map and datasheet ...
your code compiles but all there is is silence. nothing comes out of the uart or usb or whatever...
what do you do then ? printf into the stetoscope? warm finger test ?

One way is to turn port pins on/off, in a pattern, to show where in the code, it has successfully reached.  Moving it backwards or forwards, until the problem/crashing point has been found.  Actual hardware, may have features, changing what I just said, such as alternative output mechanisms, such as LEDs or sounders.

[free_electron]

and if you don't have free port pins ?  they are all in use ?
get a damn jtag probe or an ice. printf is for pdp-11.
and on a tight system where the rom is nearly exhausted : do you really want to put debugging code in there ? and who's going to debug the debugging code ?
it is a waste of effort, time and system resources.
Jtag dongles are cheap. use em. set watches and breakpoints , examine memory, change contents , single step , direct call.
I like a large toolbox. debugging using only printf is like restricting yourself to using only a hammer

[tellurium]

so how do you debug a system that does not have a communication interface ? or when you are trying to make it work ?
you get a new processor and there are no libraries. Here is the register map and datasheet ...
your code compiles but all there is is silence. nothing comes out of the uart or usb or whatever...
what do you do then ? printf into the stetoscope? warm finger test ?

Turn it off and on again..

[MK14]

debugging using only printf is like restricting yourself to using only a hammer

With a hammer, box of nails, suitable pieces of wood and some clever planning/organization.  Rather big and complicated structures can be built.  Such as large sheds (out-buildings).

Printf, is a very powerful method, particularly when combined with incremental development (as already mentioned).

But it is NOT the only way, there are lots of other ways.  As you implied, by saying you have many tools in your big tool box.

The printf method, is not so good, when the problem lies within interrupts, or involves complicated timing and/or peripheral setup issues.  Then there are memory-leaks, and problems/issues outside of the immediate source-code, which also need to be diagnosed (to find out where the real problem lies), but could be outside of the range of where the printf's can be placed (e.g. closed source libraries (can link in) and/or closed source operating systems, etc).

[free_electron]

Turn it off and on again..

yeah , that's how all the crap these days is developed.

[brucehoult]

and if you don't have free port pins ?  they are all in use ?
get a damn jtag probe or an ice. printf is for pdp-11.
and on a tight system where the rom is nearly exhausted : do you really want to put debugging code in there ? and who's going to debug the debugging code ?
it is a waste of effort, time and system resources.
Jtag dongles are cheap. use em. set watches and breakpoints , examine memory, change contents , single step , direct call.
I like a large toolbox. debugging using only printf is like restricting yourself to using only a hammer

Lots of things don't support JTAG, not to mention that if you're writing software for someone else's commercial product JTAG might not be exposed, you certainly can't do ICE etc.

I once did a lot of work on pre iPhone and Android mobile phones running the Qualcomm "BREW" OS where it was extremely easy and common to crash the system and the only debugging methods possible were drawing things on the screen and writing a file to flash. And if the thing crashed then the last X amount of your file logging never got written to the file. Our product had to run on literally a hundred or so different phone models, which we mostly bought used on ebay. Most used ARM7TDMI and ran at 1 MHz or so.

If you're designing your own board and even have a possibility to use ICE or JTAG then you're in a much more privileged position than many of us.

> and on a tight system where the rom is nearly exhausted : do you really want to put debugging code in there ?

If you're in a position where ICE is even an option, then you can probably also buy a larger part for development and then, yes, you can put debugging code in there.

[Mechatrommer]

and if you don't have free port pins ?  they are all in use ?
get a damn jtag probe or an ice. printf is for pdp-11.

what if the mcu no have jtag? why HW developers bothered to provide jtag feature in the first place? it adds nothing to the end functionality of the HW/mcu/ic/fpga right? why emulators/stimulus tools are developed? history tells us... people will develop any kind of debugging tools in order for them to make something works. in all kind of range from interactive debugger, emulators, jtag inline outline debugger whatever they are called, printf/code injection trick (when interactive jtag fails to work) etc, to the HW level debugger such oscilloscope, LA, multimeter, even maybe a resistor can be used as a debugger. when the problem is a nail, hammer is the tool, when problem is the fish, rod or lure is the tool, we dont catch fish with a hammer, but hammer has its place somewhere else. what if we dont have ready made/suitable debugger? we make/invent/improvise our own. what if we dont want to do it? we quit. not everybody has the luxury as anybody else. i have a chance to probe values in VB6 immediate panel, so i'll use it, because i have it. the way i see it, its equivalent to the printf or jtag scan or flipping pins in other environments. imagine how people do it before computers? using brains and mathematics? the precursors of what we have today, no? ymmv. cheers

[tellurium]

Engineers who don't use/trust the debugger very much tend to send debug messages over UART, or flash an LED, or toggle GPIOs while looking at scope/logic_analyser etc.

There is another aspect where printf style helps and interactive debugger does not.
That is, if a device is in the field, and crashes / misbehaves. There, logs can help a lot with forensics.

[free_electron]

and if you don't have free port pins ?  they are all in use ?
get a damn jtag probe or an ice. printf is for pdp-11.

what if the mcu no have jtag? why HW developers bothered to provide jtag feature in the first place? it adds nothing to the end functionality of the HW/mcu/ic/fpga right? why emulators/stimulus tools are developed? history tells us... people will develop any kind of debugging tools in order for them to make something works. in all kind of range from interactive debugger, emulators, jtag inline outline debugger whatever they are called, printf/code injection trick (when interactive jtag fails to work) etc, to the HW level debugger such oscilloscope, LA, multimeter, even maybe a resistor can be used as a debugger. when the problem is a nail, hammer is the tool, when problem is the fish, rod or lure is the tool, we dont catch fish with a hammer, but hammer has its place somewhere else. what if we dont have ready made/suitable debugger? we make/invent/improvise our own. what if we dont want to do it? we quit. not everybody has the luxury as anybody else. i have a chance to probe values in VB6 immediate panel, so i'll use it, because i have it. the way i see it, its equivalent to the printf or jtag scan or flipping pins in other environments. imagine how people do it before computers? using brains and mathematics? the precursors of what we have today, no? ymmv. cheers

in what age do you live ? any micro out there has either jtag, swd or another form of programming/debug interface , same for FPGA.
HW developers bothered to develop JTAG and SWD specifically to make it easy for the software developers to trace their code and debug. SWD : Single wire DEBUG. got it now ?
Next time someone designs a chip i will suggest the make a single pin for an led available that can blink error codes or you can hook  up a speaker and send out morse code.
and send the program into the device we'll do the same. you can send it in in morse code.

restricting yourself to printf for debug is just plain stupid.

[Mechatrommer]

in what age do you live ? any micro out there has either jtag, swd or another form of programming/debug interface

attiny10-85 age

Next time someone designs a chip i will suggest the make a single pin for an led available that can blink error codes or you can hook  up a speaker and send out morse code.

we've done it in attiny10 thank you! and err, we dont need cryptic morse code... plain 8 bit is "this age". attached is my latest incarnation of it, evolved several times originating from some china radio transmission project.. the drawback is.. you need an oscilloscope..

restricting yourself to printf for debug is just plain stupid

nobody is restricting you. they just give one of the option

[Siwastaja]

"I can't have even a single free IO pin, but I totally can have all JTAG pins" sounds like a made-up bogus argument.

In pin constrained small MCUs, the first thing you'd sacrifice would be the JTAG/SWD and use them as GPIO.

In large MCUs, you would have the luxury of having both JTAG and a "development/logging UART" available.

[mikerj]

Put printf() or some equivalent at interesting parts of your code, either simply saying "I am here now" or also printing the values of interesting variables.

Don't let anyone tell you that's a crude or beginner's way to debug. It's the professional's way.

A professional uses whatever method is most appropriate rather than constraining themselves to one specific method.  e.g. printf can be an execution time hog that can change the behaviour of the system on slower micros.

If you can't afford the timing changes from a printf() -- and it doesn't have to literally be printf(), it can be as simple as a unique byte put in a buffer or toggling pins on a GPIO port -- then how can you afford a breakpoint?

If you have a timing critical system you obviously wouldn't break on every loop, you'd make the break conditional on the failure mode.

[Siwastaja]

If you have a timing critical system you obviously wouldn't break on every loop, you'd make the break conditional on the failure mode.

Exactly - in other words, if(something_unexpected) {...}. What you are describing is bog-standard intrusive instrumentation. I can't imagine many cases where you can't afford to add that. Of course not printf() within critical loop or ISR, but storage of interesting data, bounds checks, error checks etc. - definitely yes. This is all intrusive, but is also all very necessary. Leaving it all out and hoping to catch error conditions on lab table with JTAG probe + debugger just does not cut it, no matter how much mental gymnastics is being done to paint this "professional approach because bells and whistles like real pro".

And once you realize this, the habit will get stuck. Because at some point you see how much time it saves you to find problems semi-automatically, before you even realized you had a problem, instead of chasing bugs. Oh, error(57) triggered with dbg1=-1259, I see! Good, it would have caused something really weird half an hour later.

Such preventive coding style has allowed to me to start writing much longer pieces of code when I have the right flow of mind: I just codify the assumptions I am making in form of asserts. If I made a mistake, chances are high one of these trigger and dumps out some kind of error report.

[free_electron]

i think you have a serious misunderstanding on how debugging a system using an ICE or TRACe probe works.
Just like you trigger events and send out a debug report the probe is set up to trigger on events and take a snapshot of anything you'd like to see.
The difference is you don't use the system processor to produce the report. The probe does that.
The probe can get to areas where your printf solution cannot go and it can do it without corrupting the processor state. You jump out and start collecting bits and bytes by copying them or starting to send them out using printf. The time it takes to collect and send out the contents , the data may have changed so you never see what is there at the moment of the trigger. This is especially true when you are trying to bring up the HAL or BSP.
i'm going to print out the content of a circular hardware buffer. while you are print'effing the contents of the buffer are still changing. so all you get is useless data.

Another issue is that , If your report produces nothing of interest, you are going through another code, compile, load, run cycle to add other fields that may or may not yield something.

With an ICE and TRACE : if the event fires you have the entire processor state and the stuff of interest in one clockcycle . "Big" processors have trace assist where they can even record many cycles before the trap event. you can replay
Another thing you can do is to call functions of your system.
event occurs -> trap and halt , examine anything you want using the trace without state change (since it is done using a backdoor and does not require processor resources). alter content of data / registers through the debugger then move the execution pointer. You can let the trapped code continue until the exitpoint , halt it again there , set up new data and call the function again , deviating form the normal flow.

Code: [Select]

main :
call some_function (123)
*breakpoint if trapped=true
call some_other_function
...

some_function (int bleh) :
  internal variables x,y,z
  do some stuff here that messes up sometimes
  *trap if condition is met. example y>9
end some_function
If the trap fires in "somefunction" the processor state is saved together with what you want (this is set up in the ice, no need to write code)
You can examine x,y and z.
You find out y is not what it should be. set y to expected value and continue to verify it is indeed the problem.
Or, since the trap fired, continue. now the breakpoint will fire (the breakpoint is conditional : only if the trap fired will the program stop there.
You can now call the function again, manually.
You can interactively just type "somefunction(17)" and the processor will execute that function again. all without the need of recompiling or spending time implementing i/o functionality to fish for data or to invoke functions.

you also do not need to do the endless fix a bug compile, find the next one , compile again.
you can fix things in place and continue the run. Once you hit the point where you can't fix in place then you stop , go fix the source for everything you found.
Emulators let you modify code.
you can create an entirely new some_function, load it in ram and alter the calls so the new some_function is used instead of the old one. no need to recompile the entire program. just the function itself.

it's damn handy to be able, not to just look at things, but interactively modify and execute things while in trap
it's not always a bug per se. it may just be an unexpected or unhandled state . your code is correct but something happens for which you have no logic. something that should not happen, but does. first you need to find  what happened, then why did it happen ? your printf debug style is useless for such cases since you don't know the "what" so you can't code a report generator for it. The ICE is a "catch all".

There's a reason companies pay big bucks for those tools. When i was still doing harddisks every bench and desk had one of those probes. Lauterbach , Greenhills , American Arium.
They key strength is go anywhere, anytime, look at anything and do anything without needing to write debugging code or recompilation / relaunch. fix in place and interactively execute.

[PlainName]

The time it takes to collect and send out the contents , the data may have changed so you never see what is there at the moment of the trigger.

This is an important thing to bear in mind with printf (and similar) solutions: the act of printing can affect the timing of tasks and introduce an artificial speed constraint. Satisfied with your product you remove the debug stuff and suddenly things aren't happening as they should...

That's not a reason to not use printf, just something that should be born in mind when doing so.

[Picuino]

One solution may be to store the message in a memory buffer and send it after passing through the point in the program that must be fast.

[free_electron]

One solution may be to store the message in a memory buffer and send it after passing through the point in the program that must be fast.

That in itself takes time. And your 'exception handling code' modifies the processor state. You are doing things that in the real application will not be done.
That's why you use an ICE or TRACE. They are non intrusive , the processor does not know they are there.
Many modern processors (Arm , Risc-V, QoriQ, MPC5xx , colfire, Tricore, xtensa, ARC)  have ETM and PTM (embedded trace macrocell and program trace macrocell. ).
You set those up and the entire processor state , including all registers , cache is cloned in once clockcycle. they use a ring buffer. so yo can see what happened BEFORE the event. it's like a digital scope : you can see before the trigger.

something as simple as a cortex-m3 already has ETM and it can run over the SWD

[PlainName]

One solution may be to store the message in a memory buffer and send it after passing through the point in the program that must be fast.

Your logging routine would likely be in a separate task, so the output would be buffered anyway. But there is a limit to buffer size and the chances are it's going to fill up and cause a tailback regardless.

[DiTBho]

Complex programs have complex interactions and state history, which absolutely requires built-in instrumentation and logging

With AI-driven ICEs, you still need instrumentation, but the ICE itself provides logging through its channels and memory (up to 256Mbyte on-board at the moment) for the state history and misc.

Researching area, but the new line also comes with optical SC/LC link from the TAP back end to the host, which makes it super-fast and Gibson-ish-Cyberpunk

[Mechatrommer]

One solution may be to store the message in a memory buffer and send it after passing through the point in the program that must be fast.

less atomic codes is less preferable i can see that now why printf is opened to debate. thats why i still prefer the simple code earlier, at best 1 cycle is needed to buffer an 8 bit register value... 2 or 3 cycles to figure out if fault has occured and trigger program halt and spitting binaries on the GPIO. KISS... buffering integer or long or double data type will need extra carefull attention esp when ISR is running.. buffering nice and charming "message" for the programmer's view is moot for me... imho the idea of debugging is to pinpoint at which part of the code thats misbehaving, or unexpected input / ISR intrusion has occured, not simply to capture the whole line of registers values, and certainly not restricting ourselves to use certain capable mcu such as JTAG/SWD feature i cant imagine how people who depend their life on JTAG alone program avr tiny (or even mega) line of mcu. no doubt jtag is charming as explained, i'm sure use it when i got to work in those line of mcus, and when its suitable/easier to do so compared to other strategies at hand.

[PlainName]

who depend their life on JTAG alone program avr tiny

Even the tiny has ICE nowadays

Doesn't help if you can't get 'em, of course

[Mechatrommer]

yup i dont have ice the hardware.. only a cloned isp mk2 if it can tells me where a recent interrupt routine.. interrupted my code (at machine/assembly/c level), maybe i should put some effort to get one?

[SiliconWizard]

There is no single, silver-bullet way of doing things.

Some bugs will only require simple logging to pinpoint, some trickier ones will require more complex code instrumentation based on your observations of the symptoms, and possibly several iterations of the instrumentation as you move forward.

Ways to debug time-critical parts of the code and non-time critical ones are also often different.

For non-time-critical stuff, and in particular when it doesn't directly interact with hardware (or that you can easily "mock" said hardware), this piece of code can be compiled on another platform such as a desktop computer and be tested/debugged there, which is much more comfortable and productive than trying to do it directly on a small embedded target.

Any part of code that is not directly using specific hardware and that has no timing requirements, I write it as portable as I can, and when it's complex enough that it would require serious testing (like some tricky algorithm), I do prototype it and test it on a desktop computer first before adding it to the embedded target's code base.

A small example: if you're implementing an ad-hoc, non-standard sorting algorithm that will run on a MCU, doing that with a desktop computer first is a very good idea and usually what I do. Once testing shows it's solid, you can use the code on your final target and as long as it's portable, there is usually no further debugging needed.

[Siwastaja]

The time it takes to collect and send out the contents , the data may have changed so you never see what is there at the moment of the trigger.

This is an important thing to bear in mind with printf (and similar) solutions: the act of printing can affect the timing of tasks and introduce an artificial speed constraint. Satisfied with your product you remove the debug stuff and suddenly things aren't happening as they should...

That's not a reason to not use printf, just something that should be born in mind when doing so.

That's the reason to bake relevant safety and instrumentation into the code from day one. Take it into account. Leave margin for logging, understand the consequences of logging.

Debugger might not add CPU instruction cycles, but if it traces variables it sure causes memory and bus cycles with equivalent delays affecting both CPU and DMA, with the difference that now you don't know where the delays exactly are, or might not even know them existing, as evidenced in this thread again.

[Siwastaja]

Some bugs will only require simple logging to pinpoint, some trickier ones will require more complex code instrumentation based on your observations of the symptoms, and possibly several iterations of the instrumentation as you move forward.

Yes! But even quite low-effort instrumentation done from day one tends to give very valuable hints about where the problem is, this is specifically useful when the bug happens in production and you can't reproduce it no matter what. If, thanks to instrumentation, you get to know it's within one particular piece of one module, you may be able to cope with just reading and re-reading that code again, unit testing that piece of code, or even just rewriting it.

I generally suggest going for low-hanging fruits and "neat tricks". At least half-decent modularity combined with argument validity checks and all assumptions you feel even a bit unsure of asserted is a very good start.

And +1 to "test hardware-independent code on PC". Hardware-dependent, that's usually tricky to breakpoint/single-step or variable-trace with a debugger because it involves exact timing and nasty footguns like reading a register having side effects, so you are left with custom instrumentation anyway (which doesn't need to be difficult even if "instrumentation" sounds fancy).

Also, beware of premature optimization. Don't use "it hogs CPU resources" as an excuse, when the real reason is laziness. Logging a few important variables is a few CPU clock cycles. You don't need to do it as the first operation of the 10MHz BW current sense comparator overcurrent ISR, you don't need to do it in the inner loop of a DSP operation. Everywhere else, a few cycles very well spent. printf() itself is obviously too slow to call randomly everywhere.

[SiliconWizard]

Regarding assumptions, the safest way of dealing with them is... you guessed it... not to make any.
So wherever you expect a specific range of values for the code to run properly - just check the values.
Only if this checking would make the code fail to meet timing requirements, could you consider removing the tests and making sure of the ranges by thorough testing and static analysis, but I suggest removing the tests only as a last resort.

As to asserts that have no effect on release code, see the quote I put in an earlier post.

[Siwastaja]

By assumptions, I mean we all make them. Otherwise we would write perfectly generic code which would solve all the problems in the Universe. But real code has built-in assumptions such as number_of_samples needs to be an even number, because that's obvious to anyone who understands the particular algorithm, right? But still, maybe your configuration bus corrupts a value every now and then. Or someone does not understand the algorithm. Or you realize down the road that odd number_of_samples would make sense after all and forgot about the hidden assumption. Make it explicit by assert(), because the other option would be to write another algorithm for odd values which could take too much design time. But an error check is way cheaper!

Make this a habit:

 instead of / in addition to:
// n_samples must be even
write:
assert(n_samples % 2 == 0);

And of course,
assert(n_samples >= 0 && n_samples <= NUM_ELEMS(sample_tbl)); // where #define NUM_ELEMS(arr) (sizeof((arr))/sizeof((arr)[0]))

[Nominal Animal]

No tool beats understanding the difference between developer intent and what the code actually does.

There are tons of different tools that can help you with that; the exact set best for the task depends on the developer and project at hand.

The worst case is when the developer culture is of the "throw it at the wall, and use that which sticks; you don't need to understand how or why it works as long as it looks like it might work" kind, and you support that by getting them the top-of-the-line debugging environments and tools.  You don't get any better code, you just get code that breaks in more inventive ways.

Just like security, debugging is something you need a human to do.  Tools can make that easier, but if the human uses the tools to "do the debugging work for them" (instead of describing what is happening and how; the why is human stuff), the end result tends to be shoddy shit the world is already full of.  I repeat, the most important tool in debugging is your own understanding, especially discovering and understanding the difference between the intent of the code and what the code actually does.

For example, many developers have difficulty noticing = instead of == (in code written by others; it is very human to not notice such typos in ones own output).  I, too, often miss those when I read the code to understand the intent.  However, when I do understand the intent of the code, I can re-read the code, and compare the intent to what the code actually does; like a hostile prosecutor of sorts, if you will.  Then, such bugs are surprisingly easy to notice.  The hardest part is switching between the two "modes", understanding the intent, and comparing the code against the intent, because a lot of code has more than one "intent": one is an immediate intent for the current block, but there may be other more general intents, like doing things a specific way to avoid having to do something undesired.  Sadly, usually those intents are not documented at all, and are only discovered when discussing the code amongst developers.  For my own code, I do need to sleep between writing it and analyzing it this way, or I won't "see" my own bugs at all.

[DiTBho]

Debugger might not add CPU instruction cycles, but if it traces variables it sure causes memory and bus cycles with equivalent delays affecting both CPU and DMA, with the difference that now you don't know where the delays exactly are, or might not even know them existing, as evidenced in this thread again.

Yup: two solutions for this
1) forcing the whole system into equivalent time, so the debugger will appear "no intrusive" (not a solution, it's rather a work-around)
2) don't use the ICE to measure time-sensitive things, use the performance analyzer, which is equal to measuring the time lapse between a rising and falling edges of a GPIO

[DiTBho]

debugging is something you need a human to do

Sure, A.I.ICEs don't understand humans. Some humans understand A.I.ICEs, but, until the strong-AI singularity (in the far far future ... 2060? 2070? 2100?), humans are the smartest things on Earth.

Don't hire an A.I.ICEs to debug humans things. It won't work.

A.I.ICEs are simply more expensive tools but also useful when they can better and faster identify common human behavioral patterns, hence common human bugs.

And don't let an A.I. (or RADs) code your programs for you. It won't work.

[Mechatrommer]

And don't let an A.I. (or RADs) code your programs for you. It won't work.

it can, only if it can understand when we convey our message to it verbally. or if it can read our mind. but then...

until the strong-AI singularity? (in the far far future ... 2060? 2070? 2100?), humans are the smartest things on Earth.

according to Einstein's postulate, it would be impossible (if i understand it correctly)...

You can't solve a problem with the ways of thinking that created it

expecting machine to do human's job will make the human is lesser than the machine, i think...

assuming certain "semantic" is/will be of infinite ambiguities/possibilities... you can put enough database into an expert system at current time, but when new problems raised in the future, you need to update the database otherwise the AI will be rendered obsolete. human is smartest, but doesnt mean free of errors or dumbless. how can human make a perfect machine if they themselves dont understand what they are made of, esp spiritually, or in the soul's world... if you understand, then you understand, if not, then i can understand why you dont

[Nominal Animal]

debugging is something you need a human to do

Sure, A.I.ICEs don't understand humans.

Rather, programming languages are still the best tool we have for conveying human intent to nonhuman processors.

If we had anything better to convey the intent, then we could write a compiler that generated code that we intended, instead of code that we wrote.  But we don't, so we can't.

It would be supremely funny to have debuggers that understand human intent better than a compiler.

(While I don't like code generators, they are still extremely useful for cases where the solution has an exact mathematical/computer-understandable description.)

[DiTBho]

If we had anything better to convey the intent, then we could write a compiler that generated code that we intended, instead of code that we wrote.  But we don't, so we can't.

It would be supremely funny to have debuggers that understand human intent better than a compiler.

(While I don't like code generators, they are still extremely useful for cases where the solution has an exact mathematical/computer-understandable description.)

It sounds "Pro-Logic", old 90s articles on the Byte-magazine 

Some inspired all the computers we see in Star Trek, pure science fiction, but full of passion and hope for a distant future.

Like the movie "BackToTheFuture part2": the movie predicted that in 2017 we should have flying cars ... have you seen any? ... 2021? 2022? OK, Google/LA (USA) has "self-driving cars" but nothing that can fly, and it's still clear if there is a quantum gravity ("graviton"? which ... !O!M!G! the name is already taken by Marvel for one of their superhero, I am afraid we need to choose a better name to avoid to be quoted in a court case), we still talk about an hypothetical quantum of gravity, an elementary particle that mediates the force of gravitational interaction (which is not a true force, but rather an effect of a potential field), so, I think we won't see any flying skate-boards for a long while ...

... except something similar (exploiting electromagnetic fields rather than anti gravity) on Youtube, thanks to HackSmithIndustry 

See
See

[SiliconWizard]

debugging is something you need a human to do

Sure, A.I.ICEs don't understand humans.

Rather, programming languages are still the best tool we have for conveying human intent to nonhuman processors.

If we had anything better to convey the intent, then we could write a compiler that generated code that we intended, instead of code that we wrote.  But we don't, so we can't.

That's actually a more general question: languages in general!

We have indeed found nothing better, over the course of humankind, than languages to convey our intents to others.
The question of *why* we would want to use any other way is in itself both fascinating and concerning.

A language (be it natural or programming) is a tool to express what's happening inside our brain without the other party having to "guess" what it is.

But are we becoming so "lazy" that languages are now becoming too much of a burden?

[SiliconWizard]

By assumptions, I mean we all make them.

Yes. What you meant by that was clear and clearly understood.

Otherwise we would write perfectly generic code which would solve all the problems in the Universe.

No we wouldn't. Or if we do, maybe that should look like:

Code: [Select]

42;

But real code has built-in assumptions

Real code is code we write. It has the built-in assumptions you choose to build in. It's either a deliberate choice (in which case you better make sure your assumptions hold), or an overlook (which is not great, but we're all human.)

such as number_of_samples needs to be an even number, because that's obvious to anyone who understands the particular algorithm, right? But still, maybe your configuration bus corrupts a value every now and then. Or someone does not understand the algorithm. Or you realize down the road that odd number_of_samples would make sense after all and forgot about the hidden assumption. Make it explicit by assert(), because the other option would be to write another algorithm for odd values which could take too much design time. But an error check is way cheaper!

Make this a habit:

 instead of / in addition to:
// n_samples must be even
write:
assert(n_samples % 2 == 0);

And of course,
assert(n_samples >= 0 && n_samples <= NUM_ELEMS(sample_tbl)); // where #define NUM_ELEMS(arr) (sizeof((arr))/sizeof((arr)[0]))

Your example is a simple, yet perfect example of what I'm saying. If n_samples must be even and you have no means of ensuring it in a reasonably proven way, check that it is, which is what you're doing above.

All I'm saying on top of that is an extra point about assertions (and Hoare did agree with me there) and how I usually favor run-time tests at all times, unless again in specific, documented cases for which it is unpractical (which are relatively rare in practice.)

I know this isn't a very popular approach, except in safety-critical software. I'm just expressing my take on it. Everyone's free to agree, disagree or go fishing. =)

[nctnico]

All I'm saying on top of that is an extra point about assertions (and Hoare did agree with me there) and how I usually favor run-time tests at all times, unless again in specific, documented cases for which it is unpractical (which are relatively rare in practice.)

I know this isn't a very popular approach, except in safety-critical software. I'm just expressing my take on it. Everyone's free to agree, disagree or go fishing. =)

Tests like these also make it possible to work on projects with less experienced programmers. In one of my recent projects I spend a couple of hours to include a mechanism that checks whether a mutex for an I2C bus is locked before use (why this is necessary is a long and sad story). It is easy to forget to lock a mutex but the consequences can be severe. With the check in place the program crashes with a clear error message.

[SiliconWizard]

All I'm saying on top of that is an extra point about assertions (and Hoare did agree with me there) and how I usually favor run-time tests at all times, unless again in specific, documented cases for which it is unpractical (which are relatively rare in practice.)

I know this isn't a very popular approach, except in safety-critical software. I'm just expressing my take on it. Everyone's free to agree, disagree or go fishing. =)

Tests like these also make it possible to work on projects with less experienced programmers. In one of my recent projects I spend a couple of hours to include a mechanism that checks whether a mutex for an I2C bus is locked before use (why this is necessary is a long and sad story). It is easy to forget to lock a mutex but the consequences can be severe. With the check in place the program crashes with a clear error message.

One of the reasons, and rationale, for not doing it, that I've heard the most, is not about the testing at run-time itself (except again when the cost in terms of timing would be unacceptable), but mostly all about error handling.

The most common claim is something like (and I'm sure many people reading this thread would say the same): "ok, but what will you do if the test fails?"

Many people assume that if something that "shouldn't" fail in software in normal circumstances, fails, then there's generally nothing you can do anyway, and then better pray or let the whole thing crash. As there is nothing reasonable we can do about it, right?

It all depends on the context of course - there is again no silver bullet and unique answer to every case.
But handling errors gracefully is possible in a surprisingly large number of cases in reality. It does require some thinking and some extra work though, no doubt about it.

The question to ask is simple: "if this test fails, what should the system do that would cause the least harm/surprise the user the least/allow the system to get back on track/etc. ?"
There's almost always an answer to this one. Even if, say, in a particular case, some test failing would be unrecoverable and would require a system reset: then I'll favor handling it explicitely and calling whatever will explicitely reset the system - in the cleanest way it can - rather than just let code execute and do whatever until the system may encounter a hard fault and reset on its own.

Again, the context matters. Those are just general rules. There may be exceptions here and there. My 2 cents. Take it or leave it, I'm not even saying this is what *you* should be doing, I'm just saying that's what I do and what I tend to promote if my advice is being requested.

[tellurium]

if you exist and you are self-aware, good news, you are already in for free! ... the problem is what happen when you stop to exist? Nobody knows. Where your where does your conscience goes? your body can be recycled, but what about your conscience? it's a complex problem, and nobody can describe what exactly is "consciousness

So much for printf debugging

[PlainName]

yup i dont have ice the hardware.. only a cloned isp mk2 if it can tells me where a recent interrupt routine.. interrupted my code (at machine/assembly/c level), maybe i should put some effort to get one?

Can't remember, sorry. Typically they will try to ignore interrupts (otherwise it's a right pain stepping non-interrupt code), but I reckon you could put a breakpoint in the ISR and then see where the return gets you to. Often that's a library function and not awfully useful, though

[MK14]

So much for printf debugging

I suppose, debugging a program/computer with printf's, is a bit like a sobriety test for humans.         

[DiTBho]

If you can debug with printf, you can prove to your consciousness that you exist! 

[free_electron]

If you can debug with printf, you can prove to your consciousness that you exist! 

if you can be debugged with a printf you must be a  small routine in a simple system.  . careful, some script kiddie may replace you with a very small script , written in java , running in an x86 emulator on top of an Apple M1 ... also known as two million lines of code, 20 billion transistors to toggle a boolean...

[Smokey]

Another thing to be aware of...

I'm currently doing a design with Bluetooth for the first time (Cortex-M0 SoC).  I've found that if you set a breakpoint in the debugger while there is an active BLE connection, the connected device will timeout and disconnect.  So breakpoints to debug stuff related to BLE isn't an option (please tell me I'm wrong if I'm just missing something).

[PlainName]

So breakpoints to debug stuff related to BLE isn't an option

That's not BLE-specific - anything that's externally time sensitive will have an issue when the related code is stopped at a breakpoint.

[free_electron]

Another thing to be aware of...

I'm currently doing a design with Bluetooth for the first time (Cortex-M0 SoC).  I've found that if you set a breakpoint in the debugger while there is an active BLE connection, the connected device will timeout and disconnect.  So breakpoints to debug stuff related to BLE isn't an option (please tell me I'm wrong if I'm just missing something).

that's because the hardware needs the firmware to run and service interrupts. There are background processes. if you are trapped on break those processes are also stopped.
That's where a real ICE / trace comes in. you can snapshot the state WITHOUT stopping the target. This is done through the ETM and PTM modules. Modern processors like Cortex, Risc-V , ARC all have hardware assisted debugging. It's like a mini logic analyser. You set up the conditions and when it fires it can make a snapshot. this is then loaded into your IDE and you can work at sourcelevel.

that's why debugging with printf is like mowing the lawn by plucking one strand at a time with tweezers and telling everyone else to stay off the grass while you are doing it..

[Doctorandus_P]

What is your general approach for any compiler to c debug code ?

There is no "general approach". You have to match the method to the problem.

If it's some general algorithm, I usually firs write the code as a PC program and then port it to a microcontroller, as the compile and re-run cycle is easier.

printf can be handy when designing new algorithms. say you have for example a sorting algorithm, then printf can leave a trace on how it goes through the algorithm without interrupting it.

Somewhat similar, a method I like is to toggle some output pins of a micrcocontroller. I've written a few small macro's to do this and these can output 8N1 messages. Output is quick enough that I just can leave it running, and then when a program crashes I can back-track through this info in a logic analyzer and also correlate it with other I/O that is captured. The code is small and quick enough that I can even use it in ISR's to monitor how the ISR is functioning (I have used it in an ISR with about 6 if() statements and even a goto to reset some states under specific data faults.

I once had a nasty bug but could not find motivation to fix it. After about a year I made a big pot of coffee, and just started reading source code. Big sections of the code were very easy to read (thanks to maintenance and refactoring) and could therefore be quickly dismissed. In C & C++ pointers are always suspicious, especially with not very experienced programmers (that was me at that time) and I knew that, so I paid extra attention to each pointer I saw. In the end it was a pointer to a local but non static variable in an ISR and I found it in a few hours.

My most nasty bug ever was at work. It was for a (then quite big)16-bit microcontroller with a MB of flash. I spend approximately 7 work days guessing at parts of code that could be deleted (It was an RTOS system with 30+ tasks running) and I managed to remove about 95% of all code in that week and a half before I got stuck and had to hand it over to a more experienced colleague. It took him another two days, including use of a logic analyzer to find the bug. It turned out to be a 16 bit pointer with a too small range so it wrapped around when it should not, and that bug was "fixed" by changing a compiler flag in the GUI.

I already mentioned it briefly, but refactoring sections of dubiously written code is an important debugging technique. Instead of trying to fix the mess afterwards, preventing it from going awry in the first place is much better. So even before refactoring (duing maintenance) you have the initial high level design of the code.

I've had a bit of toe dipped in unit testing. I quite like the idea of writing both tests and code at the same time. In one of my experiments I was learning some Python (a quite atrocious language with horrible whitespace depencency and no variable declarations, so it just "invents" new variables on a simple typo, and it does not compile so even checking if all code is even executable is a nasty exercise. /rant) In the end I had a unit test function that simulated input of about 20 packets sent over a network (and thus got fragmented etc) and those got re-assembled and processed, and the processed output was captured once and thereafter could be used to test if the whole unit still worked after further modifications, and this all worked quite nicely.

[brucehoult]

if you can be debugged with a printf you must be a  small routine in a simple system.

In fact it's the exact opposite.

Student programs can be debugged with breakpoints and single-stepping. Large complex systems demand printf and external automated analysis of the resulting MBs or GBs log file. You wouldn't live long enough to stop at a breakpoint at every one of the millions of times a printf logs to the file, let alone to make a meaningful examination of variables at that point.

[Siwastaja]

So breakpoints to debug stuff related to BLE isn't an option

That's not BLE-specific - anything that's externally time sensitive will have an issue when the related code is stopped at a breakpoint.

Now here's the thing. Let's say you have a software-controlled motor inverter or DC/DC converter. You just can't stop the program at arbitrary point, or you blow up things (semiconductors, physical things attached to motor axle, whatever).

Now you are lost trying to find some nasty bug, and the only way you seem to have is to add breakpoint in the middle of critical handler and examine internal state, for whatever reason.

The options are:
1) Add a breakpoint in debugger.

Consequence: when it is hit, all hell breaks loose.

2) Write an error handler which does safe shutdown, by doing the necessary GPIO writes to turn MOSFETs off, turn mechanical brake on, whatever is needed.

Instead of adding a BKPT, you just add call to this error handler. You achieve the same: program stops here. Difference is, it now happens safely, as defined by you.

Now you totally can use #2 to stop the system, and then attach debugger probe to continue examining the state. This is combination of manual code intervention and using the general purpose debugger.

But the thing is, once you have had to write your own handler manually anyway, why not improve it to log the state of interest? This will then break the dependency of having to rarely attach a debugger. Your workflow is simplified. This is because you can't avoid manual instrumentation anyway, so why not make good use of it.

There is also option 3), sometimes peripherals such as motor control timers have special hardware logic so that when the debugger halts the CPU core, you can also halt the timer plus make the outputs go to some predefined IDLE state.

But quite frankly, this seems like a made-up response by the manufacturer to the criticism that breakpointing critical control code is impossible. But the problem isn't properly solved: you still only can make the particular timer module go in safe state, but likely this isn't enough, you would want to toggle a GPIO, maybe with a delay, whatever is needed for completely safe state. Can't automate everything through the peripheral. And then, finally, you need to configure this debug mechanism in code. So you can't avoid modifying code for debugging purposes, after all.

[MK14]

You wouldn't live long enough to stop at a breakpoint at every one of the millions of times a printf logs to the file

You also wouldn't live long enough to read through all those millions of printf's.  So, if you can automate sifting through those printf's, because of criterion of detecting the bug situation.  E.g. a variable has exceeded 1,000.  Then a powerful enough debugger, could be set to the appropriate, conditional breakpoint(s).  To similarly find the bug locations and hence fix the situation.

Both printf's and debuggers, both have their uses, and hence pros and cons.  But to suggest that one particular method is THE method, isn't taking into account the thousands or even millions or more, of different circumstances (hardware, programming language, experience level, requirements, budgets, timescales,  etc).  One or more of which, can favor one method over others.

[Siwastaja]

Student programs can be debugged with breakpoints and single-stepping. Large complex systems demand printf and external automated analysis of the resulting MBs or GBs log file. You wouldn't live long enough to stop at a breakpoint at every one of the millions of times a printf logs to the file, let alone to make a meaningful examination of variables at that point.

This, and relatively easy to use but powerful tools like Excel/Libreoffice & Matlab/Octave are hugely useful for analyzing .csv logs. Especially if you have a system that measures or controls any "analog" parameter say acceleration or current, or has PID control loops or whatever.

[PlainName]

So breakpoints to debug stuff related to BLE isn't an option

That's not BLE-specific - anything that's externally time sensitive will have an issue when the related code is stopped at a breakpoint.

Now here's the thing. Let's say you have a software-controlled motor inverter or DC/DC converter. You just can't stop the program at arbitrary point, or you blow up things (semiconductors, physical things attached to motor axle, whatever).

This is going to go nowhere because we are polarised and arguing extremes. The points you raise are perfectly good but biases towards having a downer on ICE. Not everyone will be controlling a motor inverter and often it doesn't matter at all if halting kills the system - at that point you'll have discovered what you need to. It is also quite a bit fast than compiling, never mind writing some code (after figuring out what to write) beforehand.

In practise, having access to ICE features doesn't mean you're going to use them exclusively. The actual problem, and one's assumption as to likely cause, will determine which tool is better suited. Sometimes a combination: a breakpoint tripping (or not!) might indicate the route to the fault, and then printfs show why it took that route. Conversely, plastering printfs over some function might still miss the important bit and a breakpoint on the printf will allow quick examination of something slipping through the gaps.

But just as one doesn't get the best out of a screwdriver by using it as a hammer, a knowledge of the capabilities and use of debugging tools is important, and then the appropriate one can be chosen at the time. Of course, someone will favour one tool over another, and someone else wil prefer a different tool. Neither is 'wrong' if it works for them - in the end, fixing the problem is what matters, not so much what was used to get there.

In the context of this thread, I think it would be more useful to a beginner to know how to use the various options, so he can build enough knowledge to make up his own mind, rather than being told such and such is shit and absolutely the wrong thing to do. Think about C goto's and  how many beginners think they are spawn of the devil and any code with them in is by definition complete rubbish. But they have their place, and only by learning how to use them can one realise what that place is.

[voltsandjolts]

Let's not argue over who killed who

It's interesting to read this thread and see the various debug techniques you all use, all have their uses. Being aware of all the options is the important point, so you can make an informed decision on which best suits you, the task at hand and the available tools.

Happy debugging to all

[free_electron]

Happy debugging to all

no, debugging is for beginners. The real greybeards directly make the mask for an expensive mask rom and it works first time !

[Siwastaja]

It's a funny joke sure, but actually I have been starting to question the idea that all software will definitely have a lot of bugs and ironing them out takes a lot of effort.

I mean, I have been going up from the Dunning-Kruger valley of despair, maybe I was there 15-10 years ago.

During last 2-3 years, I have some weird experiences, like this one: I wrote quite a complex embedded communication system thingie in one go, something like 1000LoC. I know past myself, and I think it would have been a month of more-or-less painful iteration to write and test it.

Now writing the code felt somehow different than it used to: my code output was slower. So it took like 4-5 days to finish the whole large submodule. Then I compiled it for the first time. Fixed a few typos. Compiled again. Flashed on the chip. Wakes up and seems to work. Try a few test vectors. Everything seems to work in first go. Test more thoroughly. Find a bug that automagically prints error code, spend 1 minute fixing the bug. Voila, it's now in production. It's still in production after a few months, no problems with that piece. I have never had such low bug rate before in such medium complexity project.

I'm not saying there won't be any bugs. But I am saying I do not consider myself world-class programmer expert, I consider myself someone who have left the Valley Of Despair a long time ago but is still far from the Plateau of Sustainability. I'm sure many on this forum are way better and more careful programmers than I am. I'm still not capable of doing that every time and something still drives me to occasionally write beginner level crap I don't like and which causes colossal pain, but having a few experiences like this gives me hope, and it's a sign I'm on the right track.

This makes me truly feel it is completely humanly possible to write almost bug-free code, which, coupled with integrated-from-day-one instrumentation, means very very very little of those painful debugging sessions.

I like Nominal Animal's "eyeball mk II" metaphor; I know some will take it as a joke, but I suggest you take it seriously. Debugging is difficult and slow, avoiding bugs by understanding what code you write and why is a true and actual possibility.

[Mechatrommer]

Happy debugging to all

no, debugging is for beginners. The real greybeards directly make the mask for an expensive mask rom and it works first time !

and why the beard is grey?

[free_electron]

Happy debugging to all

no, debugging is for beginners. The real greybeards directly make the mask for an expensive mask rom and it works first time !

and why the beard is grey?

experience. all the best programmers are old.
- They wrote cobol on an ibm360 and that software is STILL in use today. I wonder what is the lifespan of todays software before a new version comes out with bugfixes... weeks ? days ?
- They could toggle in bootloaders using switches. Now bootloaders don't even fit on a single floppy anymore.
- They watched the blinkenlights on the front of the machine to know the state of the computer. Sometimes they hooked up four resistors to make a crude dac and play Beethoven's fifth using idle cycles while crunching massive amounts of data and serving hundreds of users over 75 baud links at a breakneck clock speed of 750kilohertz.
- their harddisks were still hard disks. none of that solid state or 2.5 inch stuff. they were belt driven from an ac motor and used a real air compressor to make the heads float. There was one head for all platters (ramac). the size was not measured in megabytes but cubic feet and psi of (raised) floor load.
- they used pencil and paper to hand draw logic diagrams and build supercomputers (Cray)
- they landed a man on the moon with nothing but a bunch of nor gates and some software
- They knew what error 1201 and 1202 meant and the machine could safely shed processes without crashing in flames (literally)
- they would never be caught using printf to debug. There was no printf , or C or unix. they had rollers and lights to examine the entire processor state. they would single-step through the program and alter machine state at will. they could go anywhere and poke at anything.
- they knew which of the 9000 vacuum tubes to replace if the machine failed. now we just hotswap the entire damn thing.
- the power of their computer was measured in kilowatts (electric) and btu (cooling) not in megaflops.
- they had full blown shared file systems, remote networking, ethernet, email , graphical user interfaces with movable and sizeable windows, laserprinters , mice, wysiwyg .. all on a box with nothing but 74 series TTL ic's. the damn thing didn't have a processor, or bit slice. it had 74181 alu's ! (Xerox Alto). those alu's didn't even have flipflops. they were just a bunch of and and nor gates

now we have machines that have a million times more memory, run 5000 times faster , have 60 times the amount of processors and what do we do ? we bolt a virtual machine on the hardware , emulate a different system , plonk java or python on it and then debug the code with .. printf (or equivalent) . ... we've lost it , completely. There is no more hope. We might as well pack it in and stop software development altogether. there is zero chance to make anything that is bug-free

[ralphrmartin]

If you are old enough to have learnt programming in an environment where you wrote your program on a coding sheet, it was typed (by someone else) onto punched cards, and submitted as a batch job whose output you got the following day, you realised in those days that relying on debugging was never going to get a non-trivial program completed. Instead, you learnt to write correct code in the first place.

[Mechatrommer]

man, jtag is not even mentioned in the list! do you mean all the best people extinct? who greybearded uses cobol nowadays? all the glorious stories told without mentioning the hardworks and experienced gained during the "debugging" stage of life. you dont expect greybearded just came out of a womb in an instance right? btw... talking about "the best men" they do still exist... its just "politics" more and more get in the way... i just spent sometime few days ago reading about Perelman's Incident.. he still live and probably in exile... even if you can meet him i believe it will not be a good experience talking with heavy left-brained person...

]t

on to something about semiconductor debugging...
https://semiengineering.com/knowledge_centers/eda-design/verification/debug/

Instead, you learnt to write correct code in the first place.

thats called debugging in your head. probably was common before the invention of turing complete universal machine (computer) with "if" and "branch"

[Siwastaja]

you dont expect greybearded just came out of a womb in an instance right?

Good point!

Maybe I can help you because my beard is still dark brown. So I remember my beginner days as they are not too far in the past. I remember the phases of getting a better embedded programmer. And the process is still going.

And: I never did that debugger thing, really. The problem is, debugger with all its fancy features seems very convenient, compared to actual printf debugging. I can't prove it, but I suspect many who start that way early get stuck with the debuggers. Being able to connect an existing tool to your code and single-step through it and examine memory causes certain mindset of writing the software.

Compare this to not having a debugger and resorting to printf debugging. At very first, it is literally just
printf("place A\n");
...
printf("place B\n");
...

This is not very powerful. Finding bugs is a struggle. But it sets you on the right track because:
* It teaches you that finding bugs is difficult, because it will always be no matter what tools,
* It teaches you, piece by piece, to write better instrumentation. Because you see, this is easy to improve:
printf("place A\n");
becomes:
printf("i=%5d, coords=(%5d,%5d)\n", i, x, y);

Then it becomes:
if(loglevel > 5) fprintf(log, __FUNCTION__ ":" __LINE__ "i=%5d, coords=(%5d,%5d)\n", i, x, y);

Then you understand to add:
assert(x > y);

Or in MCU ISR, you understand to do this:

if(logging && cur_trace_idx < NUM_ELEM(trace))
    trace[cur_trace_idx++] = latest_measurement;

because this is all you have, you have to work with the code.

And it grows to a mindset of writing robust, self-checking, instrumented code. Which is a very good target, and while it sounds fancy, you can easily grab the low hanging fruits instead of going super complex with this.

And at the end of the day, you may still connect the JTAG and debugger. Last time I did was in 2020.

[Mechatrommer]

i cant differentiate  between "debugger" and "instrumentation" that you said or even with a single innocent through hole resistor, if all of their intent is the same, ie to find fault in your code or invention. only difference is the technique or strategy, and to some extend, different effect (as already being discussed) but they all to debug, or to get rid of a bug (fault), hence a "debugger". and afaik assert is the one used by new kids to learn programming, you all sharp people will eventually come down it anyway (i dont, since after i made my first "real" program) and i remember reading book that assert can be left intact harmless when publishing the program into real world because programmers are to lazy to take care of them in the end, at least thats the impression i got after reading the book. when its triggered, users can unhappily report back the assert msg back to the programmer for the next fix patch. and if its the same as the way assert is executed in SW PC programming, it can be devastating if implemented in life-critical embedded world, no?

[Siwastaja]

Obviously, you need to come up with something assert-like of your own in a life-critical embedded system.

That's one of the hardest parts, recovery after a failure.

In order from worst to best:
1 Do not check for errors / wrong assumptions at all, let the errors propagate
2 assert and die in controlled way with logging
3 assert and recover

The point is, 2 will actually save your time AND make a better product because even if it happens in production, you will fix it faster! It's the obvious low hanging fruit you should just learn to take, yet people often won't, especially if they feel they can "easily" just find bugs on lab table with JTAG probe and debugger.

3 will be actually very difficult and challenging.

In all cases, you would be still "debugging" the code in the general wide sense of the term. Just spending different amount of effort to do it.

[free_electron]

man, jtag is not even mentioned in the list!

ever seen the frontpanel of an ibm360 ? That's JTAG on steroids ! you can see and modify any damn bit in the machine you want. singlestep it, step it forward, step if backwards, call subroutines.
they have large "rollers" on the side. you rotate that and it selects elements and registers in the computer. the contents are shown on little neon lights. The rollers are large electrical switches that also move a paper sheet that shows the function of the lights behind a plexiglass plate.

So yes, debugging from the very beginning was done by examining and tracing in hardware. not by spitting text to a teletype.
JTAg lets you still do that , but over a serialized 4 wire bus as opposed to the hefty bundles of cables going to that frontpanel
You could even modify the power rails of the computer just in case some voltage was a bit "marginal" to make the machine work properly.

[Mechatrommer]

ever seen the frontpanel of an ibm360 ? That's JTAG on steroids ! you can see and modify any damn bit in the machine you want. singlestep it, step it forward, step if backwards, call subroutines.

so i guess jtag is "interactive debugging" on steroids except, interactive debugging can be done on emulator or virtual machine, before it can run on real hardware.

2 assert and die in controlled way with logging
The point is, 2 will actually save your time AND make a better product because even if it happens in production, you will fix it faster!

so i guess assert is not much diffferent from printf, except with a little bit of addition.... if (fault) printf and halt... of course printf is not literally printf that we learnt in c... we can make our own "printf", maybe with different name like "assert" or in my case... "sendByteSerialTx" for my specific attiny project. similarly to when a hardware got overheated for unexpected reason, it halts and blink a red led. thats the debug sign!

[Bassman59]

After six pages of this shit (and counting), it seems like the correct answer to the question is, "Use whatever tools you have at your disposal and choose the one which best gets the bug found."

If that's printf() in code, then so be it.
If that's twiddling an output pin in an ISR, then so be it.
If that's using at JTAG or other debugger to inspect registers and variables, then so be it.
If that's writing correct code in the first place, then so be it.

You don't get bonus points for only using one sort of debug tool. The bosses don't care how you solved the problem. They only care that they can ship product so everyone gets paid.

[Siwastaja]

so i guess assert is not much diffferent from printf, except with a little bit of addition.... if (fault) printf and halt... of course printf is not literally printf that we learnt in c... we can make our own "printf", maybe with different name like "assert" or in my case... "sendByteSerialTx" for my specific attiny project. similarly to when a hardware got overheated for unexpected reason, it halts and blink a red led. thats the debug sign!

Yes, you got it . Incrementally develop new habits that make debugging easier. Let the frustration of bug-hunting guide you.

I prefer multi-interface error handlers. Any error will always come with an integer error code, and error handler at very least blinks the LED n times, using simple blocking delay loops. Almost every project has an LED, and it's accessible with GPIO without other peripherals.

If everything else fails, you can always count the blinks. But the same handler will try to send more detailed information on UART - and not just once, repeat once a minute so that you can keep the thing powered and plug in the cable. If the system is CAN connected, send a report to CAN. Log to SD card if you have one. And so on.

What to include in the report? I like a union of different types of general purpose debug integers that I set in code. I also include control/status registers of some suspicious peripherals like DMA, SPI etc, print a few CPU registers like link register, and a stack dump.

But most valuable is that you just store intermediate values that led into the final result in variables that can be printed in your error handler, or accessed with a debugger if you so like. It is unfeasible to trace the whole memory with a debugger all the time. But if you keep at least a mimimal record about the chain of calculations/decisions, then it's trivial to print them out.

[Siwastaja]

After six pages of this shit (and counting)

Why do you think it's shit? Like you say, the result matters. What is wrong with giving the OP with as many different tools and viewpoints as possible? And if you are so excellent and others write shit, why post here at all? What is wrong with you?

[Bassman59]

After six pages of this shit (and counting)

Why do you think it's shit? Like you say, the result matters. What is wrong with giving the OP with as many different tools and viewpoints as possible? And if you are so excellent and others write shit, why post here at all? What is wrong with you?

Here in the US (and especially New Jersey, where I'm from), when we say, "pages of this shit," we don't mean it in the pejorative. I suppose I could have said, "stuff."

But, seriously, the "shit" is the constant back and forth of "I do it this way, it's the best way, your way isn't any good."

[tellurium]

After six pages of this shit (and counting)

I got elastic bands keepin' my shoes on
Got those swollen-hand blues
I got 6 pages of shit on the forum to choose from

[Picuino]

But, seriously, the "shit" is the constant back and forth of "I do it this way, it's the best way, your way isn't any good."

All of your debugging ways are crap. The only way to do it right is DARYL's way:   

[tepalia02]

I may sound dumb. But for me, printing something in the output terminal helps most of the time.

[gnuarm]

I may sound dumb. But for me, printing something in the output terminal helps most of the time.

When I'm working in Forth, that's called REPL, Read, Evaluate, Print Loop. Maybe it won't work for everything, but it is useful 99% of the time. Instead of complex emulators, a simple serial port does the job.

[pcprogrammer]

I may sound dumb. But for me, printing something in the output terminal helps most of the time.

When I'm working in Forth, that's called REPL, Read, Evaluate, Print Loop. Maybe it won't work for everything, but it is useful 99% of the time. Instead of complex emulators, a simple serial port does the job.

But the problem with Forth is that it requires a backward way of thinking. Don't get me wrong, it is a nice language and I have used it instead of basic on micro's like 8051 and 6502, but now a days with good C compilers and fast micro's for me it is a bit outdated.

It was certainly fun and easy to do assembly coding with it. I have also used it to mimic a dongle for Xilinx software. Was not that hard, just count a number of pulses and toggle a line at the right count.

[gnuarm]

I may sound dumb. But for me, printing something in the output terminal helps most of the time.

When I'm working in Forth, that's called REPL, Read, Evaluate, Print Loop. Maybe it won't work for everything, but it is useful 99% of the time. Instead of complex emulators, a simple serial port does the job.

But the problem with Forth is that it requires a backward way of thinking. Don't get me wrong, it is a nice language and I have used it instead of basic on micro's like 8051 and 6502, but now a days with good C compilers and fast micro's for me it is a bit outdated.

It was certainly fun and easy to do assembly coding with it. I have also used it to mimic a dongle for Xilinx software. Was not that hard, just count a number of pulses and toggle a line at the right count.

Know I not backwards thinking it is.  Actually, many people object to the one tiny bit of syntax it has, Reverse Polish Notation, RPN.  That is not entirely accurate, really.  The interpreter simply interprets a word at a time, rather than waiting for a line to be digested and turned into machine code.  So the operands are normally first. 

That's not always the case.  A string is created with a word S" which then scans the input for the next quote marking the end of the text.  Many words that create words, look for their input after the word. 

Or did you mean backwards in the Appalachian back woods sense?  I prefer a simple tool.  I'm not designing systems with 27 interrupt priorities.  I design systems to get the job done while making the software easy to debug.  I'd much rather use a screwdriver or hammer, than a pneumatic fastener tool. 

[pcprogrammer]

Know I not backwards thinking it is.  Actually, many people object to the one tiny bit of syntax it has, Reverse Polish Notation, RPN.  That is not entirely accurate, really.  The interpreter simply interprets a word at a time, rather than waiting for a line to be digested and turned into machine code.  So the operands are normally first. 

That's not always the case.  A string is created with a word S" which then scans the input for the next quote marking the end of the text.  Many words that create words, look for their input after the word. 

Or did you mean backwards in the Appalachian back woods sense?  I prefer a simple tool.  I'm not designing systems with 27 interrupt priorities.  I design systems to get the job done while making the software easy to debug.  I'd much rather use a screwdriver or hammer, than a pneumatic fastener tool.

I was hinting to the RPN way of how it operates.

Agree with you that a simple tool to do a simple job is the way to go. Always choose the right tool for the job, but sometimes you have to use what is at hand and try to make the best of it. For example if all you have is a knife and you need to strip a wire and screw it into a terminal it can be done with just the knife.

[Mechatrommer]

this thread reminds me that one of the use of interactive debugger is to run through an external/3rd party/unknown library source code to see/study parameters/variables values or called procedures if i'm too lazy to chart them with paper and brain.. ymmv.

[Siwastaja]

Having to debug code by others which is brought to your project under the name of "library" sucks so hard. Normally you would assume that library is well tested and works correctly, and offers well documented interfaces.

If you really must debug library code, then really all the normal debugging ideas apply. If you don't have the source, it is indeed more appealing to look at the assembly code in debugger and utilize breakpoints/singlestepping, than to disassemble the library and start adding new instrumentation code, in assembly, within it.

But really, at that point you should be negotiating to acquire the source code for the library, and possibly further help debugging it (like further documentation and unit tests used by the original developers, etc.)

[pcprogrammer]

Or you find a solution around it if modifying the library is not possible.

Many years back I had to write asp code and basic script for a web server/page and there was a problem with dates in the basic script. Had to make my own code in basic script to overcome the problem.

But for sure, working on or with code from others can be a big pain in the bum.

[PlainName]

working on or with code from others can be a big pain in the bum

Can be, but it can also be very satisfying - there is to ego boost of having fixed an issue but without the guilty conscience of having written the bug in the first place

[tellurium]

Can be, but it can also be very satisfying - there is to ego boost of having fixed an issue but without the guilty conscience of having written the bug in the first place

Provided you didn't implant a few more, along with a fix

[Mechatrommer]

sometime its not a bug, its just that the original code doesnt work the way i want it but surely works for others. or sometime i just want to know how did they do it, how much memory is going to involve etc. the last time i did it on marlin code for 3d printer. interactive debugger isnt available in that retarded arduino ide so i have to do it by pencil and paper. otoh the funniest thing to me is that every program that i've made (for my own), and i mean every single one of them, when i want to upgrade later in like 3-10 years time is going to be like someone else's code, so i have to retrace and upgrade the code like one... so at some point of time ago, i know how much important the commenting is...

[gnuarm]

Know I not backwards thinking it is.  Actually, many people object to the one tiny bit of syntax it has, Reverse Polish Notation, RPN.  That is not entirely accurate, really.  The interpreter simply interprets a word at a time, rather than waiting for a line to be digested and turned into machine code.  So the operands are normally first. 

That's not always the case.  A string is created with a word S" which then scans the input for the next quote marking the end of the text.  Many words that create words, look for their input after the word. 

Or did you mean backwards in the Appalachian back woods sense?  I prefer a simple tool.  I'm not designing systems with 27 interrupt priorities.  I design systems to get the job done while making the software easy to debug.  I'd much rather use a screwdriver or hammer, than a pneumatic fastener tool.

I was hinting to the RPN way of how it operates.

Agree with you that a simple tool to do a simple job is the way to go. Always choose the right tool for the job, but sometimes you have to use what is at hand and try to make the best of it. For example if all you have is a knife and you need to strip a wire and screw it into a terminal it can be done with just the knife.

Yes, I much prefer to solve simple problems rather than complex problems.  That's why I try to create only simple problems.  I create designs without complex problems and are only left with simple problems.

Even simple problems can be hard to find if your methods are complex.  So simple debug methods are also preferred.

[MikeK]

I usually use very simple methods of debugging, like turning on an LED.  But I also made a simple version of this a while ago.

[nigelwright7557]

You can write loads of code then set about debugging it.
Set breakpoints and create watch variables to see what is happening.
LED's on output pins and use of an oscilloscope if timings are important.

I tend to use incremental debugging.
I write a small module of code then fully debug it before going onto next part.
Its much easier to debug a small piece of code.

[gnuarm]

You can write loads of code then set about debugging it.
Set breakpoints and create watch variables to see what is happening.
LED's on output pins and use of an oscilloscope if timings are important.

I tend to use incremental debugging.
I write a small module of code then fully debug it before going onto next part.
Its much easier to debug a small piece of code.

That's one of the fundamental ideas behind programming in Forth.  It uses "words" (the Forth term for subroutines) which should be tested before moving on.  They are typically small, so easy to debug one at a time.

[Psi]

I tend to use incremental debugging.
I write a small module of code then fully debug it before going onto next part.
Its much easier to debug a small piece of code.

Same. 
I write a little code, compile, run, check, tweak as needed. Then move on to the next block.
The size of the block I write before checking depends on how I feel at the time. I need to be able to hold the entire thing in my head.
I only use the debugger when something doesn't make sense.
Usually that ends up being a memory leak or corruption which doesn't occur when I use the debugger on it. Which only reinforces my view that the debugger is annoying and to never use it.
hehe.

I've seen some real wizards at using the debugger though,  it's an awesome tool once you master it.

[paulca]

+1 for logging. 

Flash memory for the log strings is probably the only cost.

You can, if you wish use a MACRO instead of a function, say:

DEBUG_LOG("Trying to Init UART2");

Which has a build time conditional flag to turn it on, or off.  When it's off the code for the logging isn't even compiled.

Log is W5.  Who, What, Where, When, Why?

Always put yourself into the position of some poor bugger tasked with fixing your code late at night, hunched over a laptop on a data centre floor.  What would they like to see in the logs?

Using a debugger often helps you too much and makes you leave out useful logging in tricky error prone sections.  If you had to have debugged it without one, you probably would have added debug diagnostics in them which would have helped others later if it had more bugs.

(I got angry a few weeks back because I had a framework failing with an un caught error like, "File is a directory" without telling me which one.  I looked at the code and got very angry.  The code was a mess, over complicated non-sense, it was also littered with about 60% commented out debug logging code.  Commented out.  It was very clear that, that section of code was error prone and fragile, so why the hell did they comment out all the logging and not add any error handling at all?  Argh!

Of course the downside on MCUs can be that the UART doing the debug logging is using up too much demand for "real time" in it's event handlers.  If you are trying to process timing critical streams/buffers in your main code, the debug UART code needs to be written very carefully.  No Serial.println("....");  and blocking calls.

In UART coms I have found the debugger is only helpful on the first fire.  As soon as you finish with that ITRS debug, the state will be corrupted the buffers overrun etc. etc.

[AVI-crak]

Debugging is boring. In one form or another, it is always a search for a program failure. An occupation as necessary as, for example, making money every day, especially when you are hungry.
It is much more interesting to polish the code to the ideal version.
You can turn on the optimization from "-O0 to "-Os" or "-Og", and calm down with a sense of accomplishment. Or you can change the algorithm so that it performs the same functions, but at the same time works 10 times faster, or takes 100 times less space.
Polishing can be done indefinitely, as long as there is a code for this.

I use the online compiler https://godbolt.org/ for small functions. It is not designed to run programs. It's for crazy people - who have already looked at their code 1000 times in step-by-step mode, and remember every instruction in assembler. There, any change gives an instant result.

[PlainName]

Debugging is boring

I would suggest that needing to debug is annoying, but finding and fixing a problem can be satisfying in the way soduko or wordle or mosaic can be. Or, indeed, polishing the turd until it looks shiny and goes like shit off a shovel, if that's what floats your boat.

Generally, debugging is a pain because you were intending to do something else and instead got a reminder that you're not so clever after all.

[gnuarm]

Debugging is boring

I would suggest that needing to debug is annoying, but finding and fixing a problem can be satisfying in the way soduko or wordle or mosaic can be. Or, indeed, polishing the turd until it looks shiny and goes like shit off a shovel, if that's what floats your boat.

Generally, debugging is a pain because you were intending to do something else and instead got a reminder that you're not so clever after all.

One of my early jobs was on the test floor of an array processor company.  At that time, it was essentially a supercomputer, now it wouldn't even be a floating point co-processor. 

I really liked the work.  Every machine was a murder mystery that needed to be solved, even if it had more than one death and villain.  Pretty cool when you think of it.  Often the people in test have to be smarter than they guys who design the stuff.  I recall a schematic for the part of the machine with custom ECL gate array chips had a block with no indication of what was inside, just a note that said, "Who knows what this does?" 

I'm not always that intuitive, and could get wrapped around the axle on some problems only finding the problem after multiple blind alleys.  But that's typical for a murder mystery, right? 

One guy on the floor had been a bus boy in a restaurant nearby that the head guy hired.  He took notes on everything he did, and managed to become one of the best guys out there.  He barely spoke English even.  I was very impressed. 

I work for myself now.  So I do the design work and then support the contract manufacturer to get the units working.  I don't do much debugging myself, but now I get to try to make the debugging as easy as possible.