How to debug microcontrollers in the "real world"?

[ksoviero]

I've been using Arduinos for a while now, but I want to start using AVRs directly without the "crutch" of Arduino.

One of the issues I've run into is debugging my programs when programming them with an ISP. How does one get feedback when using an ISP, they don't have any kind of serial interface.

For example, with Arduino, I'd just print stuff to the screen using the serial interface and that would give me visibility into what it's doing. How would I replicate that workflow with an ISP?

Thanks!

[Bassman59]

I've been using Arduinos for a while now, but I want to start using AVRs directly without the "crutch" of Arduino.

One of the issues I've run into is debugging my programs when programming them with an ISP. How does one get feedback when using an ISP, they don't have any kind of serial interface.

For example, with Arduino, I'd just print stuff to the screen using the serial interface and that would give me visibility into what it's doing. How would I replicate that workflow with an ISP?

Thanks!

Modern microcontrollers come with JTAG or SWD (for ARM) or C2 (for SiLabs micros) or other debug interfaces which connect to a debug pod which connects to your computer, and the computer runs a source-level debugger which allows setting breakpoints, watching registers and variables and all sorts of goodies.

We used to use "print to terminal" back when that was all that was available.

The biggest fail of Arduino is not exposing the micro's debug port.

That said, I still toggle port pins in ISRs and monitor them on an oscilloscope.

[Bassman59]

I still have scars from the Nohau 8051 bond-out pod emulator.

[rstofer]

It's just the same as with the Arduino - except you have to do the work.  There are two approaches:  write all of your own code (I prefer to do this) or use avr-libc.

Own code:

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

Write some hex output functions that will send hex character strings for 32 bit, 16 bit, 8 bit and 4 bit entities, with or without the leading 0X.  Now you have a way to send messages containing hex results of whatever you want along with strings.

Now, open your copy of "The C Programming Language" (Kernighan and Ritchie), ANSI edition, and copy the code for reverse() and itoa().  Now you can display integer values in messages.

While you're there, copy a bunch of string functions.

I'm not sure but I think the ANSI edition has the same code as the original edition.  In other words, the string functions don't use a heap and I don't want to implement a heap.

Use avr-libc:

https://www.nongnu.org/avr-libc/user-manual/group__avr__stdio.html

There is a bunch of interface code to write and it's much more complex than just rolling your own.

[mikerj]

Get yourself an Atmel ICE and experience the power of proper source level debugging.  By Arduino standards it might look expensive, but by professional debugger standards it's very cheap. 

A budget solution might be the ATmega328P Xplained board which has an integrated Atmeol Studio compatible debugger, though I've never used it.

The dowside of these is the number of pins required on a device that already starts with relatively few pins.

[SilverSolder]

Just write error free code -  what's the problem?!?   

[Sal Ammoniac]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

[rstofer]

Just write error free code -  what's the problem?!?   

I came to that realization about 50 years ago when I was writing FORTRAN for the IBM 1130 and CDC 6400.  If I didn't put bugs in my code, I wouldn't have to spend time digging bugs out of my code.  Alas,  it hasn't always worked out...

[coppice]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

25 years ago most devices had some form of JTAG for test purposes, but didn't offer debug through this. Certainly these days hardware debug is pretty much standard, and some kind of built in logic analysis is quite common, and should be used where possible. It doesn't displace things like printfs, though. Good debug strategies use a mix of approaches to debug at different levels.

[TK]

You need a combination of HW debugging with a logic analyzer and oscilloscope and source level debugging with an ICE.  If your project involves interrupts / timers, it is hard to rely only on an ICE.  You can do single step execution but I did not find it to be friendly when you have external interrupts and timers.

[ejeffrey]

I've been using Arduinos for a while now, but I want to start using AVRs directly without the "crutch" of Arduino.

One of the issues I've run into is debugging my programs when programming them with an ISP. How does one get feedback when using an ISP, they don't have any kind of serial interface.

For example, with Arduino, I'd just print stuff to the screen using the serial interface and that would give me visibility into what it's doing. How would I replicate that workflow with an ISP?

Thanks!

I agree with most, get a source level debug interface working.  That said if you have a spare serial port, definitely configure it as a logging output.  Without the arduino libs you will need to spend a bit of effort to hook that up but it isn't too hard.  I really like to have both whenever possible.

ARM has some nice options to send a serial data stream over the debug interface which is nice and doesn't take up any of your pins.  I don't know if AVR supports anything like that.  ARM has at least 3 options: semi-hosting is slow and bloated but very flexible.  SWO is fast and lightweight, but not widely supported.  Segger has their RTT thing which is fast but proprietary.  It also requires a dedicated memory buffer and can drop data.

[MK14]

Just write error free code -  what's the problem?!?   

Every piece of software by me, has worked first time, and perfectly. (Honest).

The problem I haven't solved, is why the original software specification, suddenly mysteriously (as if by magic), changes, as soon as I first run the software.

E.g. I thought the software was suppose to do, this and that. But, the original spec changes to "It should produce 153 Compile errors. 107 Warnings", and when those are fixed it should crash and produce no output, in such a way that it takes many hours to figure out why.

[rstofer]

Here is the collection of functions I described above.  This was for an ARM project I did several years ago but the idea still works.  Get the uC sending stuff to the serial port and bury debugging statements throughout the code.  I don't really like single stepping even with a really nice source level debugger.  It just doesn't seem productive.

I can surround the debug statements with #ifdef DEBUG ... #endif as I wish.  I can bury the equivalent of ASSERT statements within the #ifdef DEBUG ... #endif block.  I can turn DEBUG off and things should work well without the annoying output.

[SilverSolder]

[...]
I don't really like single stepping even with a really nice source level debugger.  It just doesn't seem productive.
[...]

Single stepping is OK for assembler, but only occasionally useful with higher level languages, in my experience.

[ksoviero]

Get yourself an Atmel ICE and experience the power of proper source level debugging.  By Arduino standards it might look expensive, but by professional debugger standards it's very cheap. 

A budget solution might be the ATmega328P Xplained board which has an integrated Atmeol Studio compatible debugger, though I've never used it.

The dowside of these is the number of pins required on a device that already starts with relatively few pins.

Are you referring to this? https://www.digikey.com/product-detail/en/microchip-technology/ATATMEL-ICE/ATATMEL-ICE-ND/4753379

[TK]

Get yourself an Atmel ICE and experience the power of proper source level debugging.  By Arduino standards it might look expensive, but by professional debugger standards it's very cheap. 

A budget solution might be the ATmega328P Xplained board which has an integrated Atmeol Studio compatible debugger, though I've never used it.

The dowside of these is the number of pins required on a device that already starts with relatively few pins.

Are you referring to this? https://www.digikey.com/product-detail/en/microchip-technology/ATATMEL-ICE/ATATMEL-ICE-ND/4753379

There are cheaper options like buying only the PCB if you can make 1.27mm pitch to 2.54mm header cables.  It is called Atmel ICE PCBA or they also have a Basic option that comes with the case but not many adapters, only the basic cable

[ksoviero]

Get yourself an Atmel ICE and experience the power of proper source level debugging.  By Arduino standards it might look expensive, but by professional debugger standards it's very cheap. 

A budget solution might be the ATmega328P Xplained board which has an integrated Atmeol Studio compatible debugger, though I've never used it.

The dowside of these is the number of pins required on a device that already starts with relatively few pins.

Are you referring to this? https://www.digikey.com/product-detail/en/microchip-technology/ATATMEL-ICE/ATATMEL-ICE-ND/4753379

There are cheaper options like buying only the PCB if you can make 1.27mm pitch to 2.54mm header cables.  It is called Atmel ICE PCBA or they also have a Basic option that comes with the case but not many adapters, only the basic cable

Oh, I don't mind spending money. $140 is nothing in the grand scheme of things. Especially in this hobby. I was just making sure that was the right one.

[mikerj]

That's the correct one.  It will also work with the SAM 32 bit ARM Cortex parts when you decide you need a bit more horsepower.

[...]
I don't really like single stepping even with a really nice source level debugger.  It just doesn't seem productive.
[...]

Single stepping is OK for assembler, but only occasionally useful with higher level languages, in my experience.

Being able to single step through source is invaluable IMO.

[tggzzz]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

When I came back to embedded programming 5 years ago, I was horrified and delighted about now quickly I picked it up at again.

1982: 8 bit micros with ADCs and DACs, C, cross compilation plus download and run, in circuit emulators (breakpoints, single stepping same as JTAG). The only significant difference is that back then the tools would have cost a years salary!

But the key point is to learn to use whatever tools are available. Printf or waggle an I/o pin are always available, and are one of the tools in a toolbox.

[ksoviero]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

When I came back to embedded programming 5 years ago, I was horrified and delighted about now quickly I picked it up at again.

1982: 8 bit micros with ADCs and DACs, C, cross compilation plus download and run, in circuit emulators (same as JTAG). The only significant difference is that back then the tools would have cost a years salary!

But the key point is to learn to use whatever tools are available. Printf or waggle an I/o pin are always available, and are one of the tools in a toolbox.

Tiny bit confused: How is printf "always available"? Don't you need serial or something set up to be able to see the output?

[rstofer]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

1976 is precisely when I started debugging 8080/8085 systems with either the front panel toggle switches or the Digital Research DDT debugger.  Prior to that I got core dumps on the 1130 - that was truly ugly.  I don't recall the details but the IBM 1130 had a TRACE option for FORTRAN - you could get a printout of the branches.  That was kind of handy with 3 way branches (-, 0, +).  Next time I'm playing with the machine (my FPGA incantation), I'll have to look into how tracing worked.  It's been about 48 years and my memory is fading fast.

I've used source level JTAG debugging although I will admit that I prefer the Rowley toolchain and dongle to anything where I have to try to configure OpenOCD myself.  I haven't messed with it much in the last several years but I NEVER had any success getting it to work.  Not EVER!

Maybe that's why I gave up on JTAG debugging - I simply couldn't get it to work.  UART output ALWAYS works (as long as I have a couple of pins).  I don't have to worry about gdb compatibility, I don't have to mess with Eclipse integration, I can just dump some print statements here and there and trace the changes in critical values.

The new boards, like the STM32 devices, have very nice debugging capabilities.  The online mbed toolchain has none - back to printf(). Toggling an LED (which I can capture on a scope) or dumping text crumbs works well if the other methods fail.

It's best to have an assortment of tools.

[tggzzz]

First thing:  Set up the clocks/dividers and then get the UART to send a character by writing the code for putchar(). Now write the code for puts() using putchar().

You've just described debugging circa 1976. The world has moved on since then and now uses on-chip debugging aids like JTAG to do source-level debugging. This is the professional standard--every company I've worked for in the last 25 years has done it this way. Sure, there are still a few grizzled holdouts that insist on debugging by sprinkling printfs in their code, but that's not the norm.

When I came back to embedded programming 5 years ago, I was horrified and delighted about now quickly I picked it up at again.

1982: 8 bit micros with ADCs and DACs, C, cross compilation plus download and run, in circuit emulators (same as JTAG). The only significant difference is that back then the tools would have cost a years salary!

But the key point is to learn to use whatever tools are available. Printf or waggle an I/o pin are always available, and are one of the tools in a toolbox.

Tiny bit confused: How is printf "always available"? Don't you need serial or something set up to be able to see the output?

That is one way. It is easy to create a 1-pin UART using bit banging, if a real one isn't available.

There are variants, if you think about it. One is to create a formatted string in the target memory, then stop on breakpoint after the string is created. Or create an in-memory log, and write very terse messages there.

All you do is a little research into whatever techniques other people are using in the environment. Then apply imagnation to make best to see of them.

[MK14]

Tiny bit confused: How is printf "always available"? Don't you need serial or something set up to be able to see the output?

You're right. You do indeed need serial or something, to be able to see the output.
In extreme circumstances (no other equipment available, and mcu doesn't even have any serial capabilities), you can bit-bang (i.e. write software which turns the port pins on and off, as if you had a proper serial capability), and then read it with an oscilloscope or some kind of serial adaptor.

The printf method, is useful, up to around 90% of time, depending on programmers experience level. But, 10% or more, the bug(s), can easily make that method very inefficient. Especially when the software mysteriously crashes, in random places, or seems to have weird data corruption etc.

You are likely to learn far more, if you can see the source code, running. Line by line, and be able to examine ram (variables), to see what values they have. Not to mention the hardware I/O registers and things, which can easily get confusing.

E.g. You setup the PWM to output a 2 KHz signal, and nothing happens.
Why ?
Being able to see the registers getting the right values, and checking/watching the software, helps narrow down, what has gone wrong.

[nctnico]

Tiny bit confused: How is printf "always available"? Don't you need serial or something set up to be able to see the output?

You're right. You do indeed need serial or something, to be able to see the output.
In extreme circumstances (no other equipment available, and mcu doesn't even have any serial capabilities), you can bit-bang (i.e. write software which turns the port pins on and off, as if you had a proper serial capability), and then read it with an oscilloscope or some kind of serial adaptor.

The printf method, is useful, up to around 90% of time, depending on programmers experience level. But, 10% or more, the bug(s), can easily make that method very inefficient. Especially when the software mysteriously crashes, in random places, or seems to have weird data corruption etc.

You are likely to learn far more, if you can see the source code, running. Line by line, and be able to examine ram (variables), to see what values they have. Not to mention the hardware I/O registers and things, which can easily get confusing.

E.g. You setup the PWM to output a 2 KHz signal, and nothing happens.
Why ?
Being able to see the registers getting the right values, and checking/watching the software, helps narrow down, what has gone wrong.

The latter can easily be implemented by having a 'mem' command together with a command line interface.

Stepping through code sounds nice but for realtime processes (like on a microcontroller) it runs out of steam quickly. What I have been doing for the past 20 years:
- have a command line interface on a UART
- have status commands which shows stuff like character received, number of interrupts
- if available, implement hard fault handlers and make these print a stack dump
- add extra commands to query status or dump memories (think about an eeprom for example; how can you be sure the correct data is written to it?)
- write complicated routines on a PC first + use a test bench to verify
- allow to dump measured and status data in CSV format for analysis

Once you have a basic command line interface going you have a very simple eco system which is portable as well. There are debuggers which allow to read variables realtime over JTAG/SWD but these aren't very cheap. And they won't do you any good solving problems in the field. If you have a problem in the field (IOW: a board mounted somewhere in a machine far away from you) you can ask a field engineer to connect to the serial port and query the status of the device or just dump data.

[MK14]

The latter can easily be implemented by having a 'mem' command together with a command line interface.

Stepping through code sounds nice but for realtime processes (like on a microcontroller) it runs out of steam quickly. What I have been doing for the past 20 years:
- have a command line interface on a UART
- have status commands which shows stuff like character received, number of interrupts
- if available, implement hard fault handlers and make these print a stack dump
- add extra commands to query status or dump memories (think about an eeprom for example; how can you be sure the correct data is written to it?)
- write complicated routines on a PC first + use a test bench to verify
- allow to dump measured and status data in CSV format for analysis

Once you have a basic command line interface going you have a very simple eco system which is portable as well. There are debuggers which allow to read variables realtime over JTAG/SWD but these aren't very cheap. And they won't do you any good solving problems in the field. If you have a problem in the field (IOW: a board mounted somewhere in a machine far away from you) you can ask a field engineer to connect to the serial port and query the status of the device or just dump data.

That sounds like a very good way, as well!

Some people, actually implement, what amounts to small/light resource scripting/interpretive languages, such as tcl, lua or Forth etc. Allowing quick examination of memory (like you described), or even running small scripts, to check out I/O functionality.

But, there are (thankfully rare), highly complicated bugs, which can be very difficult, without the right debugging equipment.
But, yes. You are right. Once you are experienced enough at this, a full/proper debugger, is only very rarely needed.

Your method, as described, can even handle most or even all, of what I just said, as well. if done well enough.

But, if it was a very big and important project. I would still like to see proper debugging equipment used, where possible. But, as you say, it sometimes isn't possible or practicable.
E.g. A remote controlled, model boat project, where it will be in the water, 50 metres away, sailing at speed, when you are trying to debug the software. A 50 metre long ribbon-cable, would probably sink the boat, and collide with other lake users.