Debugging Techniques

[SimonR]

As I was commenting on debug techniques in the "Alternative to CooCox for STM32 development" thread it occurred to me that not only was it off topic but that it might be useful to pass on some of my experience in the hope it might help someone else.
So I started this thread so people could share some of their more unusual but never the less useful debugging tips.
To start you off I thought I would share the low level trace code I mentioned in the other thread.

A few years ago I worked with a team on an ASIC design. The device had three CPU cores but absolutely no debug facilities. No JTAG, no nothing bar a single UART per core.
What it did have however was a really crude hardware trace shared between the cores.

This took the form of 64 words of RAM accessed by a single register and it worked like this.

every time you wanted to record an event in your code you wrote a value to the register.
Then whenever an ASSERT was triggered the whole trace buffer was dumped to the primary CPU UART if enabled.
This had very little impact on the speed of the code and you can put what you like in the trace, which in my case always seemed to be pointers to buffers so I could see what order they were accessed.

Just recently I needed a similar thing as I don't have trace in my JTAG debugger but I need to have low impact because is was looking at an ISR problem.

So I wrote a very quick and dirty version of the ASIC hardware trace in software. Its not polished in any way at all and there are features missing but here it is if want to use it or modify it.

It may need C99 as it is, because it inlines for speed, but you should be able to suitably modify it, and pehaps even run it on an 8 bit cpu if you have any ram to spare.

Here is an example of how to use it.

void LowlevelTest(void)
{
    uint32_t data;
    LowLevelTrace_Init();
    do{
        data = FunctionToBeDebugged();
        LowLevelTrace_SetData(data);
    while(result);
   
    while(LowLevelTrace_GetData(&data))
    {
        printf("%lu\n", data);
    }
}

Note you must have  USE_SW_TRACE defined before you can use the trace facility


Header File

Code: [Select]

/**
 * \project     Debug Facilities
 * \file        SWTrace.h
 * \brief       Public include for Software Trace top level definitions.
 * \defgroup    Software_Trace_module Low Level Trace
 */
#ifndef __SW_TRACE_H__
#define __SW_TRACE_H__
/*
* Includes
*/
/* Standard headers */
#include <stdint.h>
#include <stdbool.h>

/* Library headers */
/* Dependencies */

#ifdef __cplusplus
extern "C"
{
#endif /* __cplusplus */

/**
 * \addtogroup Software_Trace_module
 * \{
 */
/*
 * Constants
 */

/** \brief number of data entries in a trace data object
 *  \pre must be an exact binary number as we cheat with the wrap calulation to save time
*/
#define TRACE_DATA_SIZE     (64)

/** \brief Mask used in the wrap calulation */
#define TRACE_DATA_MASK     (TRACE_DATA_SIZE - 1)

/*
 * Type definitions
 */

/**
 * \brief Low level trace object
 * \details contains a circular buffer of 32 bit words
 *
 */
typedef struct LowLevelTrace_s {
    uint32_t                 buffer[TRACE_DATA_SIZE];
    volatile unsigned int    head;
    volatile unsigned int    tail;
} LowLevelTrace_t;

extern LowLevelTrace_t TraceObject;

/*
 * Public prototypes
 */
extern void LowLevelTrace_Init(void);

/*
 * Public Inlines
 */

static inline void LowLevelTrace_SetData(uint32_t data)
{
#ifdef USE_SW_TRACE
  unsigned int next = (unsigned int)(TraceObject.head + 1) & TRACE_DATA_MASK;
  if (next != TraceObject.tail)
  {
      TraceObject.buffer[TraceObject.head] = data;
      TraceObject.head = next;
  }
#endif
}

static inline bool LowLevelTrace_GetData(uint32_t *data)
{
#ifdef USE_SW_TRACE
    // if the head isn't ahead of the tail, we don't have any entries
    unsigned int head = TraceObject.head;
    unsigned int tail = TraceObject.tail;

    if (head == tail)
    {
        return false;        // quit with an error
    }
    else
    {
        *data = TraceObject.buffer[TraceObject.tail++];
        TraceObject.tail &= TRACE_DATA_MASK;
        return true;
    }
#else
    return false;        // quit with an error
#endif
}

/**
 * \}
 *//* addtogroup Software_Trace_module */

#ifdef __cplusplus
}
#endif /* __cplusplus */

#endif /* __SW_TRACE_H__ */
/**********************************************************************\
* End of file SWTrace.h
\**********************************************************************/

C file

Code: [Select]

/**
 * \project     Debug Facilities
 * \file        SWTrace.c
 * \brief       functions to define a Low Level Trace
 */
/*
* Includes
*/
/* Standard headers */
/* Library headers */
/* Dependencies */
#include "SWTrace.h"

/*
* Constants
*/
/*
* Type definitions
*/
/*
* Private prototypes
*/

/*
* Private data
*/
LowLevelTrace_t TraceObject = { {0}, 0, 0 };

/**********************************************************************\
*       Public functions
\**********************************************************************/
void LowLevelTrace_Init(void)
{
    TraceObject.head  = 0;
    TraceObject.tail = 0;
}

/**********************************************************************\
*       Private functions
\**********************************************************************/
/**********************************************************************\
*   End of file SWTrace.c
\**********************************************************************/





[jnz]

I read that topic and was surprised to see anyone seriously saying that led blinking and printf was a replacment for an actual debugger!

[tggzzz]

I read that topic and was surprised to see anyone seriously saying that led blinking and printf was a replacment for an actual debugger!

Of course that isn't a substitute. You also need imagination, clear thinking, and a design process that takes account of the limited tools available.

[Jeroen3]

A live snprintf output stream is a cheap alternative when you do not have tracepoint availability. (debuggers with tracing are expensive)
Since breakpoints are not an option anymore when running real-life scenario tests on the complete products.
You know, those tests where a code halt means destruction of hardware.

[SimonR]

Of course that isn't a substitute. You also need imagination, clear thinking, and a design process that takes account of the limited tools available.

I agree totaly, printf is not a substitute but sometimes you have  no choice, as in my case with no debug port available.
And sometimes a debugger can't help at all. Which, as you quite rightly say, is where the imagination and clear thinking come in.
A debugger is only one tool in a toolbox.

I thought I would start this to give people a few more tools. I searched  to whole forum for "Debugging" and most questions
are about specific problems on specific targets, but quite often the solutions arre universal an may useful for someone else.
But they are lost in a CPU specific thread.

I quite often see beginners ask questions in other forums only to get a reply that leaves far more questions than answers.
Having been on the receiving I hope that I won't do that in any advice I give.

So If you have a useful technique the please post it. NO matter how trivial you think it is, it may help someone, hopefully me.


 

[Kalvin]

Set the serial port to its maximum available baud rate and use a cheap logic analyzer to capture and decode the debug data. You should be get at least 1Mbit/s throughput. Tweak the printf so that it will ignore any fancy formatting for the debug output. If the CPU doesn't support the DIV/MOD instruction, consider outputting the numeric data in HEX.

For a better user experience make another CPU to capture the fast (and possibly compressed) serial data and convert the strings and the numeric HEX data for user friendlier serial debug output running at 115200 bits/seconds. With some buffering the baud rate mismatch should not be a problem. 

[Pack34]

My go-to is to send out hex flags as appropriate using 0xff as a start byte. That way at 115,200 it doesn't really slow down the program at all since it's just two bytes out a serial port and then I have some flags into what's going on.

I've never had tremendous luck with using a debugger other than to verify some initialization. Everything I've ever worked with required real-time operation to work and a debugger slows it down too much.

[dannyf]

On general strategy towards debugging a piece of code is this:
1. Try not to debug it. I try to reuse proven code pieces as much as I can to minimize the need of debugging and to speed up development. It solves 90 percent of my troubles.
2. Use modular code pieces: if I want to work with a module, either software or hardware, I go thought the same set of routines. So obcebi get those routines right, they are right everywhere else.
3. Debug as much off line as you can: If I can debug on a PC, I debug on a PC. If I can debug on a micro with trace capabilities, I debug there. If I can debug on a micro with serial port, I debug there, .... Try to keep it as high-level and as visual as you can.
4. As a last resort, I typically debug indivual code pieces and then integrate them into the final project.

So a typical project of mine may contain a multitudes of vendor libraries, my own library, chip / device specific header and source code and user / application specific files. My template for cortex m projects may take a couple minutes to compile for example.

This approach has worked well for me.

[Kalvin]

I use different print-functions for different purposes: printf, print_boot, print_fatal, print_warn, print_error, print_info, print_log etc. Those are basically redefined macros for the printf function which can be enabled and disabled globally in the project configuration file or locally per module. When they are disabled, they do not generate any code.

[hans]

A debugger is a nice tool, but sometimes completely impractical. My scenario's for that have been:

- Debugging protocols or interfaces to them (timeout's etc.)
- Systems that can't miss any interrupts (in particular ones that are generated externally)
- Systems where a halt will make electrical smoke

Trace is nicer on real hardware. I once wrote a trace utility with focus of avoiding printf() in code, but still be able to trace serial messages with arguments in them.
For that I wrote a C macro table with string printf message and an argument list by types. Only data on target was a __COUNTER__ message ID and argument list .

Then in code a trace function is called, which takes the message ID and va_arg list, and print the corresponding argument list over serial in binary. The trace function can be very quick, because it's only moving raw bytes from va_args to a buffer. The data was circular buffered, and when CPU time was free the buffer is dumped over serial @ 750k - 1MBaud.
On the other end I had a simple FT232RL cable. PC ran a small script that also parsed the C-macro configuration file, and did the printf() on the PC.
Smallest trace was ~4 bytes of traffic (included some sync's), which isn't great in particular, but e.g. when printing 8x 16-bit integers it would only grow up to 20 bytes (instead of ~44 bytes in ASCII with no human formatting, plus lots of printf time)

For protocol debugging it's very nice if you can simulate the high-level protocol in a simulator, and write some unit tests for the code. For my home IoT network I made a complete "RF environment" down to the MAC level, and could instantiate multiple nodes at once. A nice bonus: when I halt the code ALL threads are halted and I can look at the state of each particular node.

Spreading assert() around your code at sane points is also quite nice. It only costs a little/no CPU time, can easily be turned off (just redirect assert macro to a void expression), and you get instant alerts about your code breaking at some points. It also makes you think about the range of arguments a particular function can support.

[mikerj]

I've never had tremendous luck with using a debugger other than to verify some initialization. Everything I've ever worked with required real-time operation to work and a debugger slows it down too much.

A hardware debugger shouldn't slow the CPU down, which target device are you using?

On the ARM Cortex parts I find a debugger invaluable, particularly when using an RTOS.  Nothing else really gives you the capability of examining registers,  data structures or arbitrary blocks of memory without any additional code space or execution time overhead, and the serial wire viewer can also be used as a UART to send debug data.

[legacy]

A hardware debugger shouldn't slow the CPU down

yes, they do, for a specific reason: the debug engine needs to collect a lot of informations which needs to be transmitted to the host
thus, or you use a super fast link (e.g. high USB full speed, 20-40Mbyte/sec) or you need to slow down your CPU clock

it's called "debug mode", "release mode"

it happens if you don't have the money to buy a super fa$t debugger

[Sal Ammoniac]

I've sometimes had to resort to such guerrilla debugging techniques when doing firmware development professionally when I've had the choice of MCU and development tools dictated to me by management.

Now as a hobbyist, I don't have those problems. I just don't use MCUs and development environments that don't have good debugging facilities. I consider J-TAG and the ability to set breakpoints and single step code to be the minimum requirements. Trace is fast becoming a must-have for me too, even though the debugger tools typically cost more than simpler tools--even though I'm a hobbyist my time is still worth something to me.

[legacy]

@Sal Ammoniac
yes, I agree

here I am still developing my debug-enging, well improving it
it offers very basic features for a soft core (it's a pure VHDL project, plus C on the host side)
it might be modified as one wishes, including the tracing support, which implies a large bandwidth and a deep memory

anyway, ah, if my 68000 had ever had such a features 
on that 68k board, I am still using a GDB-stub, it's assembly code plus a few tricks

[SimonR]

I'm really intrigued by the VHDL solution, sounds interesting.

I know what you mean about the 68000

When I did my industrial placement for my degree back in 1985 I had to do some 68K development. There we had a Microtec MICE in circuit emulator so I had all of the facilities of a mordern debugger and a really good trace facility. No source level debugging though. I dread to think how much it cost, and they had 3 of them.

Then when I got my first full time job we only had an assembler and an EPROM programmer, what a come down

[ale500]

Then when I got my first full time job we only had an assembler and an EPROM programmer, what a come down

We didn't have fancy debuggers either, it was 1994/1995... UARTs at 9600... I used one of those text LCD (2x40) to help debug... we didn't have simulators either... we had plenty of time, though...