Using HW debuggers

[jerry507]

This came up in a recent post. nctnico suggested that half developers don't use HW debuggers. This caused me to  , but it started an interesting conversation I think. So do you use debuggers and why/why not?

[legacy]

the best hw debugger i have tried is the Analog Devices ADI solution for Blackfin
it is very cool, and efficient, so there are no reasons not to use it 

[free_electron]

those are the half that call themselves 'embedded developers'. just because you can write some code on a pre-packaged embedded os doesn't mean you know you are an embedded developer.

Here's a lump of silicon and a datasheet . You get to write a taskswapping microkernel and code up a library that implements Modbus for example, or a bluetooth stack. from scratch.

Good luck if all you have is printf ...
you will at least need a JTAG debugger. and if you need to do anything with interrupts : good luck with a software debugger. Same if you got to do code profiling and optimisation. You can't even have any 'debug code' in there. it's the hardware way or nothing..

[dannyf]

half developers don't use HW debuggers.

The other half are wannabe HW developers,

You cannot take everything you read on the internet seriously.

[Rasz]

a bluetooth stack. from scratch.

nobody does that, because BT stack is quite retarded
someone might implement ONE small subset, but not a whole thing

[Howardlong]

a bluetooth stack. from scratch.

nobody does that, because BT stack is quite retarded
someone might implement ONE small subset, but not a whole thing

Ain't that the truth. Same goes for USB, IP stack etc, you'd be nuts to want to do the whole of those yourself from scratch, and produce a real product too, although I am sure there are some who have.

[lewis]

you will at least need a JTAG debugger. and if you need to do anything with interrupts : good luck with a software debugger. Same if you got to do code profiling and optimisation. You can't even have any 'debug code' in there. it's the hardware way or nothing..

Nonsense! You can get anywhere you need to with a single LED 

[miguelvp]

half developers don't use HW debuggers.

The other half are wannabe HW developers,

You cannot take everything you read on the internet seriously.

Nah, the other half uses GHS Multi and it's horrible UI and debugger, so they are better off outputting log files.

http://www.ghs.com/Benchmarks.html

[westfw]

A hardware debugger is no substitute for having good high-level debugging tools in your embedded device.  Including printf().

High level debugging tools are no substitute for having a hardware debugger available, when you really need the latter.

Some of this depends on how much capability your hardware debugger has beyond that available via SW debugging.  Accessing a 68k or x86 class machine via a software gdb stub, over serial, over a network, is frequently not much worse than accessing a machine via a gdb bridge to a hardware debugger.   But other machines make a good software debug stub nearly impossibles, and newer chips have debugging capabilities that may not be available via "one-size-fits-all" debugging abstractions.  (OTOH, such advanced features tend to be available to software debugging tools as well.)

A lot of serious debugging is done by instrumenting your software with additional software, and distilling events into human-readable form (whether that's a pin toggle, printf, or something else.)  Conventional non-intrusive debugging, via either hardware or software, isn't close enough to 'real time' to be adequate.

[jerry507]

Indeed, I would consider all of the above as required.

I don't honestly know how people get by without a modern jtag debugger. The ability to stop and view a variable's value instantly is not only useful but also saves a lot of debugging time. And considering the price of the devices is pretty low and you're probably loading code already with one, it's surprising that anyone wouldn't have one. I've heard people say they can't use them because of space constraints or because final hardware won't have one, but these all seem like pretty poor excuses. If you really do have that issue, make a development prototype. The time saved makes it an obvious choice.

But a jtag debugger can't tell you squat about run time characteristics. Trying to get task run times in a rtos is pretty important and the debugger often can't provide that, even over a rtos provided plugin. So you're back to instrumenting code and printing to the serial port.

I guess my main question is, if you're NOT using a HW debugger (jtag, proprietary like microchip ICD etc), how does your debug chain work?

[BradC]

Trying to get task run times in a rtos is pretty important and the debugger often can't provide that, even over a rtos provided plugin. So you're back to instrumenting code and printing to the serial port.

Or having the task toggle your debug led and monitoring that with a CRO so you can time the routines.

I guess my main question is, if you're NOT using a HW debugger (jtag, proprietary like microchip ICD etc), how does your debug chain work?

First step is always to write or port a really good serial monitor, then you can just instrument as you go. You'd be surprised at how much you can learn about code performance monitoring a debug led with a cro, or even putting it into an amplifier and listening to it. The brain is very good at filtering and isolating audio anomalies. I use my audio monitor (just a small amp with a croc clip and meter probe on it) more than my logic probe these days.

I guess it helps to have been brought up in a world before hardware debuggers and being able to single step code. You learn to make do very well with extremely rudimentary tools. Not that I'd give up my debuggers now, but I'm more often still faster using basic tools and deductive logic.

[westfw]

before ... being able to single step code.

software debuggers supported single-stepping code on the 8080, on mainframes I used in the 70's, and on many microprocessors since then.   Although I guess it's a missing feature on a lot of the popular microcontrollers.

[coppice]

before ... being able to single step code.

software debuggers supported single-stepping code on the 8080, on mainframes I used in the 70's, and on many microprocessors since then.   Although I guess it's a missing feature on a lot of the popular microcontrollers.

They supported single step for code in RAM. If you wanted to single step through non-volatile memory you needed hardware support even then.

[Brutte]

But a jtag debugger can't tell you squat about run time characteristics.

ARMv7M comes with performance counters in DWT. These are not only core-readable but can be sniffed by a debugger, also run-time. And DWT can also do profiling (statistical mostly). And of course there is usually a bunch of hardware timers/counters on-board that can be clocked whenever some IO is high or toggles and freeze when uC steps on a breakpoint. Those in STM32 can even count quadrature encoder when uC is in halt mode.

[nctnico]

those are the half that call themselves 'embedded developers'. just because you can write some code on a pre-packaged embedded os doesn't mean you know you are an embedded developer.

Here's a lump of silicon and a datasheet . You get to write a taskswapping microkernel and code up a library that implements Modbus for example, or a bluetooth stack. from scratch.

If you are going to develop a protocol stack on an embedded platform you are insane. For that you use a PC. The same goes for any high level software. What remains are a small number of functions which deal with the hardware.

[jerry507]

But a jtag debugger can't tell you squat about run time characteristics.

ARMv7M comes with performance counters in DWT. These are not only core-readable but can be sniffed by a debugger, also run-time. And DWT can also do profiling (statistical mostly). And of course there is usually a bunch of hardware timers/counters on-board that can be clocked whenever some IO is high or toggles and freeze when uC steps on a breakpoint. Those in STM32 can even count quadrature encoder when uC is in halt mode.

That's good to know! Unfortunately I end up spending life in Cortex M3's and M4's mostly. But maybe I should dig into the user manual between compiles to see if NXP added anything like this...

Or having the task toggle your debug led and monitoring that with a CRO so you can time the routines.

The LED is perfect for a single task, but I typically want to know runtime over multiple tasks. Especially so if I'm doing true performance monitoring or looking for tasks blocking for excessive times. RTOS applications are really one of the few times I think this debugging method falls down though.

First step is always to write or port a really good serial monitor, then you can just instrument as you go. You'd be surprised at how much you can learn about code performance monitoring a debug led with a cro, or even putting it into an amplifier and listening to it. The brain is very good at filtering and isolating audio anomalies. I use my audio monitor (just a small amp with a croc clip and meter probe on it) more than my logic probe these days.

I guess it helps to have been brought up in a world before hardware debuggers and being able to single step code. You learn to make do very well with extremely rudimentary tools. Not that I'd give up my debuggers now, but I'm more often still faster using basic tools and deductive logic.

The speaker idea is cool. And I absolutely agree that you need a solid serial monitor solution as the base for any good debugging solution. And that goes a long way beyond just printf

[Brutte]

ARMv7M comes with performance counters in DWT.

That's good to know! Unfortunately I end up spending life in Cortex M3's and M4's mostly.

[rsjsouza]

jerry507, CortexM EQU ARMv7M...

@nctnico, where in the world did you get the idea that protocol stacks are not implemented in embedded systems?!? The biggest counter example is the whole newfangled range of IoT gadgets, but several other examples that require the developer to get his hands dirty can be found, especially in more obscure industrial communications protocols that don't get the same attention as the "consumer-like" pre-canned stacks from silicon vendors...

At times I use the features shown at the video below, which gives an idea what a JTAG debugger tied to a decent IDE can do.
http://youtu.be/HG_i_uln6Es

I also have used in the past a real-time mode of the C28x DSPs...
 

[nctnico]

jerry507, CortexM EQU ARMv7M...
@nctnico, where in the world did you get the idea that protocol stacks are not implemented in embedded systems?!?

Read much more careful. What I am typing is: Develop a protocol stack (or any other high level part of the software) on a PC and when it is finished compile it for a microcontroller or other embedded target. Even with the best hardware debugger the debugging possibilities on a PC are far superior. For example think about the ability to read a signal from a file and dump the result back into a file or do tests at high speed. Sometimes I develop DSP algorithms. Before I try on hardware I test these on a PC. Having the PC process several minutes worth of data only takes seconds.

[free_electron]

a bluetooth stack. from scratch.

nobody does that, because BT stack is quite retarded
someone might implement ONE small subset, but not a whole thing

really ? how do you think the people developing the silicon release the stack ? think before you write things like 'nobody does that'.
The fact that you get these stacks means someone had to write them in the first place If TI releases a nice CC3000 chip with on board stack : they had to code it. and you can bet top dollar it was done using hardware debuggers.

[free_electron]

those are the half that call themselves 'embedded developers'. just because you can write some code on a pre-packaged embedded os doesn't mean you know you are an embedded developer.

Here's a lump of silicon and a datasheet . You get to write a taskswapping microkernel and code up a library that implements Modbus for example, or a bluetooth stack. from scratch.

If you are going to develop a protocol stack on an embedded platform you are insane. For that you use a PC. The same goes for any high level software. What remains are a small number of functions which deal with the hardware.

only part of that can be done on the pc. Anything that requires hardware access cannot be emulated. at that point you need the hardware debugger to trace code, get profiling data.

There is a reason companies like lauterbach , American Arium and others are still around. When i was doing harddisks the Arium systems sat in every lab. There just is no way to test drive code other than trace it on a hardware debugger. All code runs in sync with the platter spinning at 7200 RPM. you can't just throw in a printf here and there to spit some stuff out, you would upset the whole timing of the system. i need to get the data when it presents itself.

these machines were coupled with logic analyzers that could trace the execution in sourcecode through an OMF file. they would sample the serial data flying through and see if the output was correct and how long it took from code execution to stream fly-by.

[nctnico]

those are the half that call themselves 'embedded developers'. just because you can write some code on a pre-packaged embedded os doesn't mean you know you are an embedded developer.

Here's a lump of silicon and a datasheet . You get to write a taskswapping microkernel and code up a library that implements Modbus for example, or a bluetooth stack. from scratch.

If you are going to develop a protocol stack on an embedded platform you are insane. For that you use a PC. The same goes for any high level software. What remains are a small number of functions which deal with the hardware.

only part of that can be done on the pc. Anything that requires hardware access cannot be emulated. at that point you need the hardware debugger to trace code, get profiling data.

Nope. Protocol stacks are primarily developed using test cases in order to test the correct behaviour. It is much harder to inject specific (wrong) messages in hardware than in a software test suite. Same goes for timing. On a PC you can set the clock your software sees as fast as you want making a 3 minute time-out last only one real world second. That makes developing much more convenient. At some point you'll need hardware in the loop but only after you are 100% sure the protocol implementation works.

[rsjsouza]

nctnico, your sentence could go both ways, but seeing through your glasses I agree with you that validating algorithms on a PC is a lot easier - I myself use the still available (but rare) device simulators quite often for that.

However, in my opinion free_electron has it right: the optimizations and intricacies of the embedded architecture are only validated on the architecture itself, and these will require a HW debugger. Also, in my experience this process takes a significant amount of time during development, especially in smaller footprint architectures such as Cortex M and other micros.

So, back to yours (and the OP's) initial statement: if half of the people do not use HW debuggers they are are either extremely smart and knowledgeable about the architecture to get it right in a few iterations or their systems are way too simple to get by with simple message passing - I have done both. 

[nctnico]

None of the embedded firmware engineers from which I learned the tricks of the trade used a hardware debugger. Curious if that was an exception I asked the other attendants of a seminar on ARM microcontrollers what they use for debugging. It turned out roughly 50% didn't use a hardware debugger. I have tried hardware debuggers in the past (for the Blackfin IIRC) but I found these very cumbersome and clunky/unstable to use.

[rsjsouza]

That is quite an interesting info, nctnico. I wonder how many were old timers, or new players that inherited older debug systems.

Just an anecdote from my old days: I recall when our GSM protocol stack used serial port for debugging, which gave good information at a time where JTAG debuggers were really slow (mostly LPT-port based), flaky and expensive. Coincidentally, at that time I also worked on the ADSP-2181 and DSP56824 which had similar crappy JTAG and IDEs.

However, nowadays I would say this changed radically - especially after USB ports became routine. Despite the ridiculous annoyances of the USB protocol stack of Windows OSes, the speed and reliability increased by a large margin.

Anyways, if you have been outside of this world for a long time, it may be worth taking a peek at what's out there. Either for curiosity or for some more practical purpose.

[free_electron]

I wonder how many were old timers, or new players that inherited older debug systems.

especially the real crufty old timers want the hardware debuggers. HP system 9000 based system HP16505 prototype analysers combined with reverse assemblers and code tracers. those things were everywhere in the 80's and 90's. Now only silicon makers still use those as they need to do the HAL work.

Once the HAL is stable then the 'softheads' can kludge around with 'emulation' and 'printf'

You may get the upper layers of the stack right on a pc. But one point you will have to flip bits in register in the chip. if you need to debug that bit flipping you WILL need a hardware debugger.

[photon]

Once the HAL is stable then the `hardheads' have fixed their bugs. 

[westfw]

Once the HAL is stable then the `hardheads' have fixed their bugs.

This is pretty key, I think.  On many embedded systems, the vast majority of the code is written at a much higher level than really benefits from a HW debugger (and benefits more from SW intelligence coded "around" things as a debugging tool.)  The idea of developing on a desktop-class system instead of actual hardware is just an extreme example of this: the hardware (and even low level SW) is so abstracted that it doesn't even need to be present to do significant debugging.