Most annoying bug....

[dmills]

This one that just cost me 7 working days: hard of thinking processor vendors supplying their own libc, and somehow managing to screw up the string handling functions, strtol and friends in this case, it is not like there is not a perfectly good one in the BSD libc they could have freely used (That actually works), guess what my fix was?

Of course the bug they introduced was a bus error that only occurred when the string started at an address NOT divisible by four....

Also of course the call was in the middle of a mess of pointer arithmetic (In the way adhock text parsers in C tend to be), so it was not at all obvious that the crash was in the library (Mips has this amusing thing it does upon function call return which made the problem appear to occur after the return).

Next project is so going to use something non Microchip.

So who else has had a good bughunt recently?

Regards, Dan.

[amyk]

MIPS alignment requirements are certainly annoying but I can see how they never tested with anything other than aligned strings.

[obiwanjacobi]

I am currently examining a situation where the Debug build puts out several bit-banged signals (LCD) too fast compared to the release version. I have tracked it down to code optimization options and have run a few isolated tests that did not reproduce the problem. It must be some structure in my code that gets 'optimized away'.

I have patched it for now by making the delay times a bit bigger/longer because I fear that hunting this one down may eat up a lot of time and I want to keep moving...

[Jope]

MIPS alignment requirements are certainly annoying but I can see how they never tested with anything other than aligned strings.

Unaligned access to char arrays is actually a pretty normal use case. Let's say you have a command string with a number and you want to extract the number with strtol():

Code: [Select]

long val = strtol (&cmd_str[index], NULL, 10);

Depending on index, it will be an unaligned access.

[dmills]

Yep, a mix of strtok and strtol is fairly standard for pulling apart a string of CSV data and exactly what I was doing.....
It does work when you have an strtol that was not written by the crack smoking intern.

Regards, Dan.

[Maxlor]

This bug cost me a couple of days a few years ago: a pointer bug caused the interrupt table to be modified, leading to a crash some time later, when the interrupt happened. (On ARMv5, the interrupt table is at address 0, so dereferencing a NULL pointer is legal, and puts you there.) That sort of problem is hard to find if you don't suspect it, because the program will crash at a random location (and usually always in the same place, if the timing remains the same), but that location has nothing to do with the actual bug. Plus, when you start debugging, the location will move. When you single step, the bug disappears altogether. It's a classical heisenbug.

After having finally found and fixed the problem, I felt motivated enough to read up on the ARM MMU unit and and figure out how to write-protect most of the address space. It cost me another two days to get that working because the docs were rather sparse, but I felt that eliminating a whole class of bugs was worth it.

Not strictly a bug, but painful nontheless: quite some time ago I was developing a custom printer driver for our application that would talk directly to the hardware. Apparently, the printer manufacturer wasn't a fan of error reporting. If you sent the printer invalid data, it would simply blink it's red LED 3 times and do nothing else. After pouring over a lot of hexdumps, and some guesswork, I did get it working in the end, so yay. But it's not a great feeling if you don't know whether you're 5 minutes or 5 days away from solving your problem.

[Dielectric]

Not strictly a bug, but painful nontheless: quite some time ago I was developing a custom printer driver for our application that would talk directly to the hardware. Apparently, the printer manufacturer wasn't a fan of error reporting. If you sent the printer invalid data, it would simply blink it's red LED 3 times and do nothing else. After pouring over a lot of hexdumps, and some guesswork, I did get it working in the end, so yay. But it's not a great feeling if you don't know whether you're 5 minutes or 5 days away from solving your problem.

Oh man, you triggered a flashback for me.  Furiously debugging a test box design, I had a couple of LEDs for user indicators.  The red one was on/off/blink for various states, driven from a PWM.  For some reason it was always either off or blinking, but this was a target board so the ICE wouldn't connect to it to set a breakpoint, etc.  UART printf didn't show much either.  After about a day, I realized I had put in just a random red LED from the junk box which turned out to be a flashing LED.  I could just barely see the little ASIC in the T1 package.  Hateful.

[dmills]

Ouch!

We will gloss over the week spent on a linker script (Why are they never properly documented?) due to the fact that 0x1000000 != 0x10000000

Regards, Dan.

[eck]

Explicitly targeting the 80517 (improved 8051), which also enables additional data pointers via a data pointer select register, without taking care to also save and restore the data pointer select register during interrupts. Random crashes every week or so, only in the fully operational system at the customer site.

After month of logging, experiments and desperate hacks, the problem was finally found reading the assembly code. Fortunately caught most of the affected systems in the warehouse before shipping, and replaced the EEPROMs.

[langwadt]

Yep, a mix of strtok and strtol is fairly standard for pulling apart a string of CSV data and exactly what I was doing.....
It does work when you have an strtol that was not written by the crack smoking intern.

one thing is writing it but it is hard to imagine how little effort when into testing if something like that wasn't caught

[rollatorwieltje]

Not strictly a bug, but painful nontheless: quite some time ago I was developing a custom printer driver for our application that would talk directly to the hardware. Apparently, the printer manufacturer wasn't a fan of error reporting. If you sent the printer invalid data, it would simply blink it's red LED 3 times and do nothing else. After pouring over a lot of hexdumps, and some guesswork, I did get it working in the end, so yay. But it's not a great feeling if you don't know whether you're 5 minutes or 5 days away from solving your problem.

At least it just blinks a LED. A long time ago I was messing around with that HP printer configuration protocol (PJL?). Turns out when you make an error, instead of complaining it just starts printing the commands you send. Really nice when you want to configure a printer with a script and suddenly it has wasted 50 sheets of paper, just because there was something stupid like an extra space somewhere

[AndyC_772]

We will gloss over the week spent on a linker script (Why are they never properly documented?) due to the fact that 0x1000000 != 0x10000000

Now I don't feel so bad about the day I lost because

load="Yes"

...isn't the same thing as...

Load="Yes"

...in a linker placement file. One is recognised and understood, the other is silently ignored. The net effect is that, although the code is perfectly good, and appears in the IDE correctly, it's not actually present in the MCU's memory. The code that gets executed is not the same as the code that appears in the debugger.

It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

[Karel]

Apart from USB and TCP/IP, I don't use any libraries at all. During my career I have written all those utils/functions myself. At least I can blame myself when there is a bug.
There are a couple of target/MIPS specific functions that I copied from the mchip libs to my own libs but only after a carefull examination and fully understanding what it does and doesn't.
I have bumped my head too much to trust vendor specific libraries. And when they work, they are usually bloated and written in a style that makes it difficult to understand for me what is going on.
But probably this is also because my coding/writing style is different than what is considered normal... For example, I hate it when the hardware is abstracted too far away. I'm a hardware guy, I like to poke registers.

[tggzzz]

It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

And that's the generic problem that you will automatically avoid in the future! Everybody knows about 0 and O, but L l 1 are just as problematic.

[diyaudio]

So who else has had a good bughunt recently?

I spent about 2 weeks with a issue where the external flash code intermittently loads into the SHARC processor, sometimes it loaded successfully and sometimes it didn't..  during many power cycles attempts it would work ... the problem was I didn't connect the external crystal loading caps and loading resistor to the crystal, so the processors PPL didn't have a stable oscillator during start-up... I was pretty close to dumping the board in the dust bin   

[Wilksey]

The question is what compiler (XC I presume, but XC8, XC16 or XC32?) and what version, I find that bugs get introduced in one version of compiler that work fine in a previous version, I believe the XC compilers are what used to be the HITEC C compiler?

Cheers

[Ian.M]

No.  XC8 is based on the HiTech PICC and PICC18 compilers (Microchip bought HiTech), but XC16 and XC32 are based on GCC.

[Howardlong]

I had to deal with some code the other week where the dev guy had let a string literal go over the end of a line. In his test system where it was deployed, and worked, his editor replaced the leading spaces on the next line with tabs. In the higher environments, the spaces remained spaces, and exceeded the space allocated. The dev guy, of course, was blaming the deployment process, and not his code (dev guys are _always_ right, right?)

It took far longer for me to show & explain to him what the problem was, and how to resolve it, than it did for me to notice it. This quite commonly happens, a second party sees an error like this right away, while the other party remains completely blind to it.

[AndyC_772]

It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

And that's the generic problem that you will automatically avoid in the future! Everybody knows about 0 and O, but L l 1 are just as problematic.

N0rmaLLy I'd expect a parser to ch0ke and c0mpLain if I transp0se '0' with 'o', th0ugh. In this case, n0 error was generated; my instructi0n t0 actuaLLy d0wnL0ad this part 0f the c0de t0 the b0ard was simpLy ign0red.

I usuaLLy expect capital and L0wer case Letters t0 be interchangeabLe, 0r t0 thr0w an err0r!

[c4757p]

load="Yes"

...isn't the same thing as...

Load="Yes"

...

It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

There's a font where L and l are hard to distinguish? Wow. I've never seen one that bad.

[AndyC_772]

Spot the difference. View at 1:1 scale from the distance you'd normally view a screen full of code!

[rollatorwieltje]

Spot the difference. View at 1:1 scale from the distance you'd normally view a screen full of code!

Looks like that garbage font Microsoft introduced in Windows Vista. I think it's called Segoe or something like that. I pretty much use Dina everywhere: https://www.donationcoder.com/Software/Jibz/Dina/

[c4757p]

Spot the difference. View at 1:1 scale from the distance you'd normally view a screen full of code!

Jesus Christ, that's bad. Whoever selected that font should be shot, as well as whoever designed it if it was intended for this sort of thing.

Real fonts make those much more recognizable:

[free_electron]

We will gloss over the week spent on a linker script (Why are they never properly documented?) due to the fact that 0x1000000 != 0x10000000

Now I don't feel so bad about the day I lost because

load="Yes"

...isn't the same thing as...

Load="Yes"

languages should be case insensitive. shoot the developers.

...in a linker placement file. One is recognised and understood, the other is silently ignored. The net effect is that, although the code is perfectly good, and appears in the IDE correctly, it's not actually present in the MCU's memory. The code that gets executed is not the same as the code that appears in the debugger.

It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

[c4757p]

Spot the difference. View at 1:1 scale from the distance you'd normally view a screen full of code!

Looks like that garbage font Microsoft introduced in Windows Vista. I think it's called Segoe or something like that. I pretty much use Dina everywhere: https://www.donationcoder.com/Software/Jibz/Dina/

Hrm. I'd not seen Dina before, and I like one thing about it - it makes the classic 1I|l 0O distinction better even than my current preferred font (Terminus). I want that. Problem is, I find the rest of the font very tedious to look at. It's ugly.

But then, I'm insanely picky about fonts. Maybe sometime I'll make myself a modified Terminus - pull in those Dina-style 1I|l 0O, and pull in the 9pt bold from Terminusmod, at least.