I remember this bug we were blamed for. Two teams, the "starts" and we. The "stars" blame us, we receive a stream of data, buffer it (double buffer) and give it to the stars' library. Boom crash. They say we overwrite the data (double buffer system with synchronized call to another task). I log every event, which buffer I use to receive and which buffer I give to the other thread... no overwrite over many many cycles... but it crashes at some point. I fire up the debugger and patiently go through the code, optimized C code is quite difficult to follow with a disassembler... after some time I realize what happens... they, the "stars" forgot to initialize a structure where the data would be copied and some fields had old data... Even showing them the problem did not convince them... but they fixed it and there were no more crashes.... I was fortunate enough to have this logging mechanism as it clearly showed that the problem was somewhere else... as someone here says "there are big lies, small lies and what is on the screen of the oscilloscope".
Alignment problems in MIPS32 (PIC32), yeah I found out the hard way too... arm cortex-m3 has no problem with unaligned data...