Accessing Hidden Diagnostics in Prog Memory ATMega1280?

[Rjc987]

Hi guys! Im working on a biotek spectrophotometer that uses a atmega1280, 1281 and 164p (1280 seems to be the main micro) trying to track down some errors which aren’t documented in the manual and decided to try to get a better understanding of the machine so I hooked up mplab via JTAG through an mplab snap and in the process of stepping through the program in debug mode, I realized a huge chunk of the program memory is dedicated to a factory setup and debug feature, with menu options in ascii (like “test mode” “select desired command character” “toggle test mode on/off ‘O’, etc. See images below - these are just 2 pages of about 15 pages of menu).

These detailed and advanced onboard diagnostic tools would be a HUGE help as currently the only obd available is a very basic self test in the windows software but I have no idea where to begin or how to figure out what I need to do to access any of it. Does anyone k have any idea how this works? Thanks!

[Berni]

You run the hex dump trough a disassembler to convert it back to assembler code and spend many hours staring at the huge assembler mess until you figure out how it works.

Usually the best hint to start with is looking for memory references to those strings.

[mjs]

Looks like you've got the flash dump, which is sometimes the hard part requiring glitching and other tricks. The next usual step is to hit it with IDA Pro (€€€) or Ghidra (paid for by generous US taxpayers.

I've used Ghidra, but the learning curve is rather steep. It's not just a disassembler/decompiler, but rather a full toolkit for reverse engineering binaries. On the other hand there are now a lot of youtube videos on how to use it.

Usually hidden menus and how to access them reveal themselves from the source code rather fast - you start from the interesting strings, check which functions access those. Then do the same for button / serial IO - which may often fiddle data through ISRs. After this the solution often awaits you very close to the place where these paths meet in the call graph.

Basic code reverse engineering is a very useful skill when dealing with existing stuff - and also while developing new, I've found out!

[T3sl4co1l]

Well, that's a damn sight easier than they could've made it.

Your first boon: they didn't lock the Flash, or encrypt it (not that there's any real-time decryption capacity to speak of on an 8-bit CPU, lol), or disable the programming port(s).

Second boon: AVR is one of the more pleasant dialects to read and understand.  Uh... if you aren't familiar with the ISA (Instruction Set Architecture), that'll be a bit of a missed point for you, understandably... but if you're familiar with any assembly, you should be able to get a good feel for it with a couple weeks' study, and if only programming generally, maybe some months.

Assuming they used most of the 1280's available program memory, you'll have a pretty good chunk of time picking out all the various subroutines, function calls, data paths... but it's by no means impenetrable.

Mind, heh, I do say this with the bias of mainly having looked at my own output (symbols included), not completely bare and scrubbed binary.  It will be quite some work to reconstruct all that, of course.

Ease of disassembly, is reasonably true for the most popular flavors of MCU; depending on age, it could've also been 8051, Z80, 68k, x86, MIPS, or a few others; and especially these days, ARM.

I'm not sure about AVR offhand, but ARM, x86 and many others are well supported by disassembly and decompilation tools such as Ghidra.  The asm might be a bit more opaque to read (particularly some of the instruction ordering say with MIPS, I think?), but the decomp is -- well, you still don't get any symbol names, but formatted C code is a far sight better than the alternatives, especially if you're not well versed in the ISA itself.  That, and the instant cross-referencing and refactoring, makes analysis so much easier.

As for just trying to figure out the interested function straightaway, the most likely candidates would be a serial port somewhere, or jumpers or dip switches to activate a service mode, or some password to enter on the main menu, or some sequence of button presses to unlock a hidden menu, or etc.  Lots of possibilities of course.  If it's a password, you might find a string somewhere in .data (usually near top of Flash), or .progmem (usually near the bottom).  It might also be checked by a sequence of compare instructions, or alternating loads and compares, or a limited crypto sort of function might be employed that uses a lot of eor instructions which can give some direction on where to look (but mind that eor rn, rn is commonly used as ldi rn, 0 shorthand, depending on compiler version or asm author, so it's not very specific).  Snooping around relevant code sections (be careful to try and spot pointers and addresses, try to backtrack along code and data paths; very hard in raw hex, easier disassembled) may lead you to a data path or state machine (esp. to do with menus?) that allows access.

If it's via serial port or whatever, it might be tricky to figure out, just because they needn't put any feedback in; it could be a binary comm channel for example, without even a terminal echo to tell what the baud rate is.  If you're lucky, it's just wide open, plain text, it blinks out some data on startup, the baud is standard, and they have a service menu and everything right at your fingertips.

Tim

[abyrvalg]

OP, would you mind posting the dump here so we’ll take a look?

[Rjc987]

Absolutely, they definitely made access easy anyway- it’s completely unlocked according to the config bits and as far as I can tell, it looks completely unencrypted (all the diagnostic menus, etc buried in the program memory you see in those screenshots are all in plain text), the program hex dumps no problem and decompiles perfectly, I’m able to live debug and single step through the machine code, monitor the I/O memory, registers, etc.

They certainly used a pretty solid chunk of the 1280s onboard memory capacity – it’s 60,000+ lines of code, only a couple thousand lines are blank. And I’ve been so preoccupied pouring over and stepping through the dump from the 1280 JTAG that I’m not even sure if that’s everything or not. When multiple atmegas are daisy chained together, does mplab x ide dump everything in one go or does the code from each micro have to be dumped separately through their individual JTAG headers?

I’m moderately familiar with ISA and I can somewhat follow my way through machine code (albeit VERY slowly…), although all my past experiences have been with PICs so I’m having to learn my way around AVR’s architecture and instruction set from scratch. Also I have much more experience working with hardware, and the software side of things is definitely not my strong suit. I’ve definitely heard a lot of great things about ghidra, so probably a good excuse to play around and start learning it - even though mplab has done a really nice job decompiling the code and providing mnemonics for everything, even without my handicaps, pouring through 60,000 lines of dense, undocumented machine code is a monumental task for anyone. Even stripped of all symbols and comments, C would be infinitely preferable!

As far as jumpers/etc. there’s no dip switches, or jumpers on the boards whatsoever. There is an RS232 port that is populated on the back of the machine near the USB connector. On the boards there are 3 separate populated JTAG connectors for each micro and associated subsystem (UI, motor control, & sensors), and a populated header marked ISP (but which goes to the motor control micro (1281) and uses the rx, tx, scl, sda, icp1, Vcc, and ground - I’ve been told is in fact a standard for the atmel ice but I can’t find documentation to back that up). As far as unpopulated items, there’s very few. One is an 8 pin header  that goes to the 1280 and is ACTUALLY isp (miso, mosi, reset, etc), as well as an unpopulated 8x2 connector that sits right next to the 10x2 connector for the sensor board (which I’m pretty sure is for a different model with extra sensors since it goes through the same circuitry).

As far as user I/o the machine only has 2 switches - the power which is a hard on/off switch, and a push button for ejecting the tray. There is no display of any kind, and all interfacing goes through USB to software. The tree eject button is also used to dismiss error messages which it annunciates by beeping, but I doubt it has anything to do with that. It seems like RS 232 or USB would be the most likely options… Maybe a terminal command?

I’ll take a look at the different pointers to see if it is leading anywhere.

And as far as the hex, I’m not at that computer tonight, but I’ll definitely upload the full dumps for you guys to look at as soon as I get to it in the morning. However, the raw hex dumps are totally different from White and pb lab decompose into, however, I haven’t found a way to ever save all the decompiled info. Is there a way to do that in Mplab x ide that I just haven’t been able to find?

[Rjc987]

Oh wait, I actually do have a copy of the hex dump here from mplab x ipe anyway: