XMODEM transfer problems

[tomshirvo]

Hi All,

I have been working on getting a bootloader up and running on the AT91SAM3X8E (Arduino Due) using Atmel Studio.

Atmel have provided an example bootloader on their site but it was written for IAR workbench so I have tried to convert it to Atmel Studio 6.2

I am able to upload the bootloader and get it to upload a program I made but a few strange things are happening and I just haven't been able to figure out why.

Below is two screen shots. The top screen shot is what the program should look like. If I upload this using SAM-BA in the flash location 0x89000 the bootloader runs it no problem.

The bottom screen shot is what gets put into the flash on the chip when I upload using XMODEM. There seems to be two 0xF0 at the start. The problem is that I don't know why and I can't just say ignore 0xF0 as it is needed in other parts of the program.



This is the code that does handles the uploading of the new program.

Code: [Select]

/**
 * Name:      communication_receive
 * Purpose:   Waits for a page of firmware code and returns it
 * Inputs:
 *  - Buffer in which to store the received page
 * Output:    Size (in bytes) of the page received
 */
unsigned int communication_receive(void * vAddress) {
//debug_printf("Address: %X\n", vAddress);
  // Local variables
  unsigned int bytes;
  unsigned int remaining;
  unsigned int endOfFirmware;

  // Restart transfer
  media_writeChar(XON);

  // Wait for data, monitoring activity to detect the end of the firmware
  endOfFirmware = FALSE;
  bytes = 0;

  while (!endOfFirmware && (bytes < MEMORY_PAGE_SIZE)) {

    bytes = media_poll();
    endOfFirmware = TRUE;

    for (unsigned int i=0; i < XON_XOFF_WAIT_COUNT; i++) {

      if (bytes != media_poll()) {

      endOfFirmware = FALSE;
      break;
      }
    }
  }

  if (bytes != 0) {

    // Read page
    if (bytes > MEMORY_PAGE_SIZE) {

      bytes = MEMORY_PAGE_SIZE;
    }
    bytes = media_read(vAddress, bytes);

    // Send XOFF control character and wait for remaining characters
    media_writeChar(XOFF);
    remaining = media_poll();
   
    for (unsigned int i=0; i < XON_XOFF_WAIT_COUNT; i++) {

      if (remaining != media_poll()) {

        remaining = media_poll();
        i = 0;
      }
    }
  }
  return bytes;
}
This is the program I use to upload the program/firmware. It was included in the example Atmel provided. I am unable to have the code for how this works. I have tried to use Putty with XMODEM transfer but it won't upload the file when I try to use it.




  This is how I feel right now

[helius]

You're using 115kbps and software flow control; do you see a problem there?
Use a lower speed if you don't have RTS+CTS.

[JVR]

As a start, your endianess is broken. Get that fixed before doing too many other things.

[tomshirvo]

Wilfred, I was using extraputty it has XMODEM (http://www.extraputty.com/index.php).

Bootloader Application Note: http://www.atmel.com/Images/Atmel-42141-SAM-AT02333-Safe-and-Secure-Bootloader-Implementation-for-SAM3-4_Application-Note.pdf
Zip download of code: http://www.atmel.com/devices/SAM3X8E.aspx?tab=documents

You're using 115kbps and software flow control; do you see a problem there?
Use a lower speed if you don't have RTS+CTS.

I will have a try with it going slower. I did some looking around and people sound confident that the Arduino Due can get upto 100Kbps. Just a scary thought if I have to go slower. The XBee that I will be using with the device has RTS/CTS so I can use that.

As a start, your endianess is broken. Get that fixed before doing too many other things.

So I am just storing the data into the Flash in the wrong way.  So does that mean I just have to just change from MSB to LSB. otherwise do I have to do something like swap joined sections like 926D0008 to 00089926d as it possibly looks like that in my screen shots.

[JVR]

As a start, your endianess is broken. Get that fixed before doing too many other things.

So I am just storing the data into the Flash in the wrong way.  So does that mean I just have to just change from MSB to LSB. otherwise do I have to do something like swap joined sections like 926D0008 to 00089926d as it possibly looks like that in my screen shots.

Ok, scratch that, having spent more than 5 seconds looking at it, it is an offset problem.

Step through the bootloader code and see what it does with the first two bytes it receives?

[magetoo]

XMODEM is more than just XON/XOFF if I cast my mind way back.

Yeah, XMODEM is an entire file transfer protocol, and the example code posted has nothing to do with it.

Edit: Made dumb suggestion, deleted dumb suggestion.

[magetoo]

Looking at the code more closely, it seems that media_poll() and media_read() is where the action is.  We probably need to see those to have any idea what is going on.

I'm guessing it's probably going to be easier to write a bootloader from scratch than to understand what's going on though.

[tomshirvo]

Looking at the code more closely, it seems that media_poll() and media_read() is where the action is.  We probably need to see those to have any idea what is going on.

I'm guessing it's probably going to be easier to write a bootloader from scratch than to understand what's going on though.

us0_poll() is the same as media_poll()

Code: [Select]

/**
 * Name:      us0_poll
 * Purpose:   Returns the number of bytes available in the read buffer
 * Output:    Number of bytes that can be read in one call
 */
unsigned int us0_poll() {

  return cyclic_GetSize();
}


#define cyclic_GetSize()    \
    ((cyclicDataEnd>=cyclicDataStart)?\
     (cyclicDataEnd-cyclicDataStart):\
     (cyclicDataEnd+US0_BUFFER_SIZE-cyclicDataStart))


us0_read) is he same as media_read()

Code: [Select]

/**
 * Name:      us0_read
 * Purpose:   Reads characters from the cyclic buffer
 * Inputs:
 *  - Buffer in which to store read characters. If NULL, the whole
 *    cyclic buffer is emptied
 *  - Size of buffer
 * Output: Number of characters read.
 */
unsigned int us0_read(void * vBuffer, unsigned int size) {

  // Local variables
  unsigned int bytes;
  char * pBuffer = (char *) vBuffer;

  // In case buffer is NULL, empty buffer
  if (vBuffer == NULL) {

    bytes = cyclic_GetSize();
    cyclicDataStart += bytes;
    cyclic_DecSize(bytes);
    return bytes;
  }

  // Loop through available characters, storing them in the buffer
  bytes = 0;

  while ((bytes < size) && (cyclic_GetSize() > 0)) {
 
    pBuffer[bytes] = cyclicBuffer[cyclicDataStart];
    cyclic_DecSize(1);
    cyclicDataStart = (cyclicDataStart+1)%US0_BUFFER_SIZE;

    bytes++;
  }

  return bytes;
}

XMODEM is more than just XON/XOFF if I cast my mind way back.

Yeah, XMODEM is an entire file transfer protocol, and the example code posted has nothing to do with it.

Edit: Made dumb suggestion, deleted dumb suggestion.

This is what I currently have planned for this in the long run and would like comments if it sounds like a terrible idea.

Each of these devices will have an xbee connected to it. There will be one device that will hold the latest firmware lets call it the server. Each of the other devices will have the bootloader start and poll the server and check if a new version is on it. If so then it will transfer all of the data to the device.
 
I was thinking that the whole firmware would be sent and then error checking would be done on the full transfer and not with each packet that was sent. This is because I would like to be able to push out new firmware in a broadcast to every device on the network. This will save a lot of time as there could be upto 200 devices and each of these devices updating its firmware one at a time will take a long time.

Am I crazy or is this doable??


Will use MD5 hash as the checksum unless someone can suggest something better?

[westfw]

"XMODEM" was both an early file transfer protocol (packet format approx SOH BLKNUM ~BLKNUM <128byte data> CKSUM),
AND a common name for programs that implemented that protocol (along with, perhaps, simpler, protocol-less "upload" features.

I'll assume:
1) You're referring to the the bootloader described in "Atmel AT02333: Safe and Secure Bootloader Implementation for SAM3/4"  (which does not include ANYTHING about "xmodem.")
2) You're using some program that you're calling "xmodem" (which one?)
3) this program is transmitting the extra F0s. (I'm not sure why; that isn't a sequence I remember being used in any XMODEM-like protocol...)
4) The "protocol" described in this app note is pretty awful; they seem to be spending more attention to encryption issue that data transmission integrity.

[tomshirvo]

So I have found my problems but not all are fixed.

1. I was using an Arduino Uno without the ATmega328 chip, this was the reason I was getting 0xF0. Changed it and am now using an FTDI chip thats on an XBEE board.
2. Possibly buffer size was to small, I made it larger just incase.

I have had such a crazy week and I don't even know where I got XMODEM from. The uploader that came with the provided example from ATMEL has XON/XOFF and I must have linked the two together.

The new problem that I have come up against is that I transfer the file but only the first and last parts get stored into the flash memory. I will work on that problem and see why it is happening and try and sort it out.

Thanks everyone for all your help, I feel yet again like an idiot because it was nothing to do with XMODEM at all.

"XMODEM" was both an early file transfer protocol (packet format approx SOH BLKNUM ~BLKNUM <128byte data> CKSUM),
AND a common name for programs that implemented that protocol (along with, perhaps, simpler, protocol-less "upload" features.

I'll assume:
1) You're referring to the the bootloader described in "Atmel AT02333: Safe and Secure Bootloader Implementation for SAM3/4"  (which does not include ANYTHING about "xmodem.")
2) You're using some program that you're calling "xmodem" (which one?)
3) this program is transmitting the extra F0s. (I'm not sure why; that isn't a sequence I remember being used in any XMODEM-like protocol...)
4) The "protocol" described in this app note is pretty awful; they seem to be spending more attention to encryption issue that data transmission integrity.

I removed the encryption all together as I spent a lot of time and just couldn't get it to compile with all the errors that would come up.

[tomshirvo]

So a few other things fixed/removed.

Slowed the serial to 57600 baud.

For some reason when making the serial buffer size larger then 1024 it only stores the first and last bits of the uploaded software. I have no idea why this would be happening.

So now the bootloader is working and will start the program that is uploaded. I am going to be fixing things up a bit and will release it into the world for anyone else that might want it. Even if it is probably messy.