Portable Safe Way To Send Struct Between 2 Microcontrollers?

[NorthGuy]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

[nctnico]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

It is. Think in terms of NRE. A lot of time & money gets spend on verification and finding problems / reasons why things fail every now and then. Having text based protocols reduces the time needed to debug systems a lot. At the moment I'm consulting on a project which has quite a few communication devices which use different communication interfaces chained up together. Using text based protocols, do emulation of higher level devices locally and plain text debug output helps a lot to verify correct behaviour of parts of the system and the system as a whole.

[tggzzz]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Ease of implementation.
Ease of debugging.
Ease of  comprehension in a deployed system.
Simple "packet" framing, parsing and error recovery.

Where efficient utilisation of the comms medium is important, it is worth expending more effort avoiding redundant data. But if that isn't an issue, KISS is a useful principle.

[NorthGuy]

It is. Think in terms of NRE. A lot of time & money gets spend on verification and finding problems / reasons why things fail every now and then. Having text based protocols reduces the time needed to debug systems a lot. At the moment I'm consulting on a project which has quite a few communication devices which use different communication interfaces chained up together. Using text based protocols, do emulation of higher level devices locally and plain text debug output helps a lot to verify correct behaviour of parts of the system and the system as a whole.

Of course, you'll spend lots of time and money when you use such archaic methods.

I have a CAN network with several devices which send lots of various messages - perhaps a hundred every second. If I had to look at them all in text format, it would take me days, if not weeks, to find something in such a stream of text. Instead, I have connected a sniffer to the network, which verifies all the messages, select the messages that I need and shows them to me. If there's any problem, it doesn't take long to figure out what's going on. I can use the same sniffer device to run automated tests which can run for hours (or days) - automatically injecting messages that I want, monitoring responses etc.

Similarly, you can sniff UART or anything else (if not too fast), decode, find errors, write selective logs, show on display, store to a file and parse later etc. etc.

[AVI-crak]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

There's always a limit when the simple things just don't work anymore.

[SiliconWizard]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Not only that, but it actually makes it more likely to contain bugs. Parsing text 100% correctly, fail-safe, without any buffer overrun and other niceties, that's a lot less trivial than properly writing a binary-based protocol and handling it (yes, as long as you have mastered first the basics, like alignment and endianness, which are really basics and way simpler to master than writing a proper parser, really).

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

But that's always interesting to see how perspectives can be different between people doing basically the same job.

[asmi]

whatever the "network byte order" means on a given platform supporting POSIX.

It always means the same thing on all platforms, which is it's raison d'etre.

[PlainName]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool, so why not just prevent that issue in the first place? Also, it's well known that the writer of bugs isn't usually the best person to check for that bug, so if there's a protocol issue why would a protocol debugger written by the same person not have the same issue?

Finally, the human eye is generally much better than some predefined general purpose tool for spotting tiny clues that may point to something. Don't get me wrong - a protocol debugger can be very useful indeed, but it's not the be-all and end-all. And if you're unlucky (which, in this case, you'd want to be!) you might spend those hours writing this tool that will never be needed. So the temptation is strong to write it when you have a problem, and then it can be targeted at that specific problem. Being able to recognise data, or handle it simply, may remove the need for that.

[tggzzz]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Not only that, but it actually makes it more likely to contain bugs. Parsing text 100% correctly, fail-safe, without any buffer overrun and other niceties, that's a lot less trivial than properly writing a binary-based protocol and handling it (yes, as long as you have mastered first the basics, like alignment and endianness, which are really basics and way simpler to master than writing a proper parser, really).

Buffer overrun is always an issue in C-like languages, however the information in the struct is encoded.

On a UART getting the "packet" framing delimiters right in the presence of errors can be error prone. Doubly so where the information is binary encoded and hence could mimic the packet framing delimiters.

If someone can't convert 32-bits to/from hex ASCII characters, they shouldn't be writing any code.

[nctnico]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool, so why not just prevent that issue in the first place? Also, it's well known that the writer of bugs isn't usually the best person to check for that bug, so if there's a protocol issue why would a protocol debugger written by the same person not have the same issue?

Finally, the human eye is generally much better than some predefined general purpose tool for spotting tiny clues that may point to something. Don't get me wrong - a protocol debugger can be very useful indeed, but it's not the be-all and end-all. And if you're unlucky (which, in this case, you'd want to be!) you might spend those hours writing this tool that will never be needed.

IMHO such a tool is massively usefull for debugging and verification purposes. And you can go a step further as well. Over the decades I've implemented a wide variety of protocols and for several of them I have written simulators to test against (partly due to lack of having the real thing available). During testing with the actual system I made sure to update the simulator as well to get a precise 'model'. The advantage of a simulator is that you can do very extensive tests (including introducing faults) in a short time without manual intervention.

[Siwastaja]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

For real, automated (unit and/or integration) testing, binary format works as well as text or better, and because the parser will be simpler (specifically, has less hidden state and uses (almost) exclusively fixed-size buffers of predictable size), it is less likely to contain bugs. It is surprising how buggy text-based parsers can be, people who think text based protocols are "simple" to implement tend to get stuff like line feeds (four different variants, some of which store state between bytes) seriously wrong.

And don't get me wrong, there is no need to be afraid of mistakes. If you want or need text-based protocol, then of course you will implement it, try to be careful, and test it thoroughly. It is entirely possible to make it completely bug-free. But thinking that this is somehow easier is so blatantly wrong that I would not trust anyone who says that. They are very likely to make bugs.

[nctnico]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

Now you are throwing initial testing and final verification on 1 pile. For text based protocols you can use existing tools (like sed, grep, etc) combined with simple scripts (bash, python, etc) that can parse & check a long log file in text (captured using a terminal emulator) to produce a report. For a binary protocol you'd need put much more effort into tooling for capturing the data and doing the verification / translation. For examples: if you have numbers as columns in text, it is super easy to plot these into a graph using Excel.

[tggzzz]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

For real, automated (unit and/or integration) testing, binary format works as well as text or better, and because the parser will be simpler (specifically, has less hidden state and uses (almost) exclusively fixed-size buffers of predictable size), it is less likely to contain bugs. It is surprising how buggy text-based parsers can be, people who think text based protocols are "simple" to implement tend to get stuff like line feeds (four different variants, some of which store state between bytes) seriously wrong.

And don't get me wrong, there is no need to be afraid of mistakes. If you want or need text-based protocol, then of course you will implement it, try to be careful, and test it thoroughly. It is entirely possible to make it completely bug-free. But thinking that this is somehow easier is so blatantly wrong that I would not trust anyone who says that. They are very likely to make bugs.

Terminal-based testing isn't approriate for unit/integration/regression tests. It is suitable for initial exploration and for "WTF is it doing now" field service tests.

[AndyBeez]

This question is rather like asking, how many lines do I need to draw a polygon? To the OP, I would say first serialise the struct  making sure its instance is serializable. Then buffer the character payload out to the TX/RX lines, using DMA if available. Interconnect the uC via RS232/485/USB, or a 10K resistor, depending on distance. Then scope the traffic with the protocol analyser to test. Other end, do this all in reverse. Then walk away because someone is bound to want to use SOAP over TCP instead. Or I2C.

[SiliconWizard]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool

No.

[TomS_]

these functions "convert values between host and network byte order", whatever the "network byte order" means on a given platform supporting POSIX.
I fail to see how that could answer the OP's question here.

Network order is not platform dependent, that's the whole point of it.

Network order is specified as big endian.

But using hton() and ntoh() would only be to solve one half of the OPs problem, which would be to provide predictable endianess for the data between the two MCUs. As functions or macros they are easy enough to implement if they are not otherwise available.

How you pack the data is the other half which can be solved another way.

[NorthGuy]

But using hton() and ntoh() would only be to solve one half of the OPs problem, which would be to provide predictable endianess for the data between the two MCUs.

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols. Look around - there are many store fronts and none of them have anything where people could tie their horses - somehow that is Ok.

[TomS_]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

I dont care for endianess arguments. 

[nctnico]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

As I wrote before: those functions don't help with alignment issues so they only solve half of the problem. You can't use those functions on multibyte variables which are cast from a random point in a byte buffer on all platforms.

[PlainName]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

As I wrote before: those functions don't help with alignment issues so they only solve half of the problem. You can't use those functions on multibyte variables which are cast from a random point in a byte buffer on all platforms.

Not hard to write ones that do work for your particular data. In fact, I'm failing to see why having a decoder (and encoder) is such a problem - if it's not needed it's blank and does nothing, but if they architecture changes and they are needed then there they are! Nothing to fix upstream. Then you're freed from intimately associating your data with your code and, blimey, don't we all know that a high level of coupling is just bad, the information equivalent of spaghetti code. And we also all know that starting with "it's just a quick hack, doesn't need to be thought about" is often the precursor to zombie shit that's buggy as hell but too useful to discard. Not to mention that just getting into the habit of Doing It Right is a big plus by itself.

So here we are preaching the exact opposite of all that, trying to outdo each other in being cheaper and more amateurish. Which would be so bad if all this wasn't aimed at someone wanting to learn something, and guess what he's learning!

[pqass]

I like concrete examples... 
so I decided to create my own implementation of message-passing plumbing.
It may be useful to others as a starting point or just to discuss.

The high-level parts (see below) are: message structures (like struct example_msg), and their corresponding encoder functions (like send_message_example_1()).  Also included are preamble/final envelope helpers.  The intent is to make the code compile/work in both sender and receiver environments (8 bit MCUs to 64 bit CPUs), rather than each having their own independent implementation linked only via documented wire-protocol.

Code: [Select]

#include "msg.h"

char    *_msg_type_example = "EXAMPLE_TYPE";

struct example_msg {
  int errcd;
  char *errmsg;
  float severity;
  char *stack[10];
  int bloblen;
  char *blob;
  int magic[5];
};


void send_message_example_1(int16_t sn, struct example_msg *s) {

  send_message_preamble(sn, _msg_type_example, 1);

  write_int16((int16_t)s->errcd);
  write_asciiz(s->errmsg);
  write_float(s->severity);
  for (int j = 0; j < NELEMS(s->stack); j++)
    write_asciiz(s->stack[j]);
  write_int16((int16_t)s->bloblen);
  write_blob(s->blob, s->bloblen);
  for (int j = 0; j < NELEMS(s->magic); j++)
    write_int16((int16_t)s->magic[j]);

  send_message_final();
}


void send_message_preamble(int16_t sn, char *ty, int16_t vn) {
  _isLittleEndian = isLittleEndian();   // write_xxx() and read_xxx() need this initialized

  write_asciiz(_msg_label); // magic start-of-message label
  crcInit();                // crc starts from here onward
  write_int16(sn);          // sequence number
  write_asciiz(ty);         // message type
  write_int16(vn);          // message version
}

void send_message_final() {
  write_int16((int16_t)crcDone());
  write_break();
}


struct example_msg ex_msg;

void send() {
  ex_msg.errcd=0x1234; ex_msg.errmsg="my errmsg text here"; ex_msg.severity=-9.3;
  ex_msg.stack[0] = "1st";  ex_msg.stack[1] = "2nd";
  ex_msg.stack[2] = "3rd";  ex_msg.stack[3] = "4th";
  ex_msg.stack[4] = "5th";  ex_msg.stack[5] = NULL;
  ex_msg.stack[6] = "7th";  ex_msg.stack[7] = "8th";
  ex_msg.stack[8] = "9th";  ex_msg.stack[9] = "10th";
  ex_msg.blob = "this is a BLOB without trailing zero";
  ex_msg.bloblen = strlen(ex_msg.blob);
  ex_msg.magic[0] = 0xE301; ex_msg.magic[1] = 0xE302;
  ex_msg.magic[2] = 0xE303; ex_msg.magic[3] = 0xE304;
  ex_msg.magic[4] = 0xE305;

  send_message_example_1(0x5678, &ex_msg);
}


Calling send() above will produce the output below.

Notice that null-terminated strings (char *, asciiz) use the \0 as a field delimiter.  Ints just occupy a fixed space (2 bytes) and are in big-endian format. One could change endian-ness or expand to 4, 8 bytes.  I'm not sure if signed/unsigned would need to be accommodated; not too many ones-compliment machines around.  write_float() just prints <whole>.<fraction> as asciiz.  And opaque blobs (which may contain zeros) require a length integer preceeding it. Pointers to/arrays of sub-structs can be described in their own encoder/decoder functions.

Code: [Select]

./mmm s | od -A x -t x1z -v
000000 4d 59 4d 53 47 00 56 78 45 58 41 4d 50 4c 45 5f  >MYMSG.VxEXAMPLE_<
000010 54 59 50 45 00 00 01 12 34 6d 79 20 65 72 72 6d  >TYPE....4my errm<
000020 73 67 20 74 65 78 74 20 68 65 72 65 00 2d 39 2e  >sg text here.-9.<
000030 33 30 30 30 30 30 00 31 73 74 00 32 6e 64 00 33  >300000.1st.2nd.3<
000040 72 64 00 34 74 68 00 35 74 68 00 00 37 74 68 00  >rd.4th.5th..7th.<
000050 38 74 68 00 39 74 68 00 31 30 74 68 00 00 24 74  >8th.9th.10th..$t<
000060 68 69 73 20 69 73 20 61 20 42 4c 4f 42 20 77 69  >his is a BLOB wi<
000070 74 68 6f 75 74 20 74 72 61 69 6c 69 6e 67 20 7a  >thout trailing z<
000080 65 72 6f e3 01 e3 02 e3 03 e3 04 e3 05 00 57     >ero...........W<


The receiver writes all characters as they arrive to a circular buffer (enough space to accommodate the largest message). Once a BREAK signal is detected, the current buffer is then processed. The receiver doesn't know what to expect, except that all messages have a envelope consisting of a magic label to denote start of message, message sequence number, type, version, some amount of fields, then a CRC. Assuming the last two bytes before the BREAK are CRC, the code can recalculate/validate the whole message before it even attempts to decode the envelope or payload fields.  The assumption being that noise can add/change random characters.

The recv_message() function (see below) is called after a BREAK is received to determine which message was sent. It will read and validate the envelope, then call the appropriate decoder function (like recv_message_example_1()) to re-hydrate the original structure.

Code: [Select]

int8_t recv_message(int16_t **snp, char **typ, int16_t **vnp, char **msgp) {
  uint8_t e;

  static int16_t sn;  // allocate sequence number
                      // recv_message_preamble() will allocate for type
  static int16_t vn;  // allocate message version

  *snp = &sn;         // tell caller
                      // recv_message_preamble() will tell caller
  *vnp = &vn;         // tell caller
  *msgp = NULL;       // init in case of early exit

  if ((e = recv_message_preamble(*snp, typ, *vnp)) < 0) return e;

  if (*typ != NULL) {
    if (strcmp(*typ, _msg_type_example) == 0 && vn == 1) {
      static struct example_msg ex_msg;
      if ((e = recv_message_example_1(&ex_msg)) < 0) return e;
      *msgp = (char *)&ex_msg;
    } // else if(strcmp(.....) == 0 && vn == ?) {
  }


  if ((e = recv_message_final()) < 0) return e;

  return SUCCESS;     // structure has been filled
}


int8_t recv_message_example_1(struct example_msg *s) {
  char *p;

  if ((p = read_int16()) == NULL) return EOBUF;
  s->errcd = (int)(*((int16_t *)p));

  static char errmsg[50];
  if ((p = read_asciiz(errmsg, sizeof(errmsg))) == NULL) return EOBUF;
  s->errmsg = p;

  // TODO: finish up the rest

  return SUCCESS;     // structure has been filled
}


int8_t recv_message_preamble(int16_t *sn, char **ty, int16_t *vn) {
  uint8_t e;
  char *p;

  *sn = 0;      // init in case of early exit
  *ty = NULL;   // init in case of early exit
  *vn = 0;      // init in case of early exit
 
  _isLittleEndian = isLittleEndian();   // write_xxx() and read_xxx() need this initialized

  if (!find_asciiz(_msg_label)) return NOMSG;     // no message leader found

//  crcInit();
//  if ((e = peek_validate_crc()) < 0) return e;    // non-destructive read until end of buffer to check crc in last sizeof(CRCTYPE) chars
 
  crcInit();

  // sequence number
  if ((p = read_int16()) == NULL) return EOBUF;
  *sn = *((int16_t *)p);

  // message type
  static char msgty[10];
  if ((p = read_asciiz(msgty, sizeof(msgty))) == NULL) return EOBUF;
  *ty = p;

  // message version
  if ((p = read_int16()) == NULL) return EOBUF;
  *vn = *((int16_t *)p);

  return SUCCESS;
}

int8_t recv_message_final() {
  char *p;

  if ((p = read_int16()) == NULL) return EOBUF;
  uint8_t sent_crc = (uint8_t)(*((int16_t *)p));

  if (sent_crc != crcDone()) return BADCRC;

  return SUCCESS;
}


I haven't finished the receiver part just yet; seg-fault somewhere + unfinished decoder code.
Yeah, I went a bit overboard with double indirction (type **) and static function vars as a way to not use malloc().