Protocols and formats for sending data between PC and MCU

[Twistx77]

Hi everyone!

For many years I have been working with devices I've made and usually they are connected to a PC either using UART, USB, BT, or WIFI (TCP-UDP).  The thing is that I always made my own "protocol" or format for data over TCP, USB, ETC.  The thing is that I was wondering if people usually does this or there is some kind of standard protocols or data formats that are usually used, kind of like JSON, MQTT, SWAP.

Do you have any suggestions I could look into?

Thanks

[helius]

In the industrial electronics field, there are several high- and low-level protocols used for communication among test equipment (ATE) and manufacturing controls (SCADA). Take a look at SCPI and VISA, for instance.

[Siwastaja]

Using a standard makes sense when there is a standard which provides the needed functionality for that particular application in an effective manner.

Since applications are case-by-case, and universal standards are difficult to impossible to design, you often roll your own - which is exactly why so many standards exist!

There is no need to feel any guilt in doing your own design, and it's often best to ignore irrelevant people who demand you follow some - almost always totally irrelevant - trendy standard. Most often, they have no clue what they are talking about since they never get anything done.

Lean, simple standards are best. And of course, sometimes it just happens there is an application-specific higher level standard that happens to suit your need directly, but sadly, this is rarely the case, and hence, you end up designing a yet another new standard.

Some sane recurring patterns make sense, however. Things like how you provide framing to low levels that provide inadequate framing (such as UART with 8-bit HW framing, inadequately small for most practical messages). In this case, I think it's a sign of good design that these things only take a few dozen lines of code, are portable, and don't require a heavy-weight library or a complex parser. Think about a complete opposite to what XML is, for example.

Beware of the "ASCII is easy" trap, often touted on these forums as well. I have seen countless of totally broken designs and complex-as-hell parser needed just because someone thought ASCII provides framing "the easy way". The fact that most terminal equipment, PCs and libraries corrupt and change line feed symbols alone breaks so many implementations that rely on these symbols for framing, in specific order. I often use ASCII anyway, but the parser is almost always at least 200-300 lines on both sides, and I don't want to lie to myself that it's "simple". Sometimes a simple binary format is one tenth of the complexity.

[Twistx77]

In the industrial electronics field, there are several high- and low-level protocols used for communication among test equipment (ATE) and manufacturing controls (SCADA). Take a look at SCPI and VISA, for instance.

Thank you I will take a look at those to see if they fit well to my application.

[Twistx77]

Using a standard makes sense when there is a standard which provides the needed functionality for that particular application in an effective manner.

Since applications are case-by-case, and universal standards are difficult to impossible to design, you often roll your own - which is exactly why so many standards exist!

There is no need to feel any guilt in doing your own design, and it's often best to ignore irrelevant people who demand you follow some - almost always totally irrelevant - trendy standard. Most often, they have no clue what they are talking about since they never get anything done.

Lean, simple standards are best. And of course, sometimes it just happens there is an application-specific higher level standard that happens to suit your need directly, but sadly, this is rarely the case, and hence, you end up designing a yet another new standard.

Some sane recurring patterns make sense, however. Things like how you provide framing to low levels that provide inadequate framing (such as UART with 8-bit HW framing, inadequately small for most practical messages). In this case, I think it's a sign of good design that these things only take a few dozen lines of code, are portable, and don't require a heavy-weight library or a complex parser. Think about a complete opposite to what XML is, for example.

Beware of the "ASCII is easy" trap, often touted in these forums as well. I have seen countless of totally broken designs and complex-as-hell parser needed just because someone thought ASCII provides framing "the easy way". The fact that most terminal equipment, PCs and libraries corrupt and change line feed symbols alone breaks so many implementations that rely on these symbols for framing. I often use ASCII anyway, but the parser is almost always at least 200-300 lines on both sides. Sometimes a simple binary format is one tenth of the complexity.

I have always thought exactly as you have explained here. Although since the project I am working on right now is focused towards education and open source, I wanted to see if there was something available and used widely or at least some kind of standard or data format with implemented libraries.

I like the binary format since I can avoid using strings but is less flexible.

On one hand binary would be more interesting for people who learns and on the other hand ASCII format is usually to understand, at least the messages for someone that have never work with binary data formats.

[towlerg]

Personally it's horses for courses. If it makes sense to use an existing standard fine, elsewise make up one that is appropriate for the task in hand.

[Jeroen3]

Protobuf.
https://developers.google.com/protocol-buffers/

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

[Twistx77]

Protobuf.
https://developers.google.com/protocol-buffers/

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Thank you very much for that info, if I don't use it I might take ideas from iot.

[djacobow]

Folks have made some good points about the pain of using text-based protocols. I don't disagree, but if you have solved the problem for how to reliably transfer strings between your host and microcontrollers, let me suggest you just go with JSON. It's flexible and straightforward, and generally easy to debug. You can add elements to your JSON that the target doesn't understand and, as long as the string still fits in memory, the target won't care. This is nice for rolling out new features to new devices, maintaining backward compatibility to old ones. On the host side, there are libraries galore in every language, so that will be trivial.

It does get a bit tricky on the controller side, especially if memory is scarce, but there are libraries for that, too. Check out Rapidjson, or JSMN for pretty minimalist JSON parsers. Or, for the ultimate in minimalist, you can try this one: https://github.com/djacobow/djs by your truly. Mine is "special" in that no buffer is required to hold the parsed tree at all. Instead, the code just 'finds' elements in the existing string. This means no memory overhead at all, at the expense of having to re-scan the string for each item you're after. I have not tested my code extensively, but I've used it successfully in a few projects.

[voltsandjolts]

Nice, I like the idea of using JSON and your library makes it more palatable on the embedded side.

[towlerg]

It may well be easier but imagine how sloooow it is. If you don't care about throughput it sounds ok but if you want to minimize uC tx/rx time maybe not so much. I'd be curious but much to lazy, to compile it and see how big it is cause after all one mans minimal may not be anothers.

[djacobow]

I think you said it earlier: horses for courses. Sometimes speed of parsing is critical, sometimes it isn't.

What I posted is just an option.

[dmills]

I would offer ASN.1 as an internally utterly horrible, but actually quite efficient choice that works surprisingly well.

there are some very useful tools that build minimal validating parsers from a textual description of the data types and ranges, and the resulting programs do validate ranges, so if you have an integer with a range of 103 to 761 you can have that enforced by the parser which is nice.

https://ttsiodras.github.io/asn1.html discusses one example of the sorts of tools available.

Regards, Dan.

[Twistx77]

Folks have made some good points about the pain of using text-based protocols. I don't disagree, but if you have solved the problem for how to reliably transfer strings between your host and microcontrollers, let me suggest you just go with JSON. It's flexible and straightforward, and generally easy to debug. You can add elements to your JSON that the target doesn't understand and, as long as the string still fits in memory, the target won't care. This is nice for rolling out new features to new devices, maintaining backward compatibility to old ones. On the host side, there are libraries galore in every language, so that will be trivial.

It does get a bit tricky on the controller side, especially if memory is scarce, but there are libraries for that, too. Check out Rapidjson, or JSMN for pretty minimalist JSON parsers. Or, for the ultimate in minimalist, you can try this one: https://github.com/djacobow/djs by your truly. Mine is "special" in that no buffer is required to hold the parsed tree at all. Instead, the code just 'finds' elements in the existing string. This means no memory overhead at all, at the expense of having to re-scan the string for each item you're after. I have not tested my code extensively, but I've used it successfully in a few projects.

Sure JSON is a good idea I thought of using it at first. The thing is complexity of the parser and throughput but I don't know yet I will try to evaluate both aproaches.

[Twistx77]

I would offer ASN.1 as an internally utterly horrible, but actually quite efficient choice that works surprisingly well.

there are some very useful tools that build minimal validating parsers from a textual description of the data types and ranges, and the resulting programs do validate ranges, so if you have an integer with a range of 103 to 761 you can have that enforced by the parser which is nice.

https://ttsiodras.github.io/asn1.html discusses one example of the sorts of tools available.

Regards, Dan.

I will take a look. Thank you.

[voltsandjolts]

Unfortunately I don't think there is any free to use ASN1 compiler with C# or Python output, which I tend to use on the PC side.

Anybody know of a free to use protocol compiler which spits out C (suitable for microcontrollers) and C#?

[carljrb]

I also use C on the MCU side and C# on the PC side (primarily).

They're working on adding that stuff to .NET Core: https://github.com/dotnet/corefx/issues/21833. In version 3, it'll finally support desktop applications: https://blogs.msdn.microsoft.com/dotnet/2018/05/07/net-core-3-and-support-for-windows-desktop-applications/.

Meanwhile, I'd look at other options. Personally, I very much dislike JSON (it feels out of place in anything but javascript and web pages, and an unnecessary pain to deal with mostly everywhere else). As for what is "better", that totally depends on the application (e.g. test equipment could use USBTMC/SCPI/LXI/HISLIP) and the transport (USB, UART, Ethernet, etc). There's no shortage of existing protocols out there.

For quick one-offs (that aren't networked), I make a simple binary message format (instead of parsing/deserializing text of an over-engineered protocol that will often have no real benefits), and send that encoded with COBS
https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

[voltsandjolts]

Yes, COBS is great for transport.
There is probably a userbase for a simple protocol compiler to make a lean-mean binary message format with decoders/encoders in C# and C.

[NiHaoMike]

I have used two approaches a lot regarding data transmission over UART links.

One is to denote two byte values as reserved, ideally ones that rarely appear in the data. One value denotes a start of frame and the other is used as an escape character. Data bytes are sent as is except for the reserved values, which are replaced with the escape character plus the value XORed with a constant so that it doesn't conflict. Such a scheme is very efficient with bandwidth, unless some condition causes more than the usual number of reserved values to appear.

The other is to convert the data bytes to hex characters, then use standard ASCII for out of band framing. The main advantage is that a standard terminal program can be used to view the traffic, but is very inefficient with respect to bandwidth.

Something else to keep in mind is that a UART link might already support some form of out of band communication! Control lines are one form (subject to race conditions), as well as sending break signals or using 9 bit signaling if both ends support it. In some cases, simply waiting can be an acceptable way to get the other end into a known state.

[thermistor-guy]

...
Beware of the "ASCII is easy" trap, often touted on these forums as well. I have seen countless of totally broken designs and complex-as-hell parser needed just because someone thought ASCII provides framing "the easy way". The fact that most terminal equipment, PCs and libraries corrupt and change line feed symbols alone breaks so many implementations that rely on these symbols for framing, in specific order. I often use ASCII anyway, but the parser is almost always at least 200-300 lines on both sides, and I don't want to lie to myself that it's "simple". Sometimes a simple binary format is one tenth of the complexity.

The key to the ASCII approach is to design the protocol so that the parser is as simple as possible. Ideally, parser = small finite state machine, so that protocol = regular grammar. Write the protocol and the parser together. If the parser starts to get tricky, change the protocol.

The advantage I find with this approach is speed of debugging: if something is going wrong you can simply log all of the traffic, and quickly scan it by eye or a text editor. The task of testing exceptional cases can reduce to: create a text file and "cat" it into a suitable pipe.

[Jeroen3]

Using readable text, because it's easy to debug, is bit of false hope. It won't make it easier in the long run.
You're better of not exchanging readability for performance, while also spending some time on an analyzer.

The bonus is that you then also have tests, since with a readable protocol you are the test, and you are not automated.

Readability by humans of what goes over the line should not be in the specifications.

[Siwastaja]

The key to the ASCII approach is to design the protocol so that the parser is as simple as possible. Ideally, parser = small finite state machine, so that protocol = regular grammar. Write the protocol and the parser together. If the parser starts to get tricky, change the protocol.

This may result in a protocol with false hopes for human-writability, which will utterly fail as soon as someone else tries to communicate with it, finding all those nice little "design simplifications" that prevent it from working using anything else than your specific encoder.

Well, when it's an actual human commanding it, we can at least use our fairly complex AI system to automatically adjust timing, sequencing, parsing... But it'll get fairly difficult when we need to write some code to do that. So, we end up looking at your parser code (if available), or documentation, and work it out "manually". In which case, ASCII becomes totally unimportant, and most likely only an extra complication. The only advantage left is human-readability. (Well, you can do worse: for example, use XML, and you won't have either human-writability or human-readability, yet you have hopes for both!)

I spent days trying to initialize a LIDAR which has total of 7 bits of configuration and command space, which would fit inside one single octet without any framing issues, but instead it implements an "easy" ASCII interface, with tens of pages of documentation, with all kinds of nasty surprises regarding timing - for example, it just outright ignores some commands at some times (and doesn't give feedback on success/failure) and replies back unexpected things, so you need a fairly complicated parser in the FSM. The funniest part? After configuration and turn-on command (which could have been one freaking single byte), it switches to a simple binary interface (which just works) for the data, since the actual data would have too much bandwidth for ASCII overhead . So there is no human readability anyway.

In simple projects, I tend to use this ASCII format:
Commands (PC->MCU): everything must fit in one byte. If not, go for something else. For example, I have used these:
c: start charging
d: start discharging
s: stop

Or in another project:
o: open the door
c: close the door
s: stop the door

Status (MCU->PC): Everything is a label=value pair, separated with spaces
time=100 curr_limit_setpoint=123 power=234 voltage=4200

This is far from perfect, but there is certain simplicity in it; especially the fact that the MCU parser will be simple-and-robust as hell because there is no framing or any state in the parser.

And, this is well controllable by both human and simple computer programs; and it's human readable (as long as one line of status fits nicely on the terminal screen line) - I tend to pad values with leading zeroes to force nice alignment. Haven't seen any problems parsing that - all atoi() and scanf() style implementations work with leading zeroes just fine.

[GeorgeOfTheJungle]

My two cents:

- I for one prefer human readable protocols, always, except when/if ultra high performance is paramount. Interactive terminal sessions are the best debugging tool there is, mission impossible with binary formats.

- You want to be able to ping the other side to check it's ready. Sort of like the "ok" reply of the AT commands of the modems.

- You want to be able to reset at any time the receiving end to a known state, the old school serial break, ctrl-c, or something else you choose. Very important.

- You may want to know that the cmd you just sent was received correctly. Easy peasy way to achieve that is to return in hex a checksum or xor or a crc of the txt of the command. You can simply ignore it in interactive terminal sessions, or check it when it matters.

- Or optionally embed the checksum, xor or crc at the end of the cmd line, like this ":xor=7f" so that the receiving end can check it's ok before executing, and abort if it's not: cmd:send missiles:chksum=deadbeef

- You may need to send a series of commands to be executed all at once, or none at all. Think about that. If you send: open the tap - wait 1 minute - close the tap and for some reason the close the tap never makes it to the receiving end, you've got a problem. So devise a way to send a group of commands to be executed, not only individual ones.