structure info and guidance

[NorthGuy]

IMHO, the single advantage of text-based protocols is the ability to view them using proverbial "terminal" software. If you want to observe other protocols, you need something different, such as USB packet viewer for USB, CAN analyzer for CAN, or your own software for custom protocols. Not to mention that observing your protocol in a binary form may be quite sufficient.

For the ability to use the "terminal" you pay with 2 things - more work for encoding/decoding, and more traffic on the wire.

[Siwastaja]

IMHO, the single advantage of text-based protocols is the ability to view them using proverbial "terminal" software. If you want to observe other protocols, you need something different, such as USB packet viewer for USB, CAN analyzer for CAN, or your own software for custom protocols. Not to mention that observing your protocol in a binary form may be quite sufficient.

For the ability to use the "terminal" you pay with 2 things - more work for encoding/decoding, and more traffic on the wire.

Hyperterminal or similar only gets you so far. Somehow, I always end up adding more and more useful features to my custom tooling; in such case, whether the actual protocol ends up being text or binary becomes only a small detail, since a lot of generation, parsing, transformations are going on anyway - so binary is fine anyway and gives the performance / code size / transfer time benefits.

Example of how my internal tools end up being after being in daily use for 2 years, mostly used by me so being a weird combination of sane design and some very ugly but quick-to-write hacks. In this case the protocols are binary and the tool allows arbitrary construction and parsing of binaries (not even byte alignment needed).

Code: [Select]

$ commander
Usage:
commander [server_id/]<devid>/<port_id>/<instance> [options]   (or [server_id/]<gw_devid>/gw/<dest_devid>/<port_id>/<instance>
-v                   increase verbosity level (e.g. -vvv)
-h <hostname>        broker IP address or hostname
-p <port>            broker TCP/IP port
-t <cafile|capath/>,<certfile>,<keyfile>
                     Enable TLS; use cafile, certfile, keyfile
                     cafile ending in / is treated as capath
-R                   Reset and start fresh commands; but keep broker connection open
                     After -R, everything except -v, -h, -p, -t, is cleared and re-evaluated, including devid/port/instance
                     Useful for doing many things quickly without the broker authentication spam
-O                   Send CONFMIT (not sent if other operations, usually -c, specified and they fail)

-T <id>              choose transaction id; by default incremented by tmpfile, now 247
-y                   don't check transaction id (quicker access without extra poll message)
-m                   simulate: do not actually send the command
-M                   modbus mode: swap endianness of 32-bit values and swap string char order
-L                   csv print mode - only parsed values
-LL                  csv print mode with fixed size fields - only parsed values
-LLL                 same as -LLL but fewer decimals and narrower fields
-B                   Print unicode bars
-e                   print timestamp at beginning of the line (useful with -L or -LL)
-E <modulo-seconds>  (only with -F logfile): parse exactly one record between modulo-seconds
                     e.g. -E 60: print the first record of every minute (print nothing if
                     no messages durin the minute)
                     Run rm commander_tmp_starttime first; then run the first logfile; then the rest
                     called with the same -E value will be synced to the first.
-Z                   completely omit print of -z parse units (no empty field printed)



Outgoing payload data format descriptions:
-d                   Send /do message: start description of the /do message
 OR
-c                   Send /setconf message: start description of the /conf message
-C<byteoffset>       Do read-modify-write getconf-setconf starting at byte offset
 THEN
-xN <value>          Add N-bit value interpreted from hexadecimal input
-uN <value>          Add N-bit value interpreted from unsigned decimal input
-iN <value>          Add N-bit value interpreted from signed decimal input
-bN <value>          Add N-bit bitfield interpreted from string of 0 and 1; LSbs first
-sN <string>         Add C string padded with zeroes to N-BYTE length. N IS IN BYTES, NOT BITS
-fN <filename>       Add a file; contents of file are sent. N IS IN BYTES, NOT BITS



Incoming payload data format descriptions:
-n                   Poll info message and parse it; start description of info message
-N                   Same, but only receive the next info message whenever it comes, don't poll for it
-Q                   Same, but keep receiving and don't quit
-P <string>          Instead of polling/receiving messages over mqtt, parse this hexadecimal string
-F <logfile|stdin>   Instead of polling/receiving messages over mqtt, parse logfile (stdin treated as special name)
                            compatible logfile format: mosquitto_sub -F %I\ %t\ %l\ %x
 OR
-g                   Send getconf and parse the conf; start description of conf message:
 THEN
-xN                  Add N-bit value shown as hexadecimal output
-uN                  Add N-bit value shown unsigned decimal output
-iN                  Add N-bit value shown as signed decimal output
-bN                  Add N-bit bitfield shown as string of 0 and 1; LSbs first
-sN                  Add N-BYTE string, shown as text. N IS IN BYTES, NOT BITS
-fN <filename>       Add a file; received contents are written to the file. N IS IN BYTES, NOT BITS
-f0 <filename>       Write the received conf to a file, to the length of received conf.
-zN                  Ignore N bits, do not print
-l <string>          print explanatory string at this point


optional server_id/ (single letter + /) before devid (defaults to P for prod) allows loading broker address, port and cert files
based on the following environment variables:
    COMMANDER_BROKER_HOST_<server_id>
    COMMANDER_BROKER_PORT_<server_id>
    COMMANDER_BROKER_CA_<server_id>
    COMMANDER_BROKER_CRT_<server_id>
    COMMANDER_BROKER_KEY_<server_id>
    COMMANDER_BROKER_FCR_CA_<server_id>   (for ports 3??)
    COMMANDER_BROKER_FCR_CRT_<server_id>  (for ports 3??)
    COMMANDER_BROKER_FCR_KEY_<server_id>  (for ports 3??)


   example: commander abcd1234/3/2 -b1 1
equivalent: commander abcd1234/3/2 -b8 10000000
equivalent: commander abcd1234/3/2 -b1 1 -b7 0
equivalent: commander abcd1234/3/2 -u8 1
     turn relay 2 (3th relay) on
With -i and -u, one can add a floating point multiplier (and offset) used for parsing: e.g. -i16*4.2+5.0: parsed value multiplied by 4.2, then 5.0 added.
Another hack: multiplier of -9 causes the number to be interpreted as ministamp and printed as local time
Another hack: multiplier of -99 causes the number to be interpreted as unix timestamp and printed as local time
with -xN, -uN, -iN, -bN, if N is 0, previously parsed value will be reused and re-interpreted (e.g., with -i instead of -u, or different multiplier)
FCR-related magics:
-u32*-999     Interpret as unix timestamp and print as UTC time with FCR-log-compliant formatting; also store value internally, see below:
-z999         At this point, increment last parsed timestamp by 0.1 seconds and print the incremented timestamp
-z9999        At this point, increment last parsed timestamp by 1 second and print the incremented timestamp
-iN*<mult>@   Saturate values (after applying multiplier) smaller than 0 to 0


[xvr]

Extensibility is not an inherent feature of binary vs text protocol at all. It's just that a text based protocol is always complicated enough to generate and parse, such that when writing those parsers (e.g., key and value extraction), some extensibility usually comes as a side effect.

Of course, both text and binary protocols can be made extensible or not. Text parsing is a more complex task than directly reading a binary structure.
But you can always use a standard protocol and get rid of the need to manually write parsers and solve extensibility problems.

I already mentioned Protobuf (https://protobuf.dev/).
It solves ALL the problems you mentioned.

Only one problem with it could be absence of offitial C binding (and MCU targeted binding)
Now it supports C++, C#, Dart, Go, Java, Kotlin, Objective-C, PHP, Python, Ruby, Rust
C binding also exists, but it maintained not by Google

[nctnico]

Extensibility is not an inherent feature of binary vs text protocol at all. It's just that a text based protocol is always complicated enough to generate and parse, such that when writing those parsers (e.g., key and value extraction), some extensibility usually comes as a side effect.

Of course, both text and binary protocols can be made extensible or not. Text parsing is a more complex task than directly reading a binary structure.
But you can always use a standard protocol and get rid of the need to manually write parsers and solve extensibility problems.

No. Because you'll always need to develop specific tooling which costs time and creates yet another project to maintain. Time = money. And the money needs to come from somewhere.

And at the moment you are going to develop a record based binary protocol, you might as well do text parsing. Keep in mind string compare exits at the first mismatching character; it doesn't go through the entire string so testing message type versus command is not going to make a huge difference in processing speed.

The same goes for size. Text based protocols allow to skip fields which don't change and only use enough characters to convey the value. In a binary protocol the field sizes are fixed so there is a lot redundant data in there. All in all there is very little difference in the amount of data to be transmitted.

[xvr]

No. Because you'll always need to develop specific tooling which costs time and creates yet another project to maintain.

Did you know what Protobuf is?

You don't need to 'develop specific tooling' - it already developed for any possible project.

Keep in mind string compare exits at the first mismatching character; it doesn't go through the entire string so testing message type versus command is not going to make a huge difference in processing speed.

What to faster - compare one byte at record header or compare unknown number of bytes at begining of string?

Text based protocols allow to skip fields which don't change and only use enough characters to convey the value. In a binary protocol the field sizes are fixed so there is a lot redundant data in there.

Who force to include records for unchanged data in binary protocol? Both of them can skip some data - it is not a question of text/binary, but question of data format/structure.

[Siwastaja]

The same goes for size. Text based protocols allow to skip fields which don't change and only use enough characters to convey the value. In a binary protocol the field sizes are fixed so there is a lot redundant data in there. All in all there is very little difference in the amount of data to be transmitted.

Sure. Being able to skip fields means stateful communication, desynchronization of the data being shared, and implicit "last values" that A) you don't even know if they have been shared (e.g., system boot), B) what are the last values; unless specially queried.

This is all more work, and higher risk for hard to find bugs.

Often it does not matter, of course. Neither the extra overhead of transmitting every value every time, though.

The latest binary protocol I designed simply updates the whole state of the subsystem every time, even when it's a bit wasteful, maybe just one field actually changed. That leaves a record in the database though what the instantaneous values were - even if they were the same. Simplicity is valuable, as you say.

The other direction, commands, are prefixed with a transaction ID, and the reply indicates a copy of this ID, and success code, so that whoever did the command can check if the response was generated after the command was processed, and that no other commands intervened.

To you, I'm sure, these kind of details do not matter. You type the commands in Hyperterminal, and get a reply. If it does not work, you retry. Works with humans; you are subconsciously the parser, error checker and recovery system. With a machine and asynchronous link like UART, if tx is being buffered, you may be processing old input data as a reply to your command. Your protocol is text-based, and hence automagically Good, Easy and Bug-free.

My experience on project-specific tooling is that it almost always saves time and money in the longer run. Using one-size-fits-all generic tools is inefficient every time you use them. If you only need to do that randomly - not a problem. But for something you do 1000 times, spending 20 hours on a tool which saves 5 minutes every time, has saved you 63 hours. Of course, avoid the other extreme, NIH syndrome.

In recent years, I have needed mostly CSV inputs and outputs formatted in a very specific ways, for testing, simulation and debug. Surprisingly quick to write in C, both ways, given that you have the binary data and struct definitions in header files!

[nctnico]

It is not like a text protocol can't be used for automation. SCPI (to name just one example) has been made for automation. Many terminal emulators have the ability to send a command repeatedly and capture the output. Handy if you want to do a measurement over a longer period. No need to type commands manually over and over and no need to develop anythin.

And if anything else fails, you can hack a Python script together quickly. Can make nice graphs as well with a few lines of code. No way I'm going to do that in C. I get that you want to use C for dealing with binary data because C sucks at dealing with text. But that is not a problem with using a text protocol, the problem is using C for tooling.

[NorthGuy]

And if anything else fails, you can hack a Python script together quickly. Can make nice graphs as well with a few lines of code. No way I'm going to do that in C. I get that you want to use C for dealing with binary data because C sucks at dealing with text. But that is not a problem with using a text protocol, the problem is using C for tooling.

If you roll out your own protocol, you already have the C code to parse it at the receiving end. What's the point in re-writing it in Python? Simply use the same code on PC or wherever you do your protocol debugging.

[westfw]

I know that the text based ones are the easiest to implement.

"text based" protocols are seen as less efficient than binary protocols, but when you combine improvements in technology (CPUs with fast multiple and divide instructions, better "dictionary" software algorithms (sometimes incorporated into programming languages!) with the desire for type-independent binary algorithms, I'm not sure that that's true any more.

I would much rather use a text-based algorithm than go anywhere near ASN.1, for instance.

[John Coloccia]

I know that the text based ones are the easiest to implement.

"text based" protocols are seen as less efficient than binary protocols, but when you combine improvements in technology (CPUs with fast multiple and divide instructions, better "dictionary" software algorithms (sometimes incorporated into programming languages!) with the desire for type-independent binary algorithms, I'm not sure that that's true any more.

I would much rather use a text-based algorithm than go anywhere near ASN.1, for instance.

It's all about your application and goals, right? Text based is actually simpler to parse in some ways, because you have all sorts of characters available to convey special meaning.

I can think of a lot of times when text wouldn't work and I didn't even bother trying, simply because of latency or size, but the most of the time it just hasn't mattered. Yes, it's possibly way less efficient, but my processors and comm channels are mostly sitting around doing approximately nothing most of the time anyway. May as well use them to make our lives easier.

[brucehoult]

It is not like a text protocol can't be used for automation. SCPI (to name just one example) has been made for automation.

G-Code for controlling machine tools, since 1963. And found new life in 3D printers.

[NorthGuy]

G-Code for controlling machine tools, since 1963. And found new life in 3D printers.

CNC operators often write G-code by hand.

[John Coloccia]

It is not like a text protocol can't be used for automation. SCPI (to name just one example) has been made for automation.

G-Code for controlling machine tools, since 1963. And found new life in 3D printers.

I know you know this, but for those who don't, our gerbers basically g-code.