Author Topic: Bakelite: A utility that makes it simple to communicate with your firmware  (Read 3447 times)

0 Members and 1 Guest are viewing this topic.

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Hello! I've just released the first version of my new open source (MIT) library called Bakelite, and thought people might find it useful. It's intended to make building custom protocols for serial/USB/etc less tedious. It does this by providing a code generator and library that provides struct serialization, framing, and error detection. Currently there's a Python and C++ implementation, but I hope to add new languages in the future.

Features:
  • Compact, easy to understand data serialization format
  • Simple message passing
  • Built in framing and error detection
  • Easy to integrate with Serial, USB, TCP, SPI, etc...
  • Use only the parts you need
  • Code generators for:
  •     C++ (header only, no STL or memory allocation)
  •     Python
  • Documentation  :)
If you're interested, you can find it here: https://github.com/brendan0powers/bakelite
 

Offline dmendesf

  • Frequent Contributor
  • **
  • Posts: 320
  • Country: br
Let me ask you an unrelated question: What tool did you use to make the SVG diagram in your project page?
 

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Google Drawings. It's pretty clunky, but it's free and was faster than searching for a real diagram tool.
 
The following users thanked this post: dmendesf

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Cool.
A code generator for plain C would be nice for us old-fashioned folks ;D
 

Offline FlyingDutch

  • Regular Contributor
  • *
  • Posts: 144
  • Country: pl
Hello,

maybe tomorrow I would like to try this project, it looks very interesting.

Best Regards
 

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Quote
A code generator for plain C would be nice for us old-fashioned folks ;D

Yes, a pure C implementation is on the to-do list. It would be particularly useful in situations where you don't have a modern C++ toolchain available. Working with an 8051, for example. Is there a specific C standards version I should target?
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
C89 would give maximum coverage I suppose, but if C99 makes things easier I would be happy with that (thinking of my older Microchip gcc compilers and sdcc for stm8/8051).
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
At a glance, I think you are using COBS for framing :-+
Plus options for CRC8/16/32 over the payload.
Nice.

Edit:
Confirmed, just seen COBS mentioned in your docs. It's the best option for serial framing over uart.
Perhaps another option would be no framing or crc, if using usb, and the rpc calls were always shorter than endpoint size (could raise error during bakelite build if not).
« Last Edit: January 13, 2022, 05:27:07 pm by voltsandjolts »
 
The following users thanked this post: brendan0powers

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14445
  • Country: fr
A number of us have our own solution for this, but OTOH, communication protocols in general is a frequently asked question, so I'm sure it can be useful for some.

I've never used COBS, but since it's mentioned, I'll have a look and see if/when the overhead is worth the trouble.
 
The following users thanked this post: brendan0powers

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Quote
Confirmed, just seen COBS mentioned in your docs. It's the best option for serial framing over uart.
Perhaps another option would be no framing or crc, if using usb, and the rpc calls were always shorter than endpoint size (could raise error during bakelite build if not).

COBS is a great choice for a framing algorithm. The biggest issue with it is that it requires at least a 255 byte buffer. At one point I was hoping to do away with the read/write buffers. I could still get rid of the read buffer, but the implementation gets a fair bit more complicated. In the meantime, if the maxLength is set to something under 255 bytes, I can assume I'll never need the full 255 byte buffer.

For the USB use case, you can disable CRC, but you can't disable framing. The current API wouldn't even support this, there's no way to tell the decode that a frame has ended. I'll have to think about that use case a bit more.
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #10 on: January 13, 2022, 07:46:37 pm »
Here's a COBS topic I posted some time ago, with some python and C code, which might be useful for someone interested in learning about it.
https://www.eevblog.com/forum/microcontrollers/implementing-uart-data-packets-with-consistent-overhead-byte-stuffing-(cobs)/

Yeh, with usb providing structured serial, COBS would be unnecessary processing there.
Maybe just a simple byte count at the start of the frame would be sufficient in that case.

Google Protocol Buffers always seemed a bit OTT for my needs.
With a plain C code generator, this could be a much better fit.
 

Offline poorchava

  • Super Contributor
  • ***
  • Posts: 1672
  • Country: pl
  • Troll Cave Electronics!
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #11 on: January 22, 2022, 08:53:50 am »
COBS makes sense, although I don't see the reason for constant length. Framing encoding in general makes life mich easier.

I've been using ordinary byte stuffing (start, stop, escape, xormask) for a while and can't see any reason to change.
I love the smell of FR4 in the morning!
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6242
  • Country: fi
    • My home page and email address
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #12 on: January 22, 2022, 10:10:42 am »
I prefer protocols that can be handled with minimal buffering.

Like others, I use my own, additionally multiplexing a single channel for multiple separate data streams.  (My Odroid-HC1 has only one 1.8V UART, so I developed a simple way to multiplex console with peripheral (GPIO) data.  The UART is interfaced to a microcontroller with native USB, which can be connected to a separate machine providing two USB endpoints; one of which is the serial console, and the other is the peripheral/GPIO data.)
Simple byte stuffing with a carefully chosen escape mechanism means I can switch between streams at almost any position, with a maximum buffer delay of a single byte, with only a few bytes (less than half a dozen, depending on the maximum number of data streams) of total overhead.
 

Offline HwAoRrDk

  • Super Contributor
  • ***
  • Posts: 1471
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #13 on: January 22, 2022, 10:58:58 am »
Let me ask you an unrelated question: What tool did you use to make the SVG diagram in your project page?

Speaking of which, that diagram is partly illegible when using GitHub's dark mode, because the background is transparent.
 
The following users thanked this post: brendan0powers

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #14 on: January 22, 2022, 02:30:18 pm »
Quote
Speaking of which, that diagram is partly illegible when using GitHub's dark mode, because the background is transparent.

Thanks for pointing this out! I've updated the image with a white background.
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #15 on: January 23, 2022, 10:13:15 am »
COBS makes sense, although I don't see the reason for constant length.
You've got that wrong, a cobs frame can be any length.

Quote
I've been using ordinary byte stuffing (start, stop, escape, xormask) for a while and can't see any reason to change.
Sounds much like slip which is ok but if your data happens to contain a lot of your chosen escape character, then the overhead goes to 100% (link bandwidth halved).
 

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #16 on: January 24, 2022, 02:56:31 pm »
Quote
Sounds much like slip which is ok but if your data happens to contain a lot of your chosen escape character, then the overhead goes to 100% (link bandwidth halved).

This was the reason for going with COBS. No need to try and find an escape character that's unlikely to be used in your messages. On the other hand, the buffering requirements for COBS might make supporting SLIP worth it.
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #17 on: January 24, 2022, 03:39:54 pm »
Yeh, why not, users like configurable options.
The buffering requirements for cobs can be minimal. Decoding received data is trivial, it can easily be done within the rx interrupt, without any additional buffering. With a little effort, transmission encoding can actually be done 'in-place' in the tx message buffer itself, as long as the buffer size can hold the few extra bytes required for overhead (see my thread linked above). Of course, if you're looking for more performance, then perhaps ping-pong buffers for DMA transmission would be used, which needs more ram but that's not a big issue on modern arm cortex. For cobs I think the juice is worth the squeeze!
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 8168
  • Country: fi
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #18 on: January 24, 2022, 04:11:41 pm »
I don't see the RAM use of buffering problematic, and for transferring large amounts of non-realtime (or "easy realtime") data I don't see the issue with delay, clearly it's an excellent encoding, especially recovery from byte loss/corruption (framing errors) in predictable time.

However, am I missing something if I say adding 255-byte delay between the generation of data (you can't look ahead something you don't yet have no matter how much RAM you have at your disposal), to data hitting the target, is a massive turn-off exactly in realtime embedded work, I don't mean audio or video streaming, but communicating smaller amounts of structured data where you need to react to the inputs quickly. For data logging / streaming, this doesn't matter, but subject says "communicate" which is different I think.

Also the claim that this delay can be made predictable and consistent in number of bytes is a red herring, because what actual matters is absolute time in seconds, and that is far from predictable, unless you send stuffing bytes all the time at the full maximum expected communication speed. Maybe this is exactly what is being done? But still, this all makes me think that this might not be the best possible protocol after all for generic microcontroller communication use.

Me, I tend to use the classic magic + msgid + len + payload + crc paradigm which causes minimal delay and keeps the raw data stream intact and only looks in it for CRC calculation. But framing error is only caught at CRC step, after which scanning for a sync packet is required, making it lose some data during that wait, so not very good for unreliable links. But I also tend to write different implementation every time, because microcontroller projects are simple but tend to have some weird specific requirements.

The only byte-delimited byte stream in MCU world I can think of is UART. UDP over Ethernet comes with packet delimitation. TCP over Ethernet comes with guarantee of no lost bytes so length + payload works perfectly. Raw ethernet frames come with packets. USB - packets. SPI - again, hardware delimited packets (despite the fact some don't understand this and do not use it in their advantage). Radio communication: packets.

Using mediums that transfer packets - just to emulate byte streams over them - then to realize you needed packets not byte streams - just to add another layer of code emulating packets - seems unnecessary work. The total amount of buffering and delay is obviously then more than just needed for the upmost (COBS) layer.

But I'm probably missing something, the intention is not to criticize, but to ask what I am missing.
« Last Edit: January 24, 2022, 04:22:38 pm by Siwastaja »
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6242
  • Country: fi
    • My home page and email address
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #19 on: January 24, 2022, 05:15:30 pm »
  • Not all packetized transports expose the packet boundaries to userspace; for example, USB Serial.
    Even when packet boundaries are exposed, it may not be possible to determine if two consecutive packets were sent in a single transmission or not.
    It is even possible for packet order to change, although this usually only happens on the internet.

    This means that if one relies on packets, each packet must be somewhat self-contained.  If the packet length is determined by hardware –– for example, using UDP over IP you'll want to use 508-byte packets to maximize bandwidth and minimize overhead and the number of dropped packets; or say as in USB –– you may need to encapsulate the application-sized packets anyway, and distribute them over multiple hardware packets.

    See IP fragmentation for a particularly relevant real world case.
     
  • It is often more efficient (and sometimes necessary; e.g. with my Odroid HC1 which only exposes a single UART and no GPIO pins) to multiplex a single "physical" connection for one or more logical data connections.

    A particularly good example is TLS-encrypted TCP connections.  The encryption itself is rather lightweight, compared to the computation needed during the two-way handshake.  If you can use a single encrypted connection for all your different logical data or packet streams, you will need less computation (cryptographic math, especially public-key crypto using key lengths in the thousands of bits) and get better bandwidth on the same hardware, than when using a separate encrypted TLS connection for each logical data or packet stream.

    If those streams have different priorities, you either need to buffer each logical stream separately (which can lead to quite a bit of buffer space needed), or you need a way to interrupt/insert data from a new logical stream on the transport at basically any point.

    Even the TCP protocol itself has a notion of out of band data ("urgent, process me first, recipient, please and thank you"), although it is rarely used and may not be correctly implemented in the TCP/IP stack one uses.
I myself don't always need additional error correction, because the underlying transports I use (UART, USB, and TCP over IPv4/v6) all have some error correction already built-in.

I do not like COBS, because of the lookahead buffer size.  I do not know how one could multiplex multiple COBS streams on top of one transport, without having a separate lookahead buffer for each logical stream; that amount of buffering (and buffer bloat) is too much in an embedded environment for me.

I also do not need to know the packet size beforehand, as almost always the packet can only be processed when complete.  In a very real sense, even text-based command streams like Gcode and HPGL can be considered delimited packets.
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #20 on: January 25, 2022, 10:29:34 am »
However, am I missing something if I say adding 255-byte delay between the generation of data (you can't look ahead something you don't yet have no matter how much RAM you have at your disposal), to data hitting the target, is a massive turn-off exactly in realtime embedded work, I don't mean audio or video streaming, but communicating smaller amounts of structured data where you need to react to the inputs quickly.
...
The only byte-delimited byte stream in MCU world I can think of is UART.

Agreed, lets forget about streaming. We are discussing embedded mcu comms with uart here i.e. small amounts of structured data over an unstructured link. Lets say you have a block of data bytes in a buffer ready to send, the data can include any values and is any non-zero length. The cobs encoding removes all 0x00 bytes from the data so that 0x00 can be used as the framing character, marking the start and end of a cobs frame. The encoding is trivial but neat, in essence 0x00 data bytes are replaced with a cobs byte which is the distance to the next 0x00 data byte. It takes very little processing to encode the data bytes, essentially a loop over all input bytes and write encoded data to another buffer. So, microseconds of processing, which is a near irrelevant delay compared to the uart transmission itself.


Also the claim that this delay can be made predictable and consistent in number of bytes is a red herring, because what actual matters is absolute time in seconds, and that is far from predictable, unless you send stuffing bytes all the time at the full maximum expected communication speed. Maybe this is exactly what is being done? But still, this all makes me think that this might not be the best possible protocol after all for generic microcontroller communication use.

We are not talking about steaming. There is no delay. Get a block of data, cobs encode it in microseconds and send it with 0x00 start and end characters.

The '255 byte delay' confusion stems from the maximum distance between cobs bytes. All zeroes are replaced with a cobs byte which is just the distance to the next cobs byte (i.e. the next 0x00 in data). If the distance is more than 0xFE, then a cobs byte of 0xFF must be inserted (creates some overhead), meaning no zero here, next cobs byte is 255 bytes ahead. When encoding you have to find the position of a zero, then go back to the last zero and write the distance there i.e. the cobs byte. Hence the buffer requirement for a 256 byte (or more) circular buffer for streaming encoding.
 

Online voltsandjolts

  • Supporter
  • ****
  • Posts: 2299
  • Country: gb
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #21 on: January 25, 2022, 10:41:16 am »
I do not know how one could multiplex multiple COBS streams on top of one transport

Lets forget about streams and instead focus on data packets.
COBS simply provides a way of framing data packets, the length and content of data is entirely up to you.
It provides bandwidth-efficient structure over unstructured serial links i.e. uart / tty / com ports.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 8168
  • Country: fi
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #22 on: January 25, 2022, 11:00:48 am »
Oh, I got it, thanks for the explanation. You can always send just one packet, and need only lookahead within this packet, or if packet is longer than 255 bytes, within that 255 bytes; whichever is smaller, and this is because the end-of-packet delimiter is added, and distance to this zero is what needs to be known during processing.
« Last Edit: January 25, 2022, 12:07:21 pm by Siwastaja »
 

Offline brendan0powersTopic starter

  • Newbie
  • Posts: 9
  • Country: us
    • My Blog
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #23 on: January 26, 2022, 05:29:24 pm »
Quote
Also the claim that this delay can be made predictable and consistent in number of bytes is a red herring,

If you really care about end-to-end latency being predictable, then COBS has a significant advantage. If you're in a situation where most of the time taken sending a message is transmitting the bytes over your serial link, then the number of bytes you transmit will determine the latency. COBS ensure that latency has a fixed upper bound. You never get into a situation where your message ends up taking twice as long to transmit because you got unlucky.

In practice, I don't think that situation isn't all that common, at least in the systems I work with.  If consistent latency isn't all that important to you, then byte stuffing works fine.
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6242
  • Country: fi
    • My home page and email address
Re: Bakelite: A utility that makes it simple to communicate with your firmware
« Reply #24 on: January 26, 2022, 07:32:24 pm »
Here is my practical use case example, showing the reason behind my "dislike" of COBS.  (It's a hobby investigation, not a product; just something I do for fun.)

I have a host SBC with a ~ 2Mbit/s UART connection to my microcontroller.  I want to use that same UART connection for the serial console (including boot log), bulk data for displaying graphics and other information on a small display, feed back entropy from a hardware random number generator back to the Linux kernel, as well as provide some user interface events (key presses et cetera) back to the kernel.
I have my own kernel driver for the UART connection, and my own firmware on the microcontroller.

The serial console data from SBC to microcontroller should have maximum priority.  This is especially important during bootup, to not slow down the bootup process (and to avoid having a larger buffer at boot time).

The bulk data from SBC to microcontroller should have minimum priority; it should only be transmitted when there is nothing more important.

The serial console events from the microcontroller back to the SBC should have equal priority with the user interface events reported.

Feeding back some entropy from the microcontroller back to the SBC should have minimum priority; it should only be transmitted when there is nothing more important.

As you can see, I kinda-sorta have to treat each of these as a stream.  Packetizing them is really not feasible, since the bulk data could be huge, and I want the serial console data to 'interrupt' bulk data flow, and have priority.

Byte stuffing, with specific byte values reserved as "serial console data follows", "display data follows", "event data follows", and "entropy data follows", means I can interrupt at any position (except between a stuffed "escape" marker and the payload byte).  If I assume I end up with 8 reserved byte values, the escape one included, I can use 8+8×8=72 other byte values following the escape marker to encode the reserved byte values and any pair of reserved byte values, limiting the maximum overhead to 50%: when the original data stream has every other byte a reserved value, giving 2 to 3 expansion, or 150% encoded length compared to original.

(If you consider the worst case in isolation, three bytes where the first and last are reserved values, that does expand to five bytes, giving 67% overhead.  With additional implementation complexity, using 64 additional three-byte escape sequences (ESC PAIR mid -> FIRST mid SECOND) would fix that, though.  Assuming 8-bit bytes, there is just enough unique byte values for this scheme to handle up to 10 reserved byte values, but no more.)

Note that even the latency requirement/preference here is not related to the physical connection, but a direct result of the preferred priorities on what data gets transmitted first.  I do not want to wait for a full packet to be transmitted before switching to the more important, higher priority data; I want to do it as soon as possible.

I do believe that similar needs are relatively rare, though.  I should have been clearer in my earlier posts: these are my preferences due to the kind of problems I work with, not universal or directly applicable to others. For example, anyone using USB can use multiple endpoints, one for each sub-stream or channel.

This can be useful in embedded environments that use TLS encrypted TCP connections, because the 50% overhead is much preferable to the work involved in TLS handshakes for using multiple TCP connections in parallel; basically, you save on both computation and memory requirements, and can even simplify the TLS+TCP code needed.

Now, if I was doing something else, where consistent overhead was more important, and I didn't need to multiplex a single connection for multiple logical connections, I'd like COBS just fine.
« Last Edit: January 26, 2022, 07:41:15 pm by Nominal Animal »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf