ASN.1 / BER / DER vs your own byte code system

[jnz]

Has anyone else used ASN.1 notation or its versions BER / DER / JER / XER / PER on their embedded systems?

I have a need to send the micro data, but won’t be 100% sure of what or how much data, nor that both sides of the transfer will be the same version.

My data may be a downloaded firmware or a uploaded log, or a list of application entries. By using a standard format like DER I think it should be possible to cover all data types even “unknown” types  -  but not sure if the overhead is dumb or there are more simple standards.

If i codes my own opcode/byte odd format in TLV triplet (Type Length Value) it just won’t be as “tolerant” to different versions or ill hit a wall by for example only reserving two bytes for length or one byte for type, etc.

ASN.1 seems to provide some good “rules” for transferring data with type information but I’ve never seen it mentioned here.

Thoughts?

[dgtl]

I've used different encodings, including ASN1. When raw binary gets too messy, I like to use google protobuf encoding. There are decoder libraries for uc's; for simpler data I've written my own small implmentation (ie no submessages etc).
The communication protocol usually needs to have its own headers/footers anyway (command code, crc etc), so it does not make sense to mix all different use cases into one message format. The different use cases (ie send FW update, send config, read status) are usually handled by the upper layer; the encoding is used for the data within (ie configuration packet). But as the structure of the data itself gets more complex and may change over time, it is better to use an extendable format for the data itself. Of course it will have overhead, but this size cost usually is not a problem with microcontrollers nowadays.

[nctnico]

In my experience the best thing to do is to use a text based communication format. It is easy to extend (without breaking backward compatibility) and super easy to debug. I have loads of these in the field even in situations with mixed versions / mixed capabilities.

[jnz]

In my experience the best thing to do is to use a text based communication format. It is easy to extend (without breaking backward compatibility) and super easy to debug. I have loads of these in the field even in situations with mixed versions / mixed capabilities.

BLE. I'm working in 20 byte chunks on most BT 4.2 devices. I have to join chunks anyhow, but worst-case we're talking 20 bytes every 50ms for older devices, the overhead with an encoded protocol is bad enough, text would be nice in some regards but WAY too heavy.

[jnz]

I've used different encodings, including ASN1. When raw binary gets too messy, I like to use google protobuf encoding. There are decoder libraries for uc's; for simpler data I've written my own small implmentation (ie no submessages etc).
The communication protocol usually needs to have its own headers/footers anyway (command code, crc etc), so it does not make sense to mix all different use cases into one message format. The different use cases (ie send FW update, send config, read status) are usually handled by the upper layer; the encoding is used for the data within (ie configuration packet). But as the structure of the data itself gets more complex and may change over time, it is better to use an extendable format for the data itself. Of course it will have overhead, but this size cost usually is not a problem with microcontrollers nowadays.

Yea, I copy on most of that. I saw Apache Thread and Google Protobuf, but both of those looked very different from ASN.1's effects. Do you have a link to a decoder library for a uc? I look into a bit and never ever saw a text description for how it's packed at the byte level. I assume it's some form of TLV triplets?

The thing is, yes, all the message types are upper layer, I just saw that I had four or more different needs for custom protocols and if I was being smarter, maybe I could just use one extendable protocol to cover them all by using types. Doing it in JSON would be MUCH more clear, but ascii text is way too heavy.

This is a bluetooth project so 20 byte chunks, the overhead could stack up to be a real factor on old devices. To the point that 1.3.6.1.4 just to get into the "private range" would be an extra 4 bytes for every type identifier. I sort of wonder if you can break spec and just go into the "3." undefined arc?

[NorthGuy]

I just saw that I had four or more different needs for custom protocols and if I was being smarter, maybe I could just use one extendable protocol to cover them all by using types.

Exactly. Why re-invent the wheel? Perhaps the same wheel will work for tricycle, Ferrari, Hummer, some tractors and even may be re-used in the TV show "Wheel of Fortune".

[tggzzz]

I don't think you have yet discovered the "touchstone" question, the answer to which will indicate which way you should or shouldn't go. Here's a few questions...

Will the information change during development?
Will the information change after first development, e.g. upgrades?
Will backwards compatibility have to be maintained?
Will the information change when deployed in the field?
Is small code/RAM size more/less important than small message size?
Is interoperability with other manufacturers important?

[jnz]

I don't think you have yet discovered the "touchstone" question, the answer to which will indicate which way you should or shouldn't go. Here's a few questions...

Will the information change during development? YES
Will the information change after first development, e.g. upgrades? YES
Will backwards compatibility have to be maintained? PROBABLY
Will the information change when deployed in the field? PROBABLY
Is small code/RAM size more/less important than small message size? CODE/RAM IS LESS IMPORTANT THAN MESSAGE SIZE
Is interoperability with other manufacturers important? NO

As it were, I'm looking at protobuffers right now as the support is high for languages the server and app will understand. However... It really appears that known messages TYPES in known message order is vital. If I knew what a message was before hand I might have an easier time with TLV triplets, seems odd they didn't build that type of support in.

[rounin]

I"m using asn1 and the asn1scc compiler - https://github.com/ttsiodras/asn1scc on a project atm. It has been working well for me. ASN1SCC is an embedded friendly compiler, its used by the ESA for spacecraft. I played with a few other ASN compiler that I would pick if it wasn't an embedded project, I don't remember which.

It generates parsers that don't need heap and have worst case size constants for packing and unpacking so you can statically allocate the space if you want/need to. Downside is you can't have unbounded variable length arrays (bounded variable length is ok though, but it will increase the size it reserves for decoding!) and some of the fancier built in types. It also can generate XER encoding code, which is nice, actually. On the laptop I can export to XML, edit a config file, and recompile to BER for transfer to the embedded target. I have 2 little 20 line c++ programs to handle generate default blob -> xml -> ber and ber -> xml.

I've also use google protobuf a lot server side, and there is supposed to be a embedded library called nanopb https://github.com/nanopb/nanopb, although I haven't tried it. Protobuf is a little too happy with dynamic memory for me now (working on a soft real time system, for general use its fine).

I liked that ASN.1 lets me be way more explicit about what the valid ranges for integers are, a big win over protobuf. I have fields that take several disjoint ranges of values, easy. Like INTEGER(0 .. 51 | 554 | 8889 .. 9999) is legal and enforced by the decoder. Its nice when you need to be specific...

[jnz]

I've also use google protobuf a lot server side, and there is supposed to be a embedded library called nanopb https://github.com/nanopb/nanopb, although I haven't tried it. Protobuf is a little too happy with dynamic memory for me now (working on a soft real time system, for general use its fine).

Great notes on ASN.1 use. Yes, I noticed that most ProtoBufs implementations used malloc and dynamic mem which I'm trying to avoid. nanoPB was designed to not do this, at least the guy went and made it an initial direction to not need dyn mem. I like that.

[lukier]

Thanks for the info on ASN.1 on embedded. I've tried nanoPB and it was quite cumbersome.

Related on the subject
https://scottlocklin.wordpress.com/2017/04/02/please-stop-writing-new-serialization-protocols/

[linux-works]

protobuf from google is a PITA.  changes a lot version to version, sometimes you get version-hell.  typical google over-engineering, sigh ;(  I see lots of projects using it but I don't love it.  makes my life harder than it has to be.

I like ascii.  readable debuggable ascii (ie, text).

I'm doing lots of python these days, so I'll use json or even just python dict()'s.  or some shortened form of json if mem is tight and i/o speed may be slow.

for a long time, I did lots of SNMP work and asn.1 was the BER for that; but I would never use that stuff today if it wasn't already part of a standard.

simple text always wins, with me, over binary formats.  very rarely do I have to squeeze every byte out.

[dmills]

Simple text is good as any right thinking unix/linux developer will tell you, but for more complex things I would take ASN.1 over say XML any day of the week.

XML claims to be human readable and editable, but have you ever tried to actually do it in something non trivial, think I would rather hand edit sendmail.cf, greater chance of the edit actually leaving a valid file.
To be fair, XML is probably fine when your data is naturally a tree structure and overhead really does not matter, but that is seldom really the case in my world.

The nice thing about ASN.1 is that it is actually properly defined, by folks who knew how to write a (mostly) unambiguous standard, (for all that it is written in the dialect of standardsese known as 'phone company'), would that quite a few 'web technology' standards were as well thought thru.

Regards, Dan.

[nctnico]

Simple text is good as any right thinking unix/linux developer will tell you, but for more complex things I would take ASN.1 over say XML any day of the week.

XML is a bit bloated for exchanging commands and/or giving smallm status updates but think about the SCPI text based commands used by test equipment.

[westfw]

I'm doing lots of python these days, so I'll use json or even just python dict()'s.

I've been looking at the Python JSON library recently, and it seems that if I have something like nested dictionaries (pseudocode: filelist[filename].valuelist[valuename] = true), the effort required is a lot less if I just use some messy native representation

Code: [Select]

filelist = {  "foo.txt": { "item1":1, "item2":0, "item3":22 }  "bar.txt": { "other1":1, "other2":0, "other3":255 }}creating json is very painless and easy, while if I want to wrap that in some defined datastructure (classes, I guess) so that I can read my own code, generating the json suddenly becomes more complicated (needs a custom JSONEncoder) - am I missing some obvious simple method for "boring" datastructures (trivial custom Encoder, perhaps?)

[calin]

Have a look at MessagePack .... just my 2c

[voltsandjolts]

I would recommend COBS for transport, then a message format of your choice.
https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing
https://www.eevblog.com/forum/microcontrollers/how-should-be-a-serial-communication-packet-format/

[dmills]

Note that COBS has a 255 byte message length limit (Actually it is a limit of 255 bytes between \0 IIRC), so it is a poor fit for some uses, but when it fits the job it is very cool.

Regards, Dan.

[voltsandjolts]

Note that COBS has a 255 byte message length limit (Actually it is a limit of 255 bytes between \0 IIRC), so it is a poor fit for some uses, but when it fits the job it is very cool.

That is wrong. COBS does not have a data length limit.
You are confusing section length with message length.
A COBS section is max 254 bytes longs before a COBS control byte must be inserted but the output message (from 0x00 to 0x00) can be any length.