Creating a communication protocol between devices

[Red Squirrel]

Been brainstorming for a while a project I want to start on.  It's not too complicated and I'm sure there are many like it but I still want to make my own.  Essentially it will be a home automation/monitoring system where you can have various sensors/controllers throughout the house, or even across multiple buildings.   My goal is to make it very plug and play, so you would start with the master which would be a PC, it can be a raspberry pi, basically anything that runs Linux. It would run software that then talks to the central hub which talks to all the sensors. Then you would have the central hub which would be essentially an arduino/MCU board connected via USB (serial interface) to the PC.   The central hub would then have various sensors, or even other hubs connected to it.  This is where the protocol comes in.  My goal is to make this super plug and play which is why I want a custom protocol.  You plug in any device you can think of that speaks this protocol, and it would just show up as a new device and then you can configure it and do what you want from there. A device can be as simple as a dry contact i/o board, or a single temp sensor, or another hub, or even something super specific like a custom hvac unit or something (actuators, sensors etc to control/read).  Goal is to make it super expandable.    You add a device and it would essentially present input and output and other similar parameters.  It would be up to the central controller (on the PC) to decide what to do with those controls.   

So I guess my question is, what is a good reading resource on designing a protocol like this?  Essentially communicating between other MCUs over potentially long distances. (let's say, up to 1km but probably much less)  I want to read up on stuff like different ways of doing it, such as doing a single wire bus, or a two wire bus where you have a clock signal, twisted pair vs not, etc...   There's many ways to go about it I'm sure.

I'm leaning towards a 4 wire setup.  You have your ground, +5v, a clock signal, and then the com wire, which would use 0/5v to speak in binary.  The actual devices would use 2 pins for this, an input and an output, with appropriate diodes etc to ensure they don't screw with each other.  Only two devices would be able to speak at once so it would use a sort of polling system.  The clock would only go on when it's that device's turn to speak.   This is very simple rough idea mind you.  Basically open to suggestions on what to read up on while designing this.   I'm sure there's lot of pros/cons I need to consider on various ways of doing it, such as how far I can go or how many wires I use up.

I would probably lean towards atmega just because I have lot of those chips, I only really know how to do arduino at this point but I would probably want to learn how to do it more "raw" to save on memory space.  On the other hand arduino has lot of libraries so I can google "how to do x in arduino" and get way more results than "how to do x in atmega" for example.  But once I find info on how to do it on arduino I can still translate it as needed. 

[Unixon]

This is almost exactly what I've been thinking of too.

First obstacle I found is that it seems that it wouldn't be possible to have a single capable PnP protocol both on PnP/discover/routing level and at end device level because of memory requirements for end device controllers.

For upper level between main PC and hubs I'm almost convinced about an Ethernet-like network over cheap RS485/422 physical level and INDI-like (https://indilib.org/) protocol for reading and writing data. For lower level between hubs and end devices I'm looking forward to something like MODBUS with custom packet payload format or something else lightweight. For both levels each device/hub/etc must have a hardware UID for MAC, 1-wire or MCU built-in.

[Syntax Error]

Sounds a bit like CANBUS?

https://en.m.wikipedia.org/wiki/CAN_bus

[Unixon]

Sounds a bit like CANBUS?
https://en.m.wikipedia.org/wiki/CAN_bus

I've dropped CAN after long hesitation. Small payload per packet, relatively expensive transceivers, short fixed length UIDs, overall rigidness.

[pwlps]

I'm leaning towards a 4 wire setup.  You have your ground, +5v, a clock signal, and then the com wire, which would use 0/5v to speak in binary.  The actual devices would use 2 pins for this, an input and an output, with appropriate diodes etc to ensure they don't screw with each other.  Only two devices would be able to speak at once so it would use a sort of polling system.  The clock would only go on when it's that device's turn to speak.   This is very simple rough idea mind you.  Basically open to suggestions on what to read up on while designing this.   I'm sure there's lot of pros/cons I need to consider on various ways of doing it, such as how far I can go or how many wires I use up.

I'm not sure I understand, this would be a half-duplex link ? I think it will be easier to design a protocol if you opt for full-duplex like in twisted-pair ethernet.  The number of wires shouldn't be a limiting factor as 10BASE-T cables are cheap and provide more wires than you need (4 pairs).  Also if you want to go up to 1km length you don't have so much choice for the cable selection, differential signalling and well controlled impedance will become important as length increases.

[DrG]

  My goal is to make it very plug and play, so you would start with the master which would be a PC, it can be a raspberry pi, basically anything that runs Linux. It would run software that then talks to the central hub which talks to all the sensors. Then you would have the central hub which would be essentially an arduino/MCU board connected via USB (serial interface) to the PC.   The central hub would then have various sensors, or even other hubs connected to it.  This is where the protocol comes in.


For consideration:

ESP8266 or ESP32 all running servers on the sensor ends.

Main controller (also an ESP32 or ESP8266) is a client that queries each of the servers.

You can just communicate using plain socket communications. You don't even need to use HTTP, although you could if you desired the punishment.
Note this can be entirely local and you can add additional security. Your router and ISP are not needed here.

Sure, you need to think about transmission distance, but hey, no wires. Within the communications packet, you can add your standard communication information to customize the packets.

There are dozens of examples of this on the web.

[ogden]

Very plug and play? - ZigBee.

Of course, there are loads "cable guy" stuff like LonWorks, KNX, Dali to name a few. I don't think there is need for next building automation wired interface.

[Syntax Error]

A general question. What layers on the conceptual OSI model is your protocol covering? You're building for the physical layers, but how do your devices plug and play with each other? You would need to specify the protocol at the higher session and application layers too.

Open Systems Interconnection model (OSI model) wiki: https://en.m.wikipedia.org/wiki/OSI_model

[max_torque]

Most important question:  COST???

An ethernet based solution is going to bring the greatest capability, with fast, high bandwidth exchange and the possibility for nodes to include a lot of processing power.


An basic two wire serial (RS-485 etc) is going to be the cheapest, but with a much lower bandwidth and "dumber" nodes



So, which is more important, cost or performance?


If you are unlike to be exchanging a lot of data in a short time, ie the networks is mostly going to be exhanging 10's of and up to hundreds of bytes rather than thousands or millions, then the advantages of the basic serial approach are difficult to ignore.  And basic serial using simple micro's with UARTS and your own protocol mean you can make the transfer as simple or as complex as you want.

A std packet with byte stuffing and a simple check sum would allow you to exchange from say 1 to 100 bytes in each packet, and you could run a fairly low baud to keep the link reliable over longer distances.  If you use RJ plugs and std ethernet cables (cheap, easily available, relatively robust) then you could have a full duplex RS-485 link and probably power all in one.

Talking of power, this is probably your limiting factor in terms of the bus architeture!

[Red Squirrel]

I'm looking at layer 1 at the moment. And CHEAP.  Something that won't require any special transceivers etc, something I can just big bang with regular I/O and add any supporting jelly bean parts as required for things like input protection etc.  That's kinda the info I'm seeking as well since I'm not too sure about those parts.    I was actually considering doing ethernet so that standard equipment like switches etc would work to add more ports but it's much more involved than I want to get into.

Essentially once I have layer 1 decided I can start thinking about the more specifics.  An end device may very well use i2c, canbus, or whatever is supported by the particular chip being used (ex: humidity sensor), but to communicate with my custom network it would use my own protocol. I feel it just makes for a more unified experience that way as every device works the same across the board.   I may also incorporate zigbee and such if I want wireless, but my protocol would probably just ride on top of it.   Ex: there would be a wireless "sender" and "receiver" then it just translates my protocol over zigbee.  When you plug it in, it won't even know the difference as the transceivers would do all the work but it would just act like a wired device.

Regular ethernet will also be used to some degree for long distance networks, as there could be multiple "RTUs" that are part of the overall system.  But that will just ride on the normal IP network and it will be the responsibility of the actual Linux host to handle all that.  Ex: the remote RTU would just be another RPI running the same setup, but the master one is just talking to it.  It will basically work like a large hierarchy network.

The data rate is going to be fairly low, like maybe a couple KB/sec and that's probably even pushing it.    Though there's a possibility I look at higher data rates if I want to be able to control more sophisticated things like say, an array of LEDs or something.  The system would work in a polling fashion where each device is polled individually, so I don't plan it to be super fast.

[S. Petrukhin]

Initializing the exchange from the server (PC) is not always convenient. This causes all endpoints to stay awake waiting for a request. Some sensors, such as thermometers, do not make sense to poll often, they can be powered by a battery and sleep most of the time, periodically wake up, send information in a short period of time and fall asleep again.

First, you need to determine the amount of data to be transmitted, the appropriate physical layer and transmission medium for this volume. You can't rely on streaming video, for example, over the RS-485 bus.

And already using ready-made systems, for example, MQTT or making your own top-level protocol is the easiest question.

[Red Squirrel]

Actually yeah that's a good point maybe going with a polling architecture is not the best way, as if I do incorporate wireless stuff I would want sensors that can go to sleep.  Though the part that is physically connected can always still stay awake and when it gets polled it would just send the last value it got from the wireless portion.

I could do a bus where everything is just connected and talks when it wants, but I feel this is more complicated from a software point of view as you need to make sure not more than two devices speak at once.   Though this is the sort of thing I'm looking for on what to read up on.   Various techniques to have multiple devices talk to each other and best way to go about it. Whether I use a bus, or other method.

[S. Petrukhin]

Actually yeah that's a good point maybe going with a polling architecture is not the best way, as if I do incorporate wireless stuff I would want sensors that can go to sleep.  Though the part that is physically connected can always still stay awake and when it gets polled it would just send the last value it got from the wireless portion.

I could do a bus where everything is just connected and talks when it wants, but I feel this is more complicated from a software point of view as you need to make sure not more than two devices speak at once.   Though this is the sort of thing I'm looking for on what to read up on.   Various techniques to have multiple devices talk to each other and best way to go about it. Whether I use a bus, or other method.

The issue of competition for the transmission medium has long been resolved by science. If you use radio communication, then high WiFi protocols, LoRa will solve the issue of competition for you, if you use ethernet, even in the form of a bus, then you will not need to worry about competition. By the way, using the WZ5500 you can very easily organize TCP / IP easier than ever. A lot of different options, but even TokenRing without fighting for the environment, even FrameRelay. Buses such as RS-485/432 require care for traffic management. If you are planning to make some kind of home network, then one of the options is to use an IR environment that implements the principle of ethernet competition, for example.

[S. Petrukhin]

Actually yeah that's a good point maybe going with a polling architecture is not the best way, as if I do incorporate wireless stuff I would want sensors that can go to sleep.  Though the part that is physically connected can always still stay awake and when it gets polled it would just send the last value it got from the wireless portion.

I could do a bus where everything is just connected and talks when it wants, but I feel this is more complicated from a software point of view as you need to make sure not more than two devices speak at once.   Though this is the sort of thing I'm looking for on what to read up on.   Various techniques to have multiple devices talk to each other and best way to go about it. Whether I use a bus, or other method.

Learn how ethernet works at the physical level, how it deals with collisions - a very simple solution.

[Unixon]

Talking of power, this is probably your limiting factor in terms of the bus architeture!

Power distribution must be through a separate network IMO, powering from the bus works only for isolated bus transceivers and low power end devices.

Another important point is bus voltage and max current.
Bus voltage must be higher than end device working voltage to allow local filtering and precise power regulation with inexpensive linear solutions.
However, putting a voltage on a bus outside of signal voltage range is dangerous, if cable is poorly crimped or damaged this failure may have a high cost.
Low bus voltage is safe but losses are higher and this may require step-up converters at each end device.
A typical twisted pair is capable of carrying few hundreds of mA of current (200-500mA AFAIR).

Cross-powering through a bus is bad idea, balancing multiple local regulators is a nightmare and without balancing it's a failure in the first place.
Power must be distributed only from top to bottom and not the other way around.

Hub to hub connection must be isolated just because there is high probability that hub will be powered from different power sources with high voltage difference between them. Hub to end device may be or may be not isolated, that depends on a particular case.

[Red Squirrel]

Yeah isolation is something I want to keep in mind too.  Either make it part of the design so each device is isolated, or only isolate the things that really do need it (Ex: voltage sensors or devices that have a secondary power source).   I do want to keep power as part of the bus though, as it minimizes the need to provide external power to a sensor (when I say sensor here it can be an output device too like a relay), and also reduces the need to have many backup sources.   Though perhaps sending only 5v is not enough, and perhaps I can do 24 or even 48v then the isolation transformer just has a tap to bring it down to 5, but also has an isolated 48v to send to other sensors if desired. (ex: for a hub) 

One approach that's crossed my mind, but would be a bit more expensive per sensor  is to make it so I have a universal board that sensors plug into.  The board would be at the point of use and handle the isolation, step up/down of voltages etc.  The sensor would just be another board that plugs in and gets it's power and I/O pins.   The nice part about that though is you can quite easily swap out a sensor for any other type if you really want.  Like say you have a temp sensor and want one that does humidity too, you would just swap them out and the system would lose the old one and the new one would show up.   Nice for a servicing point of view too. Some sensors would be fairly inexpensive while some would be more expensive so being able to switch just the sensor and keep the magnetics (power isolation xformer, opto couplers etc) intact is a bonus.  Or if magnetics part fails you swap that out and the sensor itself is still good.   

Eventually I do want to get into more complex stuff like boards that can run a real time OS, at that point I would just use ethernet, but for now this is going to be a fairly dumb and basic network. 

[pwlps]

I'm looking at layer 1 at the moment. And CHEAP.  Something that won't require any special transceivers etc, something I can just big bang with regular I/O and add any supporting jelly bean parts as required for things like input protection etc.  That's kinda the info I'm seeking as well since I'm not too sure about those parts.    I was actually considering doing ethernet so that standard equipment like switches etc would work to add more ports but it's much more involved than I want to get into.
something.  The system would work in a polling fashion where each device is polled individually, so I don't plan it to be super fast.

I don't have any specific advice, it's just to define the possible options and to emphasize on differences between them:

1) fixed clock frequency for both sides like in rs232 and ethernet, the receiver has to recover the clock phase from the signal,
2) master-slave configuration where the clock is provided by the master side, like in SPI,
3) a fully asynchronous scheme like gpib where there is no clock signal

Each scheme has its own advantages and disadvantages/limits.

[Unixon]

One approach that's crossed my mind, but would be a bit more expensive per sensor  is to make it so I have a universal board that sensors plug into.

This approach partially solves hardware part of the problem but not completely.

This gateway device must be somehow wired to an actual sensor/actuator/etc or a MCU that controls sensor/actuator/etc. If there's no MCU in an end device that would contain a software driver for an actual sensor/actuator IC in its firmware, the gateway device itself must have a software driver for that sensor/actuator/etc and hence it's either not universal or must have a bloated firmware that knows how to talk to every piece of hardware and IC out there.

[Red Squirrel]

Yeah with this approach there would basically be two MCUs involved, a standard one that has the same programming for each one and handles the isolation etc (somewhat acting as a power management device too as there would be a transformer involved).  This portion could perhaps also handle addressing and other administrative stuff, so the sensor itself has a bit less repetitious code.   Then the sensor board that connects would have it's own MCU with programming that is specific to the onboard sensor/control.

[Unixon]

My point was that this way there will be one more protocol in the system, between gateway MCU and end device MCU with specific firmware.
The purpose of the gateway in terms of software is to offload network operation from device-specific MCU thus making its firmware smaller or just not bloating it beyond available memory, and this gateway MCU has to talk to end device MCU in an uniform way across different end devices.

And the question of what this protocol may be is huge one.
End device MCU may be as tiny as 1k AVR or as large as 1M C-M3/M4 ARM for example.
Smallest of them won't do anything above very simple IO, and the largest of them may be capable of handling network functions all by itself.

[ogden]

Most important question:  COST???

An ethernet based solution is going to bring the greatest capability, with fast, high bandwidth exchange and the possibility for nodes to include a lot of processing power.

An basic two wire serial (RS-485 etc) is going to be the cheapest, but with a much lower bandwidth and "dumber" nodes

So, which is more important, cost or performance?

Even more important is actual ability to install wired system. Many houses and flats are not compatible with easy/cheap retrofit of extra wiring - no matter it is Ethernet, RS485 or not. Instead of reinventing technology of last century, you may consider wireless (wifi/lora/zigbee - pick your poison) . There are many examples, first search hit as one: https://www.aqara.com/us/products.html

[Unixon]

Even more important is actual ability to install wired system.
Many houses and flats are not compatible with easy/cheap retrofit of extra wiring - no matter it is Ethernet, RS485 or not.

That's a completely different story, they won't be able to use any good stuff anyway, only consumer level wireless or locally wired gadgets. Forget it.

Wireless systems are less secure, they both produce noise and susceptible to jamming and interference.
And most disappointing thing about wireless systems when it comes to automation: no power distribution.

However, wireless bridges may be considered useful parts of the communication system.

[ogden]

Wireless systems are less secure, they both produce noise and susceptible to jamming and interference.

Security through obscurity or physical access is way less secure than encrypted wireless. Anyway seems you need to update your knoledge about encrypted wireless and how secure it actually is. If you think that somebody is going to jam your wireless home automation system then you would want to reconsider automation in your home as such.

And most disappointing thing about wireless systems when it comes to automation: no power distribution.

Nonsense argument. What you need power distribution for? In case of sensor - use battery. Otherwise you are getting power anyway.

[Unixon]

Security through obscurity or physical access is way less secure than encrypted wireless.

I agree about obscurity part, but disagree about physical access.
You can't challenge a system you have no knowledge of its mere existence and cannot discover it.
Wired channels can be as well encryption protected as wireless channels.

If you think that somebody is going to jam your wireless home automation system then you would want to reconsider automation in your home as such.

Too many wireless modules on small area in the same band can perfectly interfere with each other operation thus disrupting a network.
 

And most disappointing thing about wireless systems when it comes to automation: no power distribution.

Nonsense argument. What you need power distribution for? In case of sensor - use battery. Otherwise you are getting power anyway.

Works for single gadget, doesn't work for many.
The logic behind this is following: if you have an electronic device it requires power => if you provide power through wires, you have wires => and if you have wires, you have place to run them to and from that device => if you have a place to run wires, you can also put there data lines.

[ogden]

You can't challenge a system you have no knowledge of its mere existence and cannot discover it.

Sorry to disappoint you - wired communication systems can be discovered as easily as wireless unless insanely shielded.

Wired channels can be as well encryption protected as wireless channels.

So what's the advantage of wired systems then?

Too many wireless modules on small area in the same band can perfectly interfere with each other operation thus disrupting a network.

Utter BS. You really shall update your knowledge of communication systems.  Start with keywords: CSMA/CA, CSMA/CD.

The logic behind this is following: if you have an electronic device it requires power => if you provide power through wires, you have wires => and if you have wires, you have place to run them to and from that device => if you have a place to run wires, you can also put there data lines.

When you have mains power somewhere it does not mean that you automagically can have Ethernet or other kind of wires there, kid.