Creating a communication protocol between devices

[Unixon]

Sorry to disappoint you - wired communication systems can be discovered as easily as wireless unless insanely shielded.

This kind of discovery requires much closer access than systems that broadcast themselves over the air to entire neighborhood.
It doesn't even matter if a particular communication system is wired or not, technically its the question of detection thresholds.
The point is - in order to go searching for a communication system one has to have at least an idea that something is there.
And until you raise what you search for from noise floor by any method you can't tell if it exists or not.

So what's the advantage of wired systems then?

They are simple and straightforward if you're not overcomplicating them intentionally.

Start with keywords: CSMA/CA, CSMA/CD.

Oh, I know. Just keep increasing number of agents until it gets too overcrowded and fails or agents start indicating timeout while waiting for a free slot.

When you have mains power somewhere it does not mean that you automagically can have Ethernet or other kind of wires there, kid.

This depends on how the wiring is done and what are the rules for electrical systems.
In older places you can't do anything about wiring if it is mounted in concrete walls behind some decoration on top of that,
in new like office places all wiring is in channels with outlets and technical ceilings/floors and you can have almost anything there.

[Red Squirrel]

I do want to stick to wired, but adding a wireless bridge option as a type of "sensor" would be something I would eventually want too.  Probably do zigbee for that part or whatever else comes up in my search at that point.  But the main advantage of wired is power distribution and overall simplicity, as the goal of the system is to be very low maintenance and simple.  Power would come from a central source that is itself battery backed up. Probably a 12v battery in a distribution cabinet that would power the whole thing.  Kind of like a fire panel.  Heck this system will be so modular it could technically be used as a fire system (maybe not officially as it would not have the proper certification etc for that purpose).  Security is not a HUGE consideration with this, but I may very well eventually look into what it would take to encrypt data and have other measures in place to prevent tampering, but that's more of an aside for now.    If I do add security then this system would be suitable for home security type sensors as well.

[Unixon]

Security through obscurity or physical access is way less secure than encrypted wireless.

Let me add few more things to the argument.
I'm not saying that a wired communication system has overall better security than a properly encrypted wireless system just because it is not wireless.
A system security is eventually measured by amount of obstacles for an adversary to get in and affect its operation and it is not right thing to dismiss inability to discover and connect to a wired system as valuable obstacles.
We use encryption as the ultimate measure because it allows us to create huge obstacles with little effort.
What I'm saying is adding protection by inaccessibility on top of encryption should not be neglected and that for communication systems that do not require high security the very inability to discover and/or access a system may serve as sufficient obstacle by itself.

[nctnico]

Security through obscurity or physical access is way less secure than encrypted wireless.

Let me add few more things to the argument.
I'm not saying that a wired communication system has overall better security than a properly encrypted wireless system just because it is not wireless.
A system security is eventually measured by amount of obstacles for an adversary to get in and affect its operation and it is not right thing to dismiss inability to discover and connect to a wired system as valuable obstacles.
We use encryption as the ultimate measure because it allows us to create huge obstacles with little effort.

But encryption isn't enough. Good security also needs a way to authenticate devices and a way to detect intrusions.

But coming back to a protocol: there are 2 things to seperate:
1) the physical layer
As a physical layer RS485 is good for long distances. You can use a UART or bit-banged Manchester code if you like. CAN bus is also be a good candidate because it also includes error detection and automatic re-transmission for un-acknowledged packets.

2) the protocol itself
In my experience readable text based protocols are by far the easiest. Look at why SCPI is so popular for test equipment; it is easy to work with, easy to prototype and easy to debug. The physical layer underneath it doesn't matter much.

[max_torque]

I'ms till going to suggest that power distribution on your network is actually more critical than signal distribution!

If you used say RS-485, then you could do cheap isolation using digital isolators to carry the TX, TX-EN and RX signals to standard (cehap) RS-485 transcievers.  The amount of power each module requires is fairly small, so could be sent over the network from a master unit. The devices then are powered by local sources, or a dedicated "high power" distribution network, so seperating signals and power.

One thing you need to consider with RS-485 / CAN is the fact that the bus needs to be linear and not in a star format, and how you can arrange the bus terminators.  This is usually done by having each module daisy chain the bus through itself, ie there is an IN and an OUT plug, and the last modulei the linear chain has a terminator plugged into it's "OUT" socket.

A basic protocol and physical layer allows for very simple and cheap slave devices.  It also allows for very simple software too, which should not be under estimated. A basic 8bit processor with a std uart could be used for the simple tasks, and a more complex, expensive ARM for the complex stuff.  ie keep simple, simple!


 if you are not going to send a clock signal as a physical signal, then your slaves will need suitably controlled clocking, or you will have to write some sort of power on "sync" functionality (not that hard) to recover the clock from say an initialisation string of data of known format sent by the bus master after it powers the bus up.

[Unixon]

Small. but important notice. Although RS485/422 is a bus, I would turn it into a set of P2P links just like Ethernet.

[Red Squirrel]

I kind of like the linear in/out approach where it's just a long string of nodes.  It could be done in a very simple and cheap way.    The way I'm thinking it could work is something like this:

PC interface card - node A - node B - node C - etc...  (technically as many nodes as I want)  Have not decided how a terminator would work yet, I'm thinking it can loop back and plug right into the PC interface card. When the card gets a signal from there it knows it's at the end.

The PC interface card would be very similar to a node, except it would have a USB uart and the PC talks to it via serial that way, and it talks to the first node.   The nodes would then have their turn to talk to it's associated sensor that plugs in to the node board, then move on to next node.  The PC interface card starts by generating a clock signal to the first node, indicating to the first node that it's time to talk.  It would talk over the bus wire which would be common throughout the whole string.  Once it's done talking it would turn off the clock, and start generating it's own clock towards the second node then that node would talk and so on...  When it gets to the end it then just repeats the process.

The actual sensor devices would then plug into a node, and have their own communication that independent of the main node loop.   The node would keep a list of values in memory and it would simply update them with the sensor as needed.  The nice thing about this is sensors that are slow to read/write would not hold up the whole bus, as the node would always be ready to send or receive values.   For example if I have something like a linear actuator it takes some time for it to mechanically move and then read to confirm it's at the right spot.  This would happen independent of the bus.

From the PC's point of view the commands would be something like this:

">" indicates PC talking,
"<" indicates PC interface board talking to PC

> get name, or some other form of start command
< send name of sensor (this can be just a number such as the order in the string, or there could be ways to program that in)
> get values
< show list of values
> write values (probably one command per variable)
< Get an ok or other form of acknoledgement
(repeat any other commands as needed)

> Next node   - This would simply move on to the next node in line and then the above commands can be repeated

The values would probably just be written into a common location like a text file on a ram drive or something.

A separate piece of software acting as an agent would then read/write to this and talk with the server.  There could be several instances of these agents spread over a large network all talking to the same network and having multiple strings of sensors.

The actual server software would then handle all the logic of what to do with these values.  It's then up to the admin to program them to do what they want. 

At least that's the jist of what I have in mind anyway...  the specifics I will of course figure out as I actually start. 

But yeah I'm actually liking this idea of a linear approach.  Probably still do 4 wire, ground, power, clock, communication. (half duplex over same wire)   Physically there are different approaches too, as I can have multiple nodes on same PCB and have it stay in the cabinet and the sensors then connect and go out to where they need to.  Or I can have the nodes separate and just string throughout the building.   If I use 4 wires it also allows to use cat6 and I can have it return back on the other 4 wires as needed to go back to cabinet.   Could also use a hybrid approach where I have node boards with multiple nodes in each room/area that requires sensors. So you have cat6 going to multiple general areas of the building, then you can plug the sensors into it's respective board that would be in it's own small cabinet.  You can of course just daisy chain as many of these boards as you want.  They could come in different variants like single, or 4 or 8 etc... port.

For power I'm thinking I could go with a high enough voltage like 24v or 48v.  Having a higher voltage available would ensure having enough power for sensors that require a certain amount of power to operate (ex: some types of sensors like hydrogen sensors etc actually have a small heating element)  The node board would do all the required power conversion stuff as well as isolation.   Here I would be tempted to do something like have +5v but also keep +48v, the sensor would get both.  That way sensors that may need a more specific voltage can have their own conversion stage and run off the 48v part.   

I'm thinking the nodes themselves would not be electrically isolated, but the sensors would be individually isolated from the node, thus, the entire bus.   The PC interface board could also provide any input protection in case anything happens to the bus like high voltage accidentally applied to it.   Or static electricity etc.

At least that's one approach I am thinking.

[max_torque]

I my experience it's easier to come up with a method of isolating the bus and power supply, meaning all modules of whatever type can use the same architecuture. And isolation at the bus level is most likely to be both a small number of signals (3 for RS4-85 half duplex) and they will be fully digital by this point, no matter what your sensor does.  You slave logic then sits on the sensor side of that isolation, giving it free access to those sensors!


The biggets descision will be how to arbitrate the bus and avoid collisions, especially with a half duplex bus.  The simplest way is to only allow the master to initate coms, and all slaves only reply to that master when called on.  No slave module can simple broadcast on the bus.

A second approach is for the master to broadcast bus sync markers, ie certain characters to which the modules can sync, and hence wait to send out their data until "their" slot has arrived.  That bus sync string sent by the master can also be used for auto baud rate determination and for baudrate clock syncing.  That would probably allow you to run with slaves that use basic, low accuracy (and high drift, esp. tempco) microcontrollers without external xtals, saving costs and board real estate for the simplest slaves

[Red Squirrel]

Yeah if I do the isolation at the node I would want to make sure that the sensor side does not really need to do anything special having to do with the isolation part, try to put as much of the work as possible on the node side, as once it's designed they are all the same and can be mass produced the same way.  The hardest part will be the transformer as that is a bit harder to automate, but if I actually got to a point where I was making 100's or even 1,000's of these I would figure that out at that point.   Or I can have two node versions, an isolated and non isolated one, as the isolation is only really needed for sensors that may connect to other voltage sources.  (voltage sensors, or devices that will have their own external power)

For arbitration, I realized that my idea of having the clock signal act as a token that goes along won't exactly work... the signal still needs to be generated by the master as it needs to be able to talk to the right node.  Just an oversight brain fart while I was brainstorming there.  So yeah the clock signal and bus need to be common and active the entire length.  So what I need to do is as part of the "start" command to talk to a sensor, I need to have the master (that would be the PC interface board) send out an address it wants to talk to.  Each node listens for the address when it sees a start command and if it matches it's own, then knows it's it's turn to talk and will issue a command to acknowledge it.   I then just need to come up with a way to set the address and make sure it is unique.    I have a couple ideas in mind for that, either it happens manually at time of install, or it can be part of a startup sequence when a brand new node is introduced into the network (ex: it could have some kind of initialization address).  given each node would have a MCU I have lot of options for jumpers/buttons etc if I want to have some processes be initiated manually too, like a reset button.

But yeah I've pretty much just been using this to brainstorm but thanks for all the things that have been brought up as it's all going to be part of the consideration as I start to actually experiment with this. 

I've never actually done anything to do with MCUs that involves having multiple talk to each other so I will want to experiment with that first at a super low level and then work from there.  I'm sure I'll run into stuff that I forgot to consider as well, such as error correction, or how to detect a cut cable in middle of communication, etc.

[ogden]

Small. but important notice. Although RS485/422 is a bus, I would turn it into a set of P2P links just like Ethernet.

I would not. - Because then adding new sensor means running new wire to the control hub. If system has to be DIY, easy plug and play then I would suggest to use ESP8266 as WiFi<->RS485(or PLC) gateways/end-nodes. I would use DC power line communication (PLC). System can have as many WiFi gateways as needed. Most likely my home controller would be rPI configured as wifi access point that serves whole home automation WiFI network - gateways, wifi sensors if any, user connections. Yes, susch system could be subject of WiFi DOS attacks any script-kiddie can do.

[Unixon]

Small. but important notice. Although RS485/422 is a bus, I would turn it into a set of P2P links just like Ethernet.

I would not. - Because then adding new sensor means running new wire to the control hub.

Yes, it is. The logic behind that is simple: an end device and its bus connection may be physically hard to reach and if that device malfunctions due to hardware or software problems this can halt all bus operation and you won't know which device did this until you disconnect it and reconnect this bus segment to a bypass. With P2P links this is just one unresponsive link and more importantly you know exactly what link/device that is.

I would use DC power line communication (PLC).

Isn't it expensive? I don't know, just asking. Mains AC modems are expensive. What parts would you suggest for this kind of circuit?

[ogden]

an end device and its bus connection may be physically hard to reach and if that device malfunctions due to hardware or software problems this can halt all bus operation

Yes indeed there are such risks. Thou what are the chances? I am not saying that one shall put whole building on single bus - I said "system can have as many WiFi gateways as needed". If you expect hardware or software problems in your DIY building automation, then I am not sure you shall be doing such project

I would use DC power line communication (PLC).

Isn't it expensive? I don't know, just asking. Mains AC modems are expensive. What parts would you suggest for this kind of circuit?

Sorry, I did mean "PLC" in a broad sense, not AC mains only. What I had in mind is simple enough - DC bus with "voltage dip pulse" communication, like KNX TP1:

[Unixon]

If you expect hardware or software problems in your DIY building automation, then I am not sure you shall be doing such project

This is a whole source of problems right there if you're building a system without thinking about things that may happen to it.
No matter how good you are at engineering, no matter how high the quality of components and overall build quality is, you must suspect that some shit may happen.
Whatever happens, a failed part of system must go down gracefully without causing any harm and with minimal effect on the rest of devices.

[ogden]

This is a whole source of problems right there if you're building a system without thinking about things that may happen to it.

If you are building system without thinking then choice of best network topology is least of your problems. There are many ways for single point of failure to bring down whole star network as well. Also when you actually think while designing common bus devices - signalling mode & interface circuit can be designed in such a way that failed device cannot bring whole segment down. What remains - disconnect or short. So it depends.

Let's look at this from different angle. There are KNX TP1 bus, Dali bus, Lonworks TP bus to name few. Do you really think that industry experts did not know what they are doing and you know better?

[Unixon]

Do you really think that industry experts did not know what they are doing and you know better?

They were solving different problem with different boundary conditions and came to bus topology as an optimal solution in their cases.

[ogden]

Do you really think that industry experts did not know what they are doing and you know better?

They were solving different problem with different boundary conditions and came to bus topology as an optimal solution in their cases.

Good. There are no such thing as "their cases". They were building as universal systems as they can, that supports all network topologies one may need - linear, ring, tree, mesh, star and ... (surprise!) P2P. Proposed by you P2P approach is limited to only one topology, P2P. That BTW requires quite complex hub because every node needs individual communication port. You perhaps want star in your house, but others will be solving different problems with different boundary conditions, will need tree or just linear bus.

[Unixon]

There are no such thing as "their cases". They were building as universal systems as they can, ...

No, I have to disagree. We all have our opinions on how things should work and what requirements they have to meet.
Red Squirrel started this thread with his ideas, I dropped in few of my ideas, and so on... and none of these has to coincide with what KNX, Dali and Lonworks had on their minds while developing their products.

You perhaps want star in your house, but others will be solving different problems with different boundary conditions, will need tree or just linear bus.

In fact, a network structure almost repeats physical location of modules, so it may have different topologies in different segments if physical interface allows that.

To put it simply, I would like to see a simplified version of 100BASE-T Ethernet with cheap physical layer components, smaller footprint, easy way to make many port solution.
For example, some of STM32F09x have 8 hardware UART interfaces in one package, 8-port RS485 hub can be compact and simple and do other functions besides communication.

[ogden]

There are no such thing as "their cases". They were building as universal systems as they can, ...

No, I have to disagree. We all have our opinions on how things should work and what requirements they have to meet.

Instead of having unfounded opinion you could actually read standards to see that requirements and targets are provided.

Red Squirrel started this thread with his ideas, I dropped in few of my ideas, and so on... and none of these has to coincide with what KNX, Dali and Lonworks had on their minds while developing their products.

Red Squirrel can take any idea including mine or yours. Just please do not cry that I have arguments against your idea. [edit] Actually Red Squirrel's targets: "super plug and play", "super expandable" do coincide with all mentioned standards. Your P2P idea - not that much.

In fact, a network structure almost repeats physical location of modules, so it may have different topologies in different segments if physical interface allows that.

In your "P2P" case every branch needs active multiport router. It increases cost and complexity, also not "very plug and play".

[Unixon]

Instead of having unfounded opinion you could actually read standards to see that requirements and targets are provided.

Oh, I will read them, but I highly doubt any of them fits perfectly.

Actually Red Squirrel's targets: "super plug and play", "super expandable" do coincide with all mentioned standards. Your P2P idea - not that much.

I don't know what was initially meant by this description of features, but that's how I see it:
- Any device must be uniquely identifiable with hardware ID, typically 32, 48 or 64bit, could be more, whatever provided;
- Any device must be addressable by its hardware ID using OSI Level 2 protocol;
- Any device must discoverable by any other device using OSI Level 2 protocol;
- Any device with 2 and more ports can be a hub [this may be mandatory];
- Hubs are responsible for automatically discovering and updating network topology and routing data between any discovered UIDs on the network;
- Any end device must be able to describe itself and provide access to its properties via OSI Level 3 protocol;
- It is highly desired that Level 3 protocol would be human-readable, e.g. XML/JSON-based;
- Any properties of any device must be remotely discoverable and accessible using OSI Level 3 protocol;
- End devices must not require configuration prior to attaching them to the network;

Maybe I missed some things, will update in later discussion.

[ogden]

Actually Red Squirrel's targets: "super plug and play", "super expandable" do coincide with all mentioned standards. Your P2P idea - not that much.

I don't know what was initially meant by this description of features, but that's how I see it:
- Any device must be uniquely identifiable with hardware ID, typically 32, 48 or 64bit, could be more, whatever provided;
- Any device must be addressable by its hardware ID using OSI Level 2 protocol;
- Any device must discoverable by any other device using OSI Level 2 protocol;
- Any device with 2 and more ports can be a hub [this may be mandatory];
- Hubs are responsible for automatically discovering and updating network topology and routing data between any discovered UIDs on the network;
- Any end device must be able to describe itself and provide access to its properties via OSI Level 3 protocol;
- It is highly desired that Level 3 protocol would be human-readable, e.g. XML/JSON-based;
- Any properties of any device must be remotely discoverable and accessible using OSI Level 3 protocol;
- End devices must not require prior configuration before attaching them to a network;

Maybe I missed some things, will update in later discussion.

  You miss absolutely everything. I offered to implement "DC bus with "voltage dip pulse" communication, like KNX TP1" - physical layer for bus network. Our discussion was about physical layer only. Now you suddenly come with OSI Level 2, 3 protocols, XML/JSON nonsense. [edit] In other words: what you describe is way too overcomplicated for "DIY home automation".

[Unixon]

Our discussion was about physical layer only.

None of "super plug and play" and "super expandable" is about physical layer.

[ogden]

Our discussion was about physical layer only.

None of "super plug and play" and "super expandable" is about physical layer.

Yes. I agree. - When we look at "plug and play" from computer engineer point of view. Plug and play in low end hardware not always means "autoconfiguration". Very often stage technicians say that DMX is plug and play because all you need is 1) set address using switches/jumpers 2) plug 3) play Same way in DIY home automation: KISS approach everywhere unless you want to build it forever and never finish.