What is wrong with my home automation project?

[wizard69]

First let me say that I don't think much of the home automation in the way it is often presented.    In fact I see it as a massive waste of money.    So I hope the comments below don't seem to be too harsh.

First; the complexity of these systems is mind boggling and frankly your first post sorta highlights this.   I mean I have no idea what you have going on with this system and I wasn't too far into the posts that I drifted off.    I just don't see the point in fiddling with such complexity in a system, I get that every day at work.

Second; The distributed system you describe has me asking why?   This seems to be the opposite of the KISS principle.   A master controller and some slaves are far easier to understand.

Third; If you are thinking a bit about ladder logic and PLC's you will have a good idea what might fly as a simplified solution for the average person.

[EtaPhi]

First; the complexity of these systems is mind boggling and frankly your first post sorta highlights this.   I mean I have no idea what you have going on with this system and I wasn't too far into the posts that I drifted off.    I just don't see the point in fiddling with such complexity in a system, I get that every day at work.

This is another my presentation fault.
If you find uAntColony model mind boggling, academic world laught at it because it's obvious, it's a toy for them.
Have you ever studied Kalman filter equations? https://en.wikipedia.org/wiki/Kalman_filter
That's academia want!

I wrote that model because it's a proof that uAntColony concept works.
So, even if you didn't build a prototype, that model grants that all is ok.

Second; The distributed system you describe has me asking why?   This seems to be the opposite of the KISS principle.   A master controller and some slaves are far easier to understand.

Sorry, a uAntColony is a simple and efficient home automation system (IMHO).
You may have not understood (and that's my fault again) that uAntHill is a master and uAnts are slaves.
There is only one active uAntHill which controls bus trafic, i.e. is a bus master.
uAnts can send data only as a response to a uAntHill request, i.e. they are bus slaves.
There are also some fail-safe features, such as when primary uAntHill fails, a secondary uAntHill may take over it.
I did not write about these details to shorten presentation since I think that it's better to have a whole view of a system that to get lost in its details.

Third; If you are thinking a bit about ladder logic and PLC's you will have a good idea what might fly as a simplified solution for the average person.

I think that PLC and ladder programming are things too complex for an average housewife.
Home automation is hard to sell to them.
Today remote are sold as home automation parts.
A smart socket that you can switch on or off by voice commands or by your smartphone is only an expensive remote controlled socket IMHO.
When things become a little complex, such as a light can be switched on only if it's dark, or an air conditioner can work at full power only if PV system produces enough energy, those smart remotes seem not so smart...

[phil from seattle]

It's my fault if I wasn't clear from the start, but to my excuse I find hard to explain a system which is like a distributed and parallel PLC by using words that every housewife can understand.

That is answer to your question..

+1. That's why we need technical writers, managers, so on.

I'll take it one step farther - and a leader that sees the best end result for the customer (user/buyer/builder/whatever). Apple didn't get successful by exposing the inner workings to the end user. 

There was a great Honda ad from 20+ years ago. It showed a car with half the surface skin removed and showing the incredible engineering detail that went into the car.  The tag line was "We make it simple".  That message really stuck with me.  If you can't make it simple for your target customer to use, don't bother.

[Renate]

What we're all struggling with, is how is your concept different than a bog-standard SCADA system?

[EtaPhi]

What we're all struggling with, is how is your concept different than a bog-standard SCADA system?

Simply put, it's like a SCADA who it's in control of all...

A SCADA usually sends state enquire commands, while low level automation runs.
PLCs answer to those state enquire commands while they execute low-level automation loop, i.e. in each cycle they acquire inputs, compute new outputs and actuate their new values.
All works well if there are no remote inputs or if a fieldbus is reserved to PLCs, otherwise state enquire commands may interfere with normal PLCs operation by subtracting communication bandwith.

A house is different from a factory. For instance it's even difficult to install a bus and reserved cabling is almost infeasible. When there is a bus, it's shared between all devices which must be small to fit in junction boxes or behind switches plates. These devices must be cheap too and have to live with existing devices because home automation does not happen immediately, but usually it takes time to replace old "dumb" devices.

State based automation allows to add devices without changes to previous ones.

Command based automation is less flexible, as the device that sends a command must know what commands accepts the device which will execute them.
For this reason, when a new device is added, all devices that send commands to it must be modified.

uAnts, on the contrary, have not to be modified, because their state doesn't change when a new uAnt is added.
Only the output functions of the new uAnt have to be defined, which is reasonable.

uAnts fail gracefully, because when a uAnt stops working, its state doesn't change anymore.
uAnts that depend from its state will keep working, perhaps their performance will degrade, but nothing stops working as happen when a PLC fails.

uAnts Protocol is efficient and makes it possible to trasmit as few bytes as possible.
For instance, SCADA state enquire and setup commands aren't needed, because all communication is a state enquire (this is true 99% of time, as there may be state write sequences too).

[james_s]

I still don't understand what any of this does for me. I add and remove devices to my automation system all the time without messing with existing devices. Commands vs state is irrelevant to me, the automation system knows the state of all of the devices, it changes the state by sending commands and monitors the state by receiving commands. How will some new system provide a tangible improvement over what is already available and in widespread use?

[Bassman59]

It's my fault if I wasn't clear from the start, but to my excuse I find hard to explain a system which is like a distributed and parallel PLC by using words that every housewife can understand.

That is answer to your question..

+1. That's why we need technical writers, managers, so on.

I'll take it one step farther - and a leader that sees the best end result for the customer (user/buyer/builder/whatever). Apple didn't get successful by exposing the inner workings to the end user. 

There was a great Honda ad from 20+ years ago. It showed a car with half the surface skin removed and showing the incredible engineering detail that went into the car.  The tag line was "We make it simple".  That message really stuck with me.  If you can't make it simple for your target customer to use, don't bother.

I think I see the disconnect here.

What Phil from the PNW is saying is true -- it has to be simple for the end user. (The user is canonical housewife mentioned earlier.)

But if you've been around stuff long enough, you realize that making things simple for the end user can require really complex engineering under the skin, like in the Honda ad.

Think about USB. The user is trained to just plug it in, maybe install a driver, and then it Just Works. If you've done any USB device development, you know that it's quite complicated.

EtaPhi describes the "under the skin" implementation. It needs a skin to hide the complexity. Users don't care about the implementation details. They care that it works and is easy to configure. Get a software developer with user-experience experience to come up with good interface and it could take off.

[james_s]

Sounds reasonable. I don't really care what is going on under the skin as long as it works. It is far more important to me that it works with a variety of off the shelf hardware, I'm not going to build my own lamp interface when I can buy a Sonoff or other commodity smart socket, pair it to my system and go. We are long past the era of people building all of their own automation hardware, the market is flooded with good and inexpensive hardware, usually where it is lacking is in the "glue" that holds the system together and that is where the open source systems really shine. I have a mix of wifi, zigbee and RF devices and everything just works, it's almost seamless.

[EtaPhi]

I still don't understand what any of this does for me. I add and remove devices to my automation system all the time without messing with existing devices. Commands vs state is irrelevant to me, the automation system knows the state of all of the devices, it changes the state by sending commands and monitors the state by receiving commands. How will some new system provide a tangible improvement over what is already available and in widespread use?

There is a small improvement from your point of view.
You pay for having a supervisor system (even if it's free and open source) because you have to run this software h24 24/7 on hardware that draws energy.
The yearly increase of electric energy bill is small, but there is.

uAntHill is like your supervisor system.
Your modem router may run its software because it needs only a UART to send and receive bytes.
You obviously have to modify your modem router firmware, so it's not a task for a housewife until, some day (I hope) modem routers have it built-in.

You may not agree with this example.
I therefore tell you a thing that has been feasible since twelve years ago and that your system could not do: lamp configuration without any computer or smartphone or software.

As I said regarding address assignment procedure, uAnts have got a button.
After a uAnt has got its address, this button may be used to configure outputs activation.

If you keep pressed this button long enough, uAnt enters in "output configuration" mode.
You select the output to configure by turning the potentiometer which is usually used to lock uAnt configuration.
Once an output is selected, you walk to the uAnt whose state must control that output and press shortly the button to put the latter uAnt in "stimulation mode".
When a uAnt is in stimulation mode, a digital input change toggles pairing mode, i.e. a bit in uAnt state say "I control you" or "I don't control you".
After a flashing led confirms that stimulation has been broadcasted, the "programming" button may be pushed again to resume normal operation.
If there are more controlling uAnts, for instance when a lamp must be switched on/off by two or more points, you repeat the above simple steps.
Finally you walk to the uAnt in "output configuration" mode and push the "programming" button.
From then on, when you push one of the controlling buttons, the lamp is switched on or off.

"Under the skin", this is simply obtained by how state byte 0 is defined.
This byte always encode 4 digital states as pairs of bit.
Pair 00 means "I don't control you"
Pair 01 means "I'm pushed"
Pair 10 means "I'm not pushed"
Pair 11 means "I control you"
When a uAnt is in normal mode, it discards digital pairs 00 and 11, so the other uAnts which use that state bit are unaffected when a uAnt is in "stimulation mode".
On the contrary, when a uAnt is in "output configuration" mode discards digital pairs 01 and 10. Since uAntHill says which address is now speaking, the uAnt in "output configuration" mode can manage the list of its controlling states.

[james_s]

There is a small improvement from your point of view.
You pay for having a supervisor system (even if it's free and open source) because you have to run this software h24 24/7 on hardware that draws energy.
The yearly increase of electric energy bill is small, but there is.

My supervisor system is a Raspberry Pi that cost me under $60 with the SD card and power supply and consumes around 4 watts. This cost is negligible and is far more than offset by having a central supervisor that allows me to manage and configure the entire system all in one place. Having a supervisor system is not a bug, it's a feature, it's not a disadvantage, it's an advantage. It makes my life easier and saves me time, for a cost that is all but irrelevant. There is huge value in a central device that monitors and configures everything, logs data and serves as a gateway to the outside world. Sorry I'm just not seeing any advantage to some other architecture.

[Fredderic]

I actually went down a similar rabbit hole about 20 years ago…  Designing a self-hosting distributed system not unlike this ant hill thing.  In mine, a HUB was used to connect the devices network to any outside network you wanted (such as your PC), and to update the rules, including when the network topology was changed (was probably going to use good ol' RS-485, think I was also looking into CAN). 

The HUB would compile the rules, and then use the current network map to build a set of instructions for each node telling it what devices needed to know about it's states, and the network paths to send the update packets there (basically a binary sequence of port numbers…  most devices had an in and out port, larger devices might have three or four ports, facilitating a star network topology, cycles allowed)…  This allowed the nodes to be extremely simple, and handle message routing without having to actually know the topology, deal with discovery, and all the rest — the HUB would receive a message about a new or missing node, query it for info, and as needed, update the end nodes with new instructions.

So long as you didn't need external connections, weren't moving nodes or network links around (including fall-back through alternative routes), you could remove the HUB entirely and it'd all just continue to work on it's own — notably, since the nodes spent most of their time in their deep sleep states and could readily fall back to batteries, I had planned to include a fall-back route support which only bothered with the battery-backed nodes to, for example, keep door locks functional as long as possible in the event of a power outage.  The HUB would also monitor traffic, though, and adapt the network accordingly when alternate routes were available.  I was kinda interested in message routing algorithms at the time, and it meant you could just throw as many links in as you wanted between nodes with sufficient spare ports, and let it sort them out, and maybe even suggest improvements.

I had it mostly working and was just about ready to start moving it beyond bread-boarded prototypes, when I noticed all that consumer stuff was appearing, and decided we didn't need yet another standard, and no one really seemed to want wired any more, anyhow…

One of many projects that in hindsight I wish I hadn't scrapped just because someone else was doing something similar too…

BUT, these days, efficiency of messaging just isn't an issue any more.  MQTT carrying JSON packets still gives my 8-bit brain the shudders, but, my current mixed WiFi and Ethernet Home Assistant based automation system manages just fine — and honestly, the UI isn't THAT bad…!  Plus there's whole-screen cards these days that let you lay out whatever you want, however you want, and even mix-and match to a degree, pretty readily.  I generally have a screen laid out for PC's (3 or 4 columns), a screen laid out for mobiles (single-column, I admit the layout algorithm is a bit naff, but, I hear they're working on it, and I generally like what I hear), and a screen where I let it just toss everything in together however it likes (catches those brand new things I've just plugged in and haven't gotten around to placing into the main views yet, or internal stuff that I really didn't think I'd even want to have exposed), and it really does work beautifully.

I completely agree it's the way to go these days, and the vast variety of gear a big core system like HA can connect to is just amazing — by far way beyond anything I'd ever hoped to achieve.  And in the few years I've been running it, it's only let me down once locally.  Biggest issues are with the cloud connection which has hiccups on occasion…  But there again, I never could have done the whole system, support for all the components I've got tied into it, and a pretty decent mobile app on top.

Do I still wish I'd built out my system?  Sure.  I think it'd still be slightly better for controlling the lights and things like that over its wired connection.  But is it worth doing it at this point?  Not really, no.  The appropriate compliance marks on the switch nodes I use instead aren't nearly as likely to make the insurance company run screaming if there's ever a problem.  And quite frankly, I've had more failures due to bulbs dying, than I have with Home Assistant failing in any way — it's performed admirably, survived countless updates, and runs just fine on its R.Pi without so much as a heatsink (though, I probably should actually stick on the heatsinks I have for it…  some day…).

[wizard69]

Third; If you are thinking a bit about ladder logic and PLC's you will have a good idea what might fly as a simplified solution for the average person.

I think that PLC and ladder programming are things too complex for an average housewife.

The idea behind the PLC statement isn't to replicate a PLC but rather the concept or abstraction of an industrial control system.    What you would need is something that abstracts you home control system into and easily operated graphical interface

Home automation is hard to sell to them.

It is hard to sell to anybody.    I mean seriously what is the motivation here, I consider myself to be a normal male with a deep involvement in technology and I just don't see a benefit.    The set back thermostat that I installed years ago and tore out last year, was more of a pain than it was worth.   There was a huge challenge in keeping the batteries from leaking and frankly it ended up emulating a mechanical thermostat with a bunch of useless technology in the box.

Today remote are sold as home automation parts.

Sold is not the same thing as grabbing a huge market.

A smart socket that you can switch on or off by voice commands or by your smartphone is only an expensive remote controlled socket IMHO.
When things become a little complex, such as a light can be switched on only if it's dark, or an air conditioner can work at full power only if PV system produces enough energy, those smart remotes seem not so smart...

A PV integrated air conditioner might be a good thing but we still don't need a whole home automation system to accomplish it.

Maybe I'm biased but I'm nearing retirement and frankly I'm thinking about moving.   I don't need a huge home anymore and as such I'd likely move into a smaller home, with a open floor plan (as much as possible anyways).   So I really don't see the point in home automation as long as I'm healthy enough to adjust the temperature when I'm cold.   If I was to build such a home tomorrow I just don't see home automation as being very high at all on the list of must haves.   Solar power would be much higher on the list as would be a workshop to fiddle in.  In fact I probably could come up with a very long list of must haves before any sort of home automation lands on the list.

Now I realize that not everybody feels this way but on the other hand I don't see a huge segment of the population running out to buy home automation hardware.   There have been shots taken at the home automation concept for decades now and I think what these developers don't get is the WHY?

[james_s]

It is hard to sell to anybody.    I mean seriously what is the motivation here, I consider myself to be a normal male with a deep involvement in technology and I just don't see a benefit.    The set back thermostat that I installed years ago and tore out last year, was more of a pain than it was worth.   There was a huge challenge in keeping the batteries from leaking and frankly it ended up emulating a mechanical thermostat with a bunch of useless technology in the box.

For me it's just a hobby, I guess I inherited my grandfather and my father's love of gadgets. My grandpa especially loved that stuff, he had every kitchen gizmo imaginable, electric carving knives, the first immersion blender I ever saw, soda maker, ice cream maker, sandwich maker, burger cooker, one of the very first Amana RadarRange microwave ovens, the first color TV my mother ever saw, I'm sure you get the point. Home automation wasn't much of a thing back then but he would have been all over it if it was.

I do like being able to control lights from my phone or a wireless remote, my partner has a tendency to leave lights on and now I can turn them off without getting up from the sofa. I've also retired all of my electromechanical lamp timers and I use Home Assistant to schedule all that stuff. When I go on vacation I can keep the place looking occupied by setting lights to turn on and off at various times or just pull out my phone once in a while to switch them manually from wherever I am. One of the more useful bits of automation I set up is using the motion detection of my front porch IP camera to turn on the pathway lights if it detects motion while it's dark out and this works independently from the timed schedule that turns them on from sunset until 11pm. I put water leak sensors under all of the sinks and that has saved me from a big mess once so far when the dishwasher vent clogged and started dumping water into the cabinet under the sink, something that had happened in the past and caused some damage. I wired modified door/window sensors to my washer and dryer to send a notification to my phone when either of them finishes. None of this stuff is anything I *need*, but cheap off the shelf hackable hardware has meant I have not had to invest very much money and I've been having fun playing with it.

It's also worth mentioning that the Tasmota firmware I use on most of my esp8266 based devices is capable of working standalone without a host supervisor. It has built in timers, control through a web page hosted by the device itself and they can send and receive mqtt messages so it's possible to set up a standalone peer to peer system, useful for very small installations where maybe you just want to control some lights with motion sensors and timers or something. Overall it isn't something I'd recommend to non-technical folks though, if I didn't enjoy tinkering I would not have bothered.

[EtaPhi]

I thank all who posted.

Their replies let me realize that there is nothing wrong with the technology of my home automation project.
People however don't care of technology as long as their needs are satisfied.

Home automation concept is different from people to people.
Automation experts see it alike other automation systems they deal with daily, i.e. a set of PLC and SCADA connected by some field bus.
Technophiles see it as something they can control with their smartphones or voice.
I see no intelligence in the latter view, as that home automation has not to be smart, but execute what it's orderer them to do.
Even thermostats, which should be a bit autonomous, are a pain to manage.

It doesn't matter if a 32 bit or 64 bit computer does what a humble 8 bit microcontroller did more than a decade ago for a tenth of the cost using only 8192 bytes of code and 384 bytes of RAM, as these computer are cheap.

I don't want to sell anything, as I wrote in the preliminary remarks.
I wanted to share an idea which influenced my following projects, i.e. a photovoltaic system supervisor and anti theft system, a drone control system where aviation safety standards are followed (but certification has to be started, because it's still an unfinished hobby project).
This latter project uses the state propagation concept too.
There are however two CAN buses as one may fail and there are safety measures which halt a misworking unit if it sends CAN messages when it's not allowed to speak.

Different requirements, different solutions.

I wondered if my view of automation was biased, if uAntColony and state based automation was a dead end experience.
It wasn't, as what really counts are results, not how you achieve them.

Thanks again to all