Lots of I/O management + fast communication protocol

[Rutherberg]

Hi !

I'm on a project consisting of managing at least 128 digital inputs and as much outputs, Arduino based.
The operation is very simple : when an input is HIGH, put the corresponding output to HIGH.
The devices to control with the outputs are between 5 and 20 meters away from the inputs so I would like to use serial protocol to limit cable count, RS-485 seems obvious.

Now the difficulty is that the latency between input and device activation must be less than 5 milliseconds, and in best case less than 1ms.
To manage as much I/O, the only solution I found so far is cascading shift registers but the latency increases with I/O count.

Furthermore, all the inputs are at the same place, but the outputs are scattered in different zones. So I thought of Modbus to address each zone, but I have the feeling it is oversized and too laggy for this application.

Any idea or help will be greatly appreciated ! 

Thanks in advance !

[mikeselectricstuff]

Latency isn't a problem as long as baudrate is high enough. RS485 will easily do 2Mbaud over 100 metres of Cat5
On-board shift regs can be clocked at a few MHz, so no problem there.

[Rerouter]

5 milliseconds is almost an eternity in relation to a 16Mhz base level arduino uno, (roughly 40,000 operations on the avr's)

As a simple way to approach the read if you prefer shift registers, run 8 in parrellel, and have 2 other pins manage latch and shift, if you line up your pins, you can read all 8 pins as a bank in 1 operation and send it however you please, (on an 8 bit AVR)

As for sending, assuming a maximum of 1ms delay, your data rate is 128,000 bits per second, add some overhead for addressing nodes, and your still within the specs of a RS485 link

So your read system would be punching in blocks of 8 reads each shift in, probably addressing it as required and storing it in a frame buffer to send (ideally you would wire up the inputs into logical groupings to prevent having to shift too much around), then when a frame is complete load it to the uart buffer and send him out, in reality if you stick to hardware read and write operations, you have plenty of time to even add checksums and such, but it may be a bit much to do debouncing,

[tggzzz]

Don't forget that isolation and EMI/EMC might be a consideration in your environment.

The simplest solution to that is often to use optical fibres rather than copper wires.

[hans]

You probably will want to keep the software as straight forward as possible. Poll the shift registers as frequently as you can.

A chip like an AVR will not be fast enough to be a problem for regular logic. Tie it up to a SPI port and you could clock in/out bits at 8MHz.

Time to clock in or out: 128 bits at 8MHz = 16us
Then I would check if any bit has changed (something like XOR on each byte of a shift register, then pull out the I/O bits that were changed). Let's say this state checking & saving costs 200us.
Format those into a packet, depending how you put your protocol together maybe 1 packet per I/O. If such each packet is 4-8 bytes, at like 400-500kbaud that takes ~100-200us to send.

So the transmit side should take 400us-500us at most I suppose.

The other end has probably even less to do, but maybe you want to have some extra logic in it when both nodes power up in different orders (so on bootup the output side can request inputs and copy them over).

[max_torque]

What sort of "input" signal are you having to read?


I'd suggest there might be a case for say making 16 channel (or 32 channel) Input and output modules, that sit on a CAN or RS485 bus, and spit out there input state, or listen for those messages and set an output state?

That way, things are scaleable / replaceable etc

[rstofer]

I wouldn't bother to packetize 'changed bits'.  Just read them as fast as you can, send one packet (including checksum?) to the far end and be done with it.
Suppose your RS485 runs at 1 MHz. Sending 128 bits plus another 16 for checksum might take 140 uS.  Personally, I would slow the bus down go 100 kHz since that would only take 1.4 mS.  I might even go slower.  Maybe I would send the packet as ASCII or even something like ROBIN:
http://www.bdmicro.com/code/robin/

Five mS is a very long time.  OTOH, I would still increase the horsepower to something like an mbed.  I would at least want to have 16 bit packets for SPI so that I could hang just 8 IO expanders on the bus.  I would need 8 CS' lines plus the usual 4 SPI signals.  Something like the Microchip MCP23S17 IO expander comes to mind.  I have never tried it but the SPI peripheral can be connected to the DMA gadget on the LPC1768 version of the mbed (the only version I use).  I would still have to deal with CS' after every 16 bit transfer so I don't know if the DMA is worth the effort.  But I don't know that it isn't, either.

I might also use DMA with the serial port.  I would probably have two buffers of 160 bits (I just decided to use a 32 bit checksum), one where SPI data is being accumulated (by DMA?) and the other where the UART grabs data via DMA.  Swap buffers at the end of every transaction.  The reason for the longer checksum?  Well, the ARM is a 32 bit machine so the entire 128 bits is merely 4 words so add another word for the checksum and the calculation won't take a lot of time.  Add 4 words, complement the result and stuff it in the 5th work.  Something like that.

Given the 96 MHz speed of the mbed, I could probably compute a few thousand digits of PI in the spare compute time.  That time will be set aside for 'feature creep'.

[rstofer]

Also another concern with MCUs is fail-safe design.  What happens if one or more of the MCUs crash?  What state will the outputs be in while a device is booting / rebooting / reinitializing?  What is to prevent a cosmic ray or power glitch or ESD spike or program bug from causing a node to crash?  What will reboot it?  Will it be in a "safe state" while it is down / rebooting / whatever?  What will be the impact on the system?  Do you have / need any means of fault monitoring / handling if any particular node doesn't work and needs repair / rebooting / whatever?

This may be the hard part of the project.  In my description above, I assumed all RS485 traffic went to the same device at the far end.  Since this isn't the case, the ROBIN thing takes on more importance.  The far end devices can be 'addressed' individually.  Making up the packets on the input end is a little more complicated, especially if the bits don't line up and you want separate packets for each far end device.  I'm not sure I would do that.  I would let the receiver pick its own bits out of a full 128 bit packet.

The checksum takes on more importance in verifying the integrity of the system as a whole.

Since this is a hobby forum, I sincerely hope we aren't talking about some kind of machine where injury or death is a possibility.  If there is this possibility, disregard everything I wrote.  I don't do that kind of work.  When I did, I didn't worry about the cost of wire and I certainly wouldn't have used a gaggle of Arduinos.

[RogerRowland]

Since this is a hobby forum, ...

Is it? Really?

[Siwastaja]

You can meet the easy 5ms constraint in practically any way you want, there are dozens of solutions which all work fine. Shift register chain is fine.

Maybe even with an Arduino - that's why it keeps popping up in suggestions, even when it's only designed for artists to easily blink an LED, with huge overhead in basic IO functionality. (Another question would be why would you. It's totally the wrong tool for the job, and chances are you need to redesign completely.)

[Siwastaja]

BTW, it's always funny when people start that MCUs are unreliable because of glitches and what if the cosmic rays crash the program!

Yes, it's indeed a remotely possible event. But what is the alternative? In this case, this could be done without an MCU, but it would require a whole bunch more of logic ICs, with complex and long routing between them. In that design, glitches and maybe even cosmic rays become a real problem orders of magnitude higher, than on any MCU.

To answer evb149's question about what is going to reset a crashed processor: a watchdog. Hopefully.

Digital logic of this complexity is going to be most robust using an MCU, by far. Use as much of the MCU resources as possible, it's going to be there anyway.

Any other solution will be big, hard to design, hard to verify, and really prone to glitches. What is worst, it's hard to fix, compared to software bugs that don't require a PCB respin every time.

If you want to fully avoid cosmic rays, don't do it at all. If you want to minimize risks, use a familiar MCU and utilize it as much as you can, minimizing everything else.

[Someone]

Yes, it's indeed a remotely possible event. But what is the alternative? In this case, this could be done without an MCU, but it would require a whole bunch more of logic ICs, with complex and long routing between them. In that design, glitches and maybe even cosmic rays become a real problem orders of magnitude higher, than on any MCU.

To answer evb149's question about what is going to reset a crashed processor: a watchdog. Hopefully.

Digital logic of this complexity is going to be most robust using an MCU, by far. Use as much of the MCU resources as possible, it's going to be there anyway.

Any other solution will be big, hard to design, hard to verify, and really prone to glitches. What is worst, it's hard to fix, compared to software bugs that don't require a PCB respin every time.

The modern world of digital logic sits in CPLDs and FPGAs, have fun trying to say they're less logically robust than a micro controller (micro controller physical IO is often more robust to electrical stress however).

[Siwastaja]

The modern world of digital logic sits in CPLDs and FPGAs, have fun trying to say they're less logically robust than a micro controller (micro controller physical IO is often more robust to electrical stress however).

Most FPGAs (which is my earlier expertise area, although haven't been doing that for a few years now, so there may be more suitable chips on market now) would be total overkill for the project.

CPLD (or some easy-to-use, small, cheap (less than $50), FPGA not requiring three power supplies, configuration memory chip and 6-layer board) could work very well, and in theory, would be kind of "ideal" solution, with good integration, good robustness and clock tree analysis from the synthesis tool, but the problem is that few of us have the expertise on how to program these. If OP had the required skills, he would already be doing it that way, or would have at least mentioned it. Otherwise, diving into the world of HDLs and logic synthesis is quite a big step.

Today's micros are a lot more than just a CPU. The peripherals are like hard macros on FPGA, and as you should know, the hardest parts on FPGA are complex state machines, communication, configuration, etc.; for that purpose, you almost always synthesize a soft core, or use an FPGA with a hard CPU core. (I was a masochist/purist and did lot of things in logic, avoiding CPUs, but it was not easy nor robust. There are loads of bugs in my old overly complex FPGA projects.)

And trust me, when you design a state machine on FPGA using Verilog or VHDL, you very easily end up in states that crash the thing. Writing and analyzing everything takes 10-100x more time than writing the similar state machine on a CPU, even when you are experienced in both.

Once I debugged a system crashing into an unexpected state (state register outside of possible range), doing random things, for days. The fault was a missing synchronization register structure in one input, in a totally different place. Can't see that in simulation. It was a awkward mistake, but we make them.

The good visibility of ModelSim was of limited help when the shit hit the fan only on the real FPGA prototype.

So really, unless you want to do very special or performance demanding things on logic, you don't want to use an FPGA. And there is nothing special in this case. This is literally a textbook example of a simple MCU project.

74 logic would be a nut job, FPGA would be an overkill (and really requires a lot of expertise!! I know this very well!), MCU is exactly the right tool for the job - an easily configurable and robust cheap little block that can be programmed to read input and toggle output pins, using the input peripherals; for most input schemes, there is an MCU that provides it: be it RS232/RS422/RS485 UART, SPI, I2C, CAN, USB, Ethernet... And it's hardwired and tested to work, no need to synthesize it first :-).

And any of these solutions can glitch and have bugs (and is susceptible to cosmic rays).

But with an MCU, the lines-of-code figure is most likely the lowest; the expertise of the OP is most likely the highest; and the solution is by far the most supported by the community, which makes getting help (or handing the project out later) easier.

This is a funny discussion, because the thing itself is trivial, you can do it robustly and easily with almost any MCU in one weekend (more time needed for robust mechanical construction, electrical protection, PCB design type of crap that needs to be dealt with regardless of the solution), but the discussion jumps between the two extremes of using an Arduino and thinking about FPGAs and cosmic rays

[Someone]

The modern world of digital logic sits in CPLDs and FPGAs, have fun trying to say they're less logically robust than a micro controller (micro controller physical IO is often more robust to electrical stress however).

Most FPGAs (which is my earlier expertise area, although haven't been doing that for a few years now, so there may be more suitable chips on market now) would be total overkill for the project.

CPLD (or some easy-to-use, small, cheap (less than $50), FPGA not requiring three power supplies, configuration memory chip and 6-layer board) could work very well, and in theory, would be kind of "ideal" solution, with good integration, good robustness and clock tree analysis from the synthesis tool, but the problem is that few of us have the expertise on how to program these. If OP had the required skills, he would already be doing it that way, or would have at least mentioned it. Otherwise, diving into the world of HDLs and logic synthesis is quite a big step.

FPGAs have come a long way since you looked at them, and even 10 years ago there were simple and cheap options. Single or dual supply, on two layer boards with the entire "solution" under $10 in one offs are easily found to match this problem. CPLDs and FPGAs get sprinkled around modern systems because they are so cost effective.

Today's micros are a lot more than just a CPU. The peripherals are like hard macros on FPGA, and as you should know, the hardest parts on FPGA are complex state machines, communication, configuration, etc.; for that purpose, you almost always synthesize a soft core, or use an FPGA with a hard CPU core. (I was a masochist/purist and did lot of things in logic, avoiding CPUs, but it was not easy nor robust. There are loads of bugs in my old overly complex FPGA projects.)

So really, unless you want to do very special or performance demanding things on logic, you don't want to use an FPGA. And there is nothing special in this case. This is literally a textbook example of a simple MCU project.

The requirement for so many inputs in this system is already a clue this isn't a good fit to traditional ideas, its not simple when you need to start chaining multiple microprocessors or servicing a large number of peripherals.

This is a funny discussion, because the thing itself is trivial, you can do it robustly and easily with almost any MCU in one weekend (more time needed for robust mechanical construction, electrical protection, PCB design type of crap that needs to be dealt with regardless of the solution), but the discussion jumps between the two extremes of using an Arduino and thinking about FPGAs and cosmic rays

Its not trivial or there would be no discussion, making a realtime system on this scale and complexity needs some thought and planning. But dismissing FPGAs or CPLDs from your limited experience would be foolhardy.

[mikeselectricstuff]

CPLD (or some easy-to-use, small, cheap (less than $50), FPGA not requiring three power supplies, configuration memory chip and 6-layer board) could work very well, and in theory, would be kind of "ideal" solution, with good integration, good robustness and clock tree analysis from the synthesis tool, but the problem is that few of us have the expertise on how to program these. If OP had the required skills, he would already be doing it that way, or would have at least mentioned it. Otherwise, diving into the world of HDLs and logic synthesis is quite a big step.

e.g. Lattice Xo2 - single supply, onboard flash, easily useable on 2L PCB, cheap, chep devboard ( with onboard programmer) . For something like this which isn't anywhere near pushing speed or size capabilities, the learning curve isn't necessarily that big - if you're good at learning new things, a couple of days to get something basically running on a devboard would be doable.
If you're more of a hardware person than software, HDL may well be more natural to you than code.

However in general an MCU will pretty much always be a quicker, cheaper and easier solution than an FPGA or CPLD where you don't need the speed and/or parallelism of the FPGA.
One possible advantage of FPGA is you have a lot of I/O on one chip

For this particular application, an MCU is entirely feasible, even an AVR, and will offer a lot more flexibility than an FPGA/CPLD - e.g. error detection/checking, built-in test functions, timeouts, retries etc.
The whole "what if it crashes" is pretty much irrelevant these days, and no more or less of a risk than problems with a CPLD or FPGA solution.

[max_torque]

Without knowing what this network is feeding, no one except the OP can determine which is the "right" way of doing this!

128 digital outputs with max 5ms latency?  Could be a lot of things, from an LED array to some sort of weird data bus controller!   And in case of a bit error, does it matter?  (clearly, for the led array, probably not, you just get an led on at the wrong time till the next update corrects it)

AT low volume, for projects such as these, human error is the big problem. The chances of wiring up lots of shift registers and looming the whole thing up without messing it up are slim, whereas any mass produced micro (AVR/PIC etc) is going to be broadly speaking, very reliable and extremely well tested.  Software bugs can be fixed with less resource, and software can even correct other errors (for example, wrong output pin connected to wrong channel) and it can self test, immediately bringing a lot of functional reliability to the table.

[Jeroen3]

Most of the 5ms will be eated by input filtering though. Especially if the source is a relay or button.
I can only recommend CAN bus. Due to is being cheap, robust, and mostly taken care of in hardware.

[CM800]

I'd vote that CPLDs are a good way to do things, I've fallen in love with them.

You don't have to do VHDL, though it's not that bad when you arn't crazy worried about ns timing.
Just use an Altera CPLD (really easy to integrate) and program with a Schematic entry where you can drag/ drop logic.

[Ian.M]

OTOH for a one-off project, the main costs are likely to be the PCBs and the development time.   The shift registers are fairly cheap even though you need a lot of them, and are also very easy to code for.

If you order eight off of your input PCB and of your output PCB, (a very small cost increase over a single board + the area will be a lot smaller which also keeps costs down) and put two shift registers on each board + balanced line drivers and receivers and a local regulator, you have a modular system that can be hooked up as a daisy chain over CAT5 cabling (+ separate power and ground).  Output boards pass MISO straight through from their daisychain connector and input boards pass MOSI straight through the same way. I'd probably use TPIC6C595 power SIPO shift registers to provide robust outputs, protected by clamps and polyfuses if something PLC-like is required, or ordinary logic shift registers if logic level I/O is what's needed.

For a one-off, a reliable connection to the master Arduino SPI can be made by assembling an 8P8C socket + the line drivers/receivers on a proto-shield.  An Arduino UNO can sustain 2.2 MBPS SPI transfer rate @ 8MHz SPI clock, so there shouldn't be any problems meeting the 1ms latency requirement.

If however you need >100 units, a high pin count FPGA, CPLD or slave MCU based solution starts to make a lot more sense.

[rstofer]

Since this is a hobby forum, ...

Is it? Really?

I certainly hope so!

I spent my youth doing industrial automation in an aircraft plant.  True, we didn't have the magic that is now commonly available but we were deeply concerned about safety.

I can't say I know anything at all about the OPs applications beyond the bit count and that is pretty trivial knowledge.  Where does the hard 5 ms number come from?  If these are human inputs, time is irrelevant because you have no idea how long it took the operator to respond.  They are probably still BSing at the coffee machine.  OTOH, for machine control, 5 ms is a lifetime if one of the signals is some kind of physical limit switch.  Switching time for industrial relays probably can't do 5 ms.  In fact, the first relay I came across has 16 ms pickup and 8 ms drop-out:
https://www.zoro.com/schneider-electric-iec-control-relay-2no2nc-480vac-10a-ca2kn22t7/i/G2108547/

If ECCs are going to be used (they seem like a really good idea), an FPGA will almost certainly be required simply because the uC is going to get bogged down in the calculation.  I haven't personally done this so I don't know how many machine cycles it might take but I'm betting quite a few.  But it's exactly the kind of thing the FPGA/CPLD was designed for.  The CPLD will almost certainly have faster startup since it retains its configuration.  Some small FPGAs also retain configuration.

Transmitting bits is easy but, more generally, how do we frame the packet?  How much overhead is involved?


Here's where I come out as an industrial electrician turned EE with experience through the entire range of wire in conduit to FPGAs modeling retro computers:  If this is some kind of machine where safety is a consideration, use wire.  Believe me, 128 wires isn't all that much.  Put up a 4"x4" wireway and lay them in. Need another wire?  Just pull it through!  Compare that to the complication of adding another IO expander, changing the packet size, changing the code on every uC and re-verifying the operation!  That's certainly the way it is done on machine tools.  Or, use PLCs with networking.  Something like the Fanuc series with their network (used to be called the GE Genius Bus) and spend a lot of time worrying about how the machine starts up from scratch.  Usually, there will be some way to prevent motors from starting until the control system is up and verified.  I don't know if the Genius Bus can make the 5 ms real-time limit.  Nor have I ever thought much about the loop time for the PLC.  I'm guessing they can't meet the time constraint.

I certainly hope this is for a hobby.  Something where there is no risk of injury from a failure.

[Jeroen3]

At work I have a design ready with SN65HVS881. (http://www.ti.com/lit/ds/slas642/slas642.pdf)
Earlier generations used direct-mcu wiring, with only attenuation. Which has been proven to be very unreliable if it's used in an installation that has contactors and vfd's.

The topic seems to have drifted to engineering ethics. Which is important, but if there is a hard deadline of 5ms. Someone did some thinking beforehand.

[RogerRowland]

Since this is a hobby forum, ...

Is it? Really?

I certainly hope so!

....

I certainly hope this is for a hobby.  Something where there is no risk of injury from a failure.

I wasn't querying whether the OP's project was a hobby, I was querying your assertion that this is a hobby forum.

[Jeroen3]

It's not so much about the protections. It more about the thresholds.

[hamster_nz]

Have you considered ethernet shield and UDP for the transport?

Why? Plenty of reasons!

 Frame checksums, long cable runs, standard cables, low cost commodity parts, interface ruggedness, no EMI worries, debug/monitor with a laptop, standard wall plates, can interface to a PC (or small SBC), fan out as much as you like, can go over fibre with media converters, use power over  ethernet if you want, WiFi, DHCP, can also send data over internet if you need to, can ping nodes for monitoring...