"DIY instrumentation bus" (or DIB)

[prasimix]

I'd think that is a good time to initiate discussion about a few typical use case/module functionality that should help us to better shape demands on the hardware/backplane level. I believe that we don't need more then three examples that belongs to e.g.:

a) entry level,
b) mid-level and
c) high-level.

Please take into account that my ambition here is not to compete with professional/commercial solutions, but to find an acceptable balance between often very basic DIY/hobbyist and entry-level or accidentally mid-range commercial devices (depends which brand is in question ). What I'm going to propose I'd like to build in a foreseeable future and could serve as proof-of-concept (to avoid reference design title):

a) Multiple digital I/O with and without isolation and low-power/signaling switching matrix (e.g. 4 x 4)
b) 4-quadrant programmable power module, with triggering/sync, and low-bandwidth function generation/AWG and
c) Data acquisition module, e.g. min. 20 MHz bandwidth, min. 100 Msamples/s (8- to 12-bit), single or multiple channels (multiplexed or simultaneous), eventually with single/dual high speed DAC for function generator/AWG or basic "network analyzer" (Bode chart plotting) with min. 10 MHz output (8- to 12-bit).

Your candidates are highly welcomed, together with comments to what is suggested above.


 

[jbb]

I’ve kept thinking (yes, it’s been hard for me :-) ) about a few issues here.

How does the following sound?

• Quite dumb backplane with master slot and N slave slots.
• Isolation to be done in module to keep backplane simple.
• Hot swap is probably not required - it adds a lot of complexity. I think prasimix’s sketches show a bottom plane (rather than a back plane) which would complicate the mechanics anyway. I still like the thought of using early mating ground pins to reduce the risk of ESD damage.
• Use of M-LVDS, which is a mulidrop differential standard explicitly designed for backplanes. Bandwidth of 100 MHz (200 Mb/s) is achievable.

Core bus

• A single shared SPI bus for module ID and dumb cards (e.g. basic digital IO) that don’t need a micro (thanks prasimix).
• One dedicated UART per module (Could easily run at 10 MHz / 16 = 625 kb/s).
  (I suggest UART rather then separate SPI because it’s a bit fiddly to write an SPI slave).
• 10 MHz clock distribution over M-LVDS. Clock source is master (default) or timing slave module.
• Synch pulse (maybe 50ns pulse @ 10 kHz?) over M-LVDS. Source is master or timing slave module.
• Bidirectional trigger line over M-LVDS.

Standard power bus

• Plenty of 5 / 12 V power for master
• Moderate amount of power @ 3.3 /  5 V for unisolated side of modules.
• Medium freq AC bus (10s of kHz?) for isolation transformers on modules (thanks David Hess)

Extended bus

• Several (8, 12, 16?) lanes of M-LVDS.
• A small menu of standard options, e.g. trigger, single/dual/quad SPI data (SDR and DDR), 8b/10b packet (DC levelled, can go through coupling transformer or fibre optic), custom use-at-your-own-risk.

[Richard Crowley]

The Gigabit Ethernet technology seems pretty good as a command/control/data bus.
It can handle very fast speeds, bi-directional, between multiple nodes on the bus.
And it is galvanically isolated (transformer-coupled). (Or optically coupled with some standardized bus configuration)
At least for the OSI (Open Systems Interconnect) Layer 1 (Physical).
The hardware is dirt-cheap and standardized even for Gigabit speed.

And there are open source software solutions for the other six layers.
Depending on which level(s) would make sense for the project.

1. Physical layer  - bit level - electrical/optical and voltages, connectors, pinouts, etc.)
2. Data link layer  - frame level - IEEE 802 defines Medium Access Control (MAC) and Logical Link Control (LLC)
3. Network layer - packet level - datagram addressing, routing and traffic control
4. Transport layer - Segment (TCP) or Datagram (UDP) level - segmentation, acknowledgement and multiplexing
5. Session layer - data level - session management
6. Presentation layer - data level - translation between network comms and the actual "application"
7. Application layer - data level - high-level API

https://en.wikipedia.org/wiki/OSI_model

[xani]

Sounds interesting. Shame there are no micros (afaik, just looked thru some Cortexes and not much more) supporting differential signaling out of the box, but then transceiver woudn't be needed just to receive trigger/clock signal.

The Gigabit Ethernet technology seems pretty good as a command/control/data bus.
It can handle very fast speeds, bi-directional, between multiple nodes on the bus.

gigabit ethernet is not multidrop, it is point-to-point only.

Which means that either:

• master module of say 8 slot chassis would have to have 8x7 = 56 lanes connected to it
• backplane itself would have to have ethernet switch in it - ain't that expensive but significantly complicate the backplane compared to "dumb bus" solution

It also makes "dumb but fast" modules like fast ADC/DAC harder to make, as instead of "just" bus arbitration you'd need to implement MAC/ARP/DHCP/IP at the very least and probably at least UDP (as going anywhere below that just makes it pain in arse to interact from PC) and probably just straight LXi support.

But on the other side, it would make "master" module side "just a rPI" and using it a breeze as you'd basically use module like any other ethernet-equipped test equipment.

[Richard Crowley]

gigabit ethernet is not multidrop, it is point-to-point only.

That assumes you are using all the levels of 802.3 "Ethernet" protocol.
I am suggesting that you CAN implement multidrop and/or whatever else is needed by establishing new protocols for the upper layers.

[jbb]

Going back a few posts, prasimix asked about example modules. Here’s some thoughts which might help us clarify requirements.

Basic modules
Slow digital IO
Slow analog IO (multimeter style or power meter)
UART breakout
Basic PSU / load

Medium modules
Pulse / PWM generators
Medium speed IO
Waveform generation
Short duration waveform capture

Advanced modules
Real time signal processing (requires low, consistent latency)
Wide / fast analog capture

On the isolated power front, I think that a separate forum thread might be more appropriate?

Edit: constrain things -> clarify requirements.
Edit: forgot 'forum thread'

[xani]

Might be useful to have more than one trigger in "core config". Like 2 would be useful if you wanted to do "start recording at X condition, stop at Y".

Or "start recording at trigger 1, start power supply at trigger 2"

Or use second one as "panic button", make it stop all power supplies and hook it to temperature sensor attached to battery and to voltage sensor that triggers if battery goes above 4.22V, for testing battery chargers.

Also, option to tie one of digital I/O pins to trigger might be pretty neat
As for examples:

• External trigger IO -  receive triggers, generate them and send them to chassis/external output with different delays, say "send trigger to ext out then 100ns later to the chassis
• Thermocouple module  - few typical profiles + some flash to load your own profile + option to just work like millivoltmeter because why waste ADC only on temperature
  
• "Arduino" - bare minimum to run an arduino-compatible bootloader on AVR or better, some Cortex

That sounds like royal pain in arse for everyone involved honestly
gigabit ethernet is not multidrop, it is point-to-point only.

That assumes you are using all the levels of 802.3 "Ethernet" protocol.
I am suggesting that you CAN implement multidrop and/or whatever else is needed by establishing new protocols for the upper layers.

Using a standard just to throw away most of it seems like royal pain in arse for everyone involved. You're basically throwing most of advantages of having ethernet while compensating for none of it disadvantages.

Also "multidrop" is function of layer 1, layers above it have nothing to do with it. So you can't just take ethernet chip and "implement multidrop"

100Mbit ethernet *can* actually do it (via passive hubs), but 1Gbit can't .

[David Hess]

Using a standard just to throw away most of it seems like royal pain in arse for everyone involved. You're basically throwing most of advantages of having ethernet while compensating for none of it disadvantages.

I agree.  One of the appeals of Ethernet is cheap expansion hardware allowing me to plug an instrument in anywhere the LAN reaches which can be a considerable distance.  For an instrumentation bus which will live inside of a single chassis, most of this capability is superfluous.

Also "multidrop" is function of layer 1, layers above it have nothing to do with it. So you can't just take ethernet chip and "implement multidrop"

100Mbit ethernet *can* actually do it (via passive hubs), but 1Gbit can't .

I considered mentioning this.  I even keep a 10/100 ethernet hub for special applications where it will work better than a switch.

It has been a while since I looked at this and someone can correct me if I am wrong but 10Mbit and 100Mbit Ethernet on twisted pair use one pair for transmit and one for receive so the hub function is more than just bridging the signals; it selects and repeats one transmitted signal to all of the receivers so it is more than just a passive function.  Switches do actual store and forwarding.

I always wondered if 100 Mbit Ethernet could be adapted to operate with carrier-sense multiple access with collision detection on a shared media like 10 Mbit Ethernet can but I never saw this implemented.  I have a pretty good idea how it could be done while using a standard 100 Mbit Ethernet physical layer interface.

[xani]

I considered mentioning this.  I even keep a 10/100 ethernet hub for special applications where it will work better than a switch.

It has been a while since I looked at this and someone can correct me if I am wrong but 10Mbit and 100Mbit Ethernet on twisted pair use one pair for transmit and one for receive so the hub function is more than just bridging the signals; it selects and repeats one transmitted signal to all of the receivers so it is more than just a passive function.  Switches do actual store and forwarding.

Yes. That allows for few neat tricks like passively tapping traffic on 100Mbit ethernet, or having fully passive hubs (at cost of signal degradation and possibly dropping speed to 10Mbit thanks to that).

As for switches we actually went full circle and some enterprise switches are cut-thru (they read only enough to get the header then immediately start sending before receiving whole packet) and only switch to store-and-forward if outgoing port for a packet is congested. That is done to cut few extra us (or rather ns in case of 10Gbit+) off latency

I always wondered if 100 Mbit Ethernet could be adapted to operate with carrier-sense multiple access with collision detection on a shared media like 10 Mbit Ethernet can but I never saw this implemented.  I have a pretty good idea how it could be done while using a standard 100 Mbit Ethernet physical layer interface.

I'm guessing that with prices falling it just didn't make sense to bother with it. Especially considering universal hatred for 10base2 shared by basically any old networking guy I ever talked to

[prasimix]

Thanks all once again for valuable inputs. I'd try over the weekend to compile all what is said in new proposal draft and publish it for your review.

[prasimix]

PCBs of XMOS evaluation board mentioned in #28 just arrived from ALLPCB (I managed to order it before Chinese holidays). They sent me 10 pcs (it seems that is minimum value despite the fact that 5 can be selected during ordering process) and I don't expect to need more then 3. Therefore if someone found this circuit usable in one or other part (since you don't need to populate everything) feel free to contact me and I'll send one PCB (for the price of shipping cost, that should be about 8 EUR in the EU). You can find schematic on the GitHub here, or complete pre-release here.
Due to yesterday's unfortunate event I'll not be able to start assembling my boards soon.

[prasimix]

I'm working on new pin mappings trying to include some of your thoughts and saying that is quite challenging is simply understatement. Before posting another proposal, and waste your time reviewing it, I'd like to know what you can say about the following:

1. If differential transmission line is used, I presume there is no need to have GND next to line+/- pins. But, if that is not a case, and we we have multiple SPI channels as in first proposal (see in post #29 Slot#0, pins C5 thru C32) and presume that isolation is going to be deployed, do we really need that 7 GND pins used as "separation"? Such "better grounding" is actually only valid on the Slot #0 side since we have here star topology. I'd like to spare that pins since introduction of more LVDS lines ask for many additional pins.

2. Are you aware of any mechanism/wiring that can be implemented when same signal line(s) can be used as single-ended OR differential? In that case a part of module identification info stored in EEPROM could contain info about module capability / speed requirement. When such signal lines are not shared (i.e. multidrop) then lo-speed module could be connected with single-ended signal. In extreme case a whole chassis could be lo-speed or on other side hi-speed.

3. Does it make sense to spare some pins (primarily on the hi-speed LVDS signals) by introducing Isochronous self-clocking signaling?

[jbb]

Sorry about the 'scope; what a pain.

I'm working on new pin mappings trying to include some of your thoughts and saying that is quite challenging is simply understatement.

Good plan.  Sorry about the flood of mutually-conflicting ideas.
I do have one proposed simplification about Ethernet (which has cropped up a bit here); if people want an Ethernet-based instrument system, they should consider an open source EtherCAT slave for low cost micros.

1. If differential transmission line is used, I presume there is no need to have GND next to line+/- pins.

I'm not sure - this is a signal integrity matter.  While 1 ground pin per LVDS lane would be nice, it will indeed chew through 96 pins really fast. Do we have any signal integrity people in the house?? Note that when the next question comes into play, the concern becomes stronger...

If using point to point (standard) LVDS I suspect it might be possible to reduce the number of grounds a bit.  If using M-LVDS (Multipoint LVDS), I would recommend against it because a) the signal integrity issues are more complicated and b) there just wouldn't be as many lanes chewing up pins.

2. Are you aware of any mechanism/wiring that can be implemented when same signal line(s) can be used as single-ended OR differential?

That's an interesting idea.  I know FPGAs can do it, and I guess that some carefully-deployed driver chips could do it too.  There might be some crosstalk issues through the backplane (not a signal integrity expert...)?

3. Does it make sense to spare some pins (primarily on the hi-speed LVDS signals) by introducing Isochronous self-clocking signaling?

I would be nervous about that.  The catch with this sort of thing is that you either a) need some analogue systems to do PLL-type things to generate the receive clock in each module or b) need to over-sample digitally (limits max baud rate) and then manage bit slips etc.  A synchronous reference clock makes it much easier to receive things, and also helps with overall system jitter when fast triggers are involved.

It goes a bit against the 'dumb backplane' idea, but it would be possible to put a fanout buffer on the backplane. Multipoint LVDS (M-LVDS) offers multi-drop, and so would be good for this clock distribution task.  However, it looks like standard LVDS (while needing the fanout buffer) can pass through an Ethernet transformer for isolation.  M-LVDS almost certainly can't...

[prasimix]

I agree that backplane should stay passive, therefore no fan-outs and similar thing should reside on it.

I'm still not sure that differential transmission lines require GND pin in close proximity for "return path". Other thing is that GND plane is needed beneath lines to set correct impedance.

Regarding (re)using diff. lines as single-ended I'm still searching for more information. It's simple to imagine that some chassis will be populated with lo-speed modules only without all overhead related to LVDS drivers and isolations (e.g. mentioned Ethernet transformers). Maybe in that situation unused line can be used for return GND. If that is a case, the question is do we need some kind of "hard-coded" protection (e.g. connector key pin) to prohibit inserting module with LVDS drivers into chassis with modules that use diff. lines as single-ended.

[xani]

Maybe just have additional connector for hi-speed differential ? Then low-speed module would just... not mount it.

One option for relatively fast and low pinout bus would be just USB2.0, just 2 pins for >400MBits of bandwidth, that is also very easy to interact with directly from micro.

[prasimix]

One option for relatively fast and low pinout bus would be just USB2.0, just 2 pins for >400MBits of bandwidth, that is also very easy to interact with directly from micro.

But a sort of USB hub or switch is then needed to be implemented on the processor module (#0) or on the backplane, or not?

[xani]

Yes it would, however there are pretty much commodity. Disadvantage is quite a bit of overhead when coding for it (altho there is plenty of ready-made code to support) and possible issues with bus contention (as you can't really control how usb hub handles traffic)

[jbb]

Just FYI, isolating LVDS with a transformer requires the use of DC-level coding eg 8b/10b (which is commonly used). Unfortunately that will likely require SERDES chips or FPGAs (could use ICE40s).

It’s possible to get quite good software triggering, especially if a good clock is distributed.

[Vtile]

I didn't read this through, but I hope there have been though of:

- Power consumption, should be compatible with low power battery equipment.
- Separation, Should be galvanically isolated.
- Complexity, should fit in a minimum of memory and be build with ease.
- Cost, Should use simple parts and techniques
- Measurable, medium speed.


IE. Interesting protocol SW/HW were HP-IL

[prasimix]

Yes it would, however there are pretty much commodity. Disadvantage is quite a bit of overhead when coding for it (altho there is plenty of ready-made code to support) and possible issues with bus contention (as you can't really control how usb hub handles traffic)

Great, and we are again in situation that asks for MCU on all modules (master/control and slaves/peripherals) or at least a kind of USB-to-somethingelse bridge on peripheral modules.

Just FYI, isolating LVDS with a transformer requires the use of DC-level coding eg 8b/10b (which is commonly used). Unfortunately that will likely require SERDES chips or FPGAs (could use ICE40s).

It’s possible to get quite good software triggering, especially if a good clock is distributed.

He, he, and guess which MCU (not FPGA) has serious built-in SERDES functionality: XMOS again

I didn't read this through, but I hope there have been though of:

- Power consumption, should be compatible with low power battery equipment.
- Separation, Should be galvanically isolated.
- Complexity, should fit in a minimum of memory and be build with ease.
- Cost, Should use simple parts and techniques
- Measurable, medium speed.

Power consumption is currently not discussed, nor specific pins or functionality is assigned for power management (so far only +5VAUX as stand-by power is listed).

Isolation is indeed addressed, and it's a must for some instruments.

Min. memory yes, as far as it's 256KB or more (talking from existing experience). Avoiding QFN, BGA and similar packages can make building simpler (no stencil, oven, etc.).

Cost and BOM (i.e. parts count) is important and that why we are discussing here what underlaying HW protocols/buses to use as "basic/cheap" and "advanced/expensive".

Medium speed yes, if that mean something in-between 10 and 200 Mbps.

IE. Interesting protocol SW/HW were HP-IL

Yes, it was, let rest it in peace

[Vtile]

How you measure something like 200mbps without 2k5+ eur/dollar instrument?? Sounds pretty far from diy.

[prasimix]

Yes, one thing is measuring (or sampling) something 200 million times per second, and another one is to have a bus that is capable of transferring with such speed. If you take a look at e.g. LVDS you can see that 100/200 Mbps drivers are available for decent price.

Even if we come to measuring at least 100 MSps is achievable without spending multi Kdollars. Currently we are not discussing any instrument capabilities in details, but yes, a few use cases should be useful in shaping a whole thing as I suggested that in #76. To get a better picture I'd like to suggest you to spend a little more time to read thru all what is already proposed and discussed. You'll see among other stuff that topic is highly antagonistic: it easily gather two groups of "believers" and "non-believers" in just-another-standard as it can be baptized . But, let's be realistic, I'm not the one who can alone set any standard (by any standard!), I just want to make development of my future toys more manageable and reusable. Getting feedback from community is highly appreciated (and proven to be very fruitful if talking from previous experience). Finally, if DIY in topic title looks misleading, I can tell you that is not, because everything what I was designed so far belongs to DIY or in other words, I believe that if I succeed to make some stuff then many others definitely could do the same thing (even better).

[xani]

Yes it would, however there are pretty much commodity. Disadvantage is quite a bit of overhead when coding for it (altho there is plenty of ready-made code to support) and possible issues with bus contention (as you can't really control how usb hub handles traffic)

Great, and we are again in situation that asks for MCU on all modules (master/control and slaves/peripherals) or at least a kind of USB-to-somethingelse bridge on peripheral modules.

I meant USB as an option for mid-hi-speed, not as requirement for every module. Extracting data is also easier, as it is "literally any rPi-like board with USB and some basic coding" vs "probably an FPGA with a bunch of LVDS channels and few RAM chips"

Besides, I don't really see a problem in including $1 micro on each one considering even connector is few times more expensive than that. It would probably be even possible to program it in-chassis for micros that have serial bootloader in ROM

Just FYI, isolating LVDS with a transformer requires the use of DC-level coding eg 8b/10b (which is commonly used). Unfortunately that will likely require SERDES chips or FPGAs (could use ICE40s).

It’s possible to get quite good software triggering, especially if a good clock is distributed.

He, he, and guess which MCU (not FPGA) has serious built-in SERDES functionality: XMOS again

Without builtin 8/10b that's still some work to make it work and still requires driver; but it also does have USB

[Vtile]

No I did mean that if this is something that is supposed to be done at home or small outsourced batches. Then how you can troubleshoot the electrical side of the things without expensive instruments out of the most hobbyists reach.. Ie verifying that the edges aren't ringing and that sort of measurements.

Heck I can't even troubleshoot 100Mbps ethernet in my current job, since it is cheaper just to swap parts compared to expensive equipment needed. Gain compared to 'vintage bus' not much at all tbh.

[xani]

Hobbyist with 50MHz scope wont be able to debug 100Mbit LVDS very well either so pretty much any high-speed option would be hard for most hobbyists to debug