Distributed Data Acquisition Scheme

[Evan.Cornell]

Here's the setup:

• Central controller/processor (clock master)
• 2-5 sensor boards (connected to central processor board by undetermined cable of up to 10-15m length

The central processor would produce clock signal(s) that need to get distributed to each of the sensor boards, and those clock(s) would result in data being transmitted back from each sensor board to the central processor. The idea is to have all sensor data synchronized and simultaneously sampled.

I saw some material about Automotive Audio Bus from Analog Devices that may do all this well, but it looks like it's behind an NDA-wall, and availability probably isn't great, since this wouldn't be a high-volume automotive application.

That then led me to SerDes, but it seems like I'd have to have two channels to each sensor board, one for outgoing clock(s) and one for incoming data.

Another option could be Precision Time Protocol (PTP) via Ethernet links, but I'm leaning towards a more hardware-based solution.

Each board may or may not have it's own power source (ultimately battery-powered, but could consider powering the sensor boards over Coax lines).

Any other thoughts on a clever solution?

[ajb]

What's your expected maximum data rate and what is your allowable skew between sensors? 

I would probably start with RS485, which works nicely over Cat5 cable, so you can have one cable carrying a clock pair, a data pair, and possibly power on the other two pairs.  The data link can be half duplex or simplex depending on whether or not the master needs to configure the sensors, and you can daisy chain from sensor to sensor if you want.  A single decent size MCU should easily get you up to 8 UARTs, so the master can just sit there sending out clock pulses and absorbing the data from the sensors and then doing whatever you need with it (throw it onto an SD card? Forward it over USB/Ethernet?).

[sokoloff]

If your measurements are small, I'd consider CANBus for this.

You can send a CAN message (received by all devices at once), have each node take a measurement and report its measurement back over CAN using different CAN IDs. CAN will take care of the hardware collision-detection/avoidance and message integrity, though you can help out a bit by randomizing or scheduling the timing offset.

Primary downside of CAN is that it's not a particularly high-bandwidth solution and each message packet is small in terms of payload.

[Evan.Cornell]

So each sensor board would have 8 data lines, 2 clock pairs (bit clock & frame sync). The data lines could be daisy-chained to reduce quantity to 4. Sample rate for individual data line is somewhere between 96 and 192kHz. Assume 32 bit per sample.

Measurements could be of long duration (at least several minutes at a time), so think continuous data stream to a recorder (either to local memory or packaged up and sent from processor board over network).

RS485 as the physical layer is a good idea, although perhaps with multiple cat5.. cabling is TBD. Any concerns about clock/data synchronicity if the cable lengths are different (say one cable is ~1m, one is 15m)?

[max_torque]

Without specifying the sampling rate, data bandwidth and maximum allowable jitter, it's tricky to suggest a suitable solution.

If your requirements are modest, then each slave module can just sample continuously, and the master can ask from the latest data at the appropriate moment over the serial link.   Simultaneous sampling is really pretty pointless unless you have a significant requirement to do so (driven by the thing you are sampling changing very fast)

[JVR]

Jitter and offset is going to kill you, even on a SERDES. And at 200Khz odd, you really don't want to use a SERDES.  Give us a block diagram and at least some spec, then we can make better suggestions. Because given the current info, you will find anything from a Arduino to a Artix-7 suggested, depending on the suggester.

[Evan.Cornell]

See attached for rough block diagram.

[ajb]

But again, how simultaneous is "simultaneous"?

How much offset between devices (skew) can you tolerate?  How much offset between the actual and nominal sampling times (jitter) can you tolerate?

The lower these quantities are, the harder your job becomes.

192kHz x 32 bits is 6.14Mbps per sensor, so one 10Mbaud link per sensor would barely allow you to add a bit of framing.  That's doable with RS485 transceivers.  You can actually get transceivers that operate at much higher baud rates, but everything else gets a bit trickier, so it would be easiest to have a dedicated link to each sensor.

The degree to which cable length matters depends on your allowable skew.  It takes something like 5ns for a signal to propagate 1m (speed of light times the velocity factor of the cable), so if you can tolerate, say 100ns of skew, then you can tolerate 20m of difference in cable length (less other things that would eat into that skew budget).  There's a simple workaround on the cable length, though, which is to simply make all of the cables the same length.

[nctnico]

What is you budget? These kind of data acquisition systems can be bought commercially so unless you have lots of time and no money you might be better off buying something.

[Evan.Cornell]

I'm not looking for off-the-shelf solutions, this is to integrate together pieces of a solution that already exist. It seems like multiple RS485 links for the clock and data lines may be the way to go, especially if I can ensure that all connecting cables are of equal length.

Timing tolerance should be within one bit-clock, so all data lines are synchronous. If we have 192kHz sample rate (for instance), and assume that two data paths are TDM'd on one data pin, then that yields bit clock of 12.288MHz (period = 81.38ns). So the allowable jitter/skew is going to depend on setup/hold times of the serial interface on the main processor board. Looks like I need to do some timing analysis to determine if the RS485 scheme would work reliably.

[sokoloff]

I can't help but think of the "X Y Problem" when reading this thread.

[nctnico]

I'm not looking for off-the-shelf solutions, this is to integrate together pieces of a solution that already exist. It seems like multiple RS485 links for the clock and data lines may be the way to go, especially if I can ensure that all connecting cables are of equal length.

Timing tolerance should be within one bit-clock, so all data lines are synchronous. If we have 192kHz sample rate (for instance), and assume that two data paths are TDM'd on one data pin, then that yields bit clock of 12.288MHz (period = 81.38ns). So the allowable jitter/skew is going to depend on setup/hold times of the serial interface on the main processor board. Looks like I need to do some timing analysis to determine if the RS485 scheme would work reliably.

This is much harder to achieve than you think. The solution I know about about uses the White Rabbit time & frequency protocol developed by CERN. CERN has developed White Rabbit for exactly the same reason you want to develop something new: synchronising data acquisition in a distributed network of sensors.

[David Hess]

So 10 to 15 meters in length, 2 channels of 192kHz (audio) sample rate for a bit rate of 12.288MHz (81.38ns), and simultaneous sampling?  That seems pretty straightforward with one complication.

Each board needs a 12.288MHz outgoing clock from the master, a strobe to control sampling and framing, and any data lines if required; some ADCs only require the strobe/framing signal.  RS-485 would work but so would a number of other digital transmission standards including logic with some care.

The complication is that cables have a delay of roughly 5 nanoseconds/meter which produces a round trip delay of 100 to 150 nanoseconds compromising the relationship between the 12.288MHz (81.38ns) outgoing clock and returning data.  A source synchronous returning clock and framing signal solves this and is probably the easiest way.  There are some transmission schemes which include clock and framing like asynchronous serial saving two returning signals.

[Evan.Cornell]

I see the point of round-trip prop. delay complicating a direct usage of bit & frame sync clocks. What is method to create returning clock and frame-sync on all remote boards that will ensure rough synchronization, assuming all cables are same length?

I took a quick look at the White Rabbit solution, which seems to be built on top of AVB / TSN ethernet backbone. As I posted previously, I would prefer a primarily hardware-based solution, mostly for development time purposes.

[ajb]

If the only purpose of the clock is to time your samples, then it doesn't matter as much.  Your central controller can simply send out a sample clock pulse, the remote devices take whatever measurement, and then they can send back the result asynchronously.  You could even use a UART, as long as you have enough headroom to frame it somehow (break signal, 32->40 bit encoding or something).  I'm assuming you have an MCU at each node, but you haven't specified whether that's the case or not, so I think others have different assumptions.

[David Hess]

I see the point of round-trip prop. delay complicating a direct usage of bit & frame sync clocks. What is method to create returning clock and frame-sync on all remote boards that will ensure rough synchronization, assuming all cables are same length?

The remote boards already have a clock and frame sync from the master controlling their data output so they repeat them back aligned with the data they are returning.

Or like ajb and I said, they return the data using an asynchronous or other scheme which does not rely on the clock and frame sync signals.

I am not sure what ADCs you are planning on using but they may have provisions to handle this in one way or another.

[Evan.Cornell]

I'm thinking now that the sensor boards ought to have locally generated clocks, transmit to main board using SerDes, and clock-sync everything using SRC to account for clock drifts. That's probably close enough in terms of synchronicity, and should make the cabling relatively simple.