syncronise multiple ADCs over an Ethernet network?

[mansaxel]

10MHz is what is the de-facto instrumentation standard, that can be interface to almost any existing instruments.

Either word-clock (sampling rate) shall be provided directly or a common 10MHz reference to all, that'd be used to derive sampling frequency.

Exactly, 10MHz can be had from so many sources, and there is distribution hardware that satisfies very tight phase relationship requirements, many measurement devices can consume 10MHz, et c.

[jbb]

Do you need every sample delivered? Or is missing the occasional sample OK?

To open up a completely new front, EtherCAT might be a contender. It’s an industrial Ethernet thing, where the slaves each have 2 ports and daisy-chain off each other. No fancy switch required. If using hardware space drivers (highly recommended), it’s got highly deterministic latency and each additional slave adds only 1.2us latency to the chain.

Pretty good (< 300ns if using a hardware timing master and hardware slaves) clock distribution is built in and can do regular timing.

We ran an EtherCAT system for the backbone of a megawatt class power conveyer with 100us command cycle timing.

[Yansi]

No fancy switch required

Haha...  Either very specialized EtherCAT driver IC, or an FPGA with lot of work to make it work.

Bute sure, this could work too.

You have just moved the fanciness from a simple PTP aware switch to a completely new level

[jbb]

Oi. I said no fancy switch required.

The slaves (especially the timing master which objects timing information) are a bit fancy.  We used an FPGA with licensed core from Beckhoff. The license was eye-wateringlh expensive. But I think TI offers ARM chips now with EtherCAT hardware.

Actually, I’m being a bit thick. I think Analog Devices have a new series of “A2B” chips for distributing audio over twister pair in a car. Maybe some of that stuff could be repurposed?

[max_torque]

Ethercat is (probably) out because of the cost / complexity.

We have looked at just buying a suitable DAQ system off-the-shelf, and one of the options was a Beckhoff based Ethercat system, and the costs skyrocket (compared to basic home grown ethernet connected slaves) and the hardware becomes non optimum in terms of it's physical layout  (ie mostly din rail mounted I/O without suitable environmental sealing, as our system needs to work in a reasonably harsh environment)

We are going to have to do a trade-off analysis between the various options, currently the least understood part is the master software that is going to bring all this data together under one roof so to speak, option shere range from DIY home grown software, up to National Instruments based Labview/Veristand etc.

Although the home grown option is by far the most work, it offers the most flexibility, but realsitically we will probably plump for something of a halfway house, buying the component parts we can buy, and developing the rest

[nctnico]

10kHz is one sample every 100us.

NTP might not reach that accuracy, but PTP (IEEE1588) will keep a pair of clocks synchronised much more closely than that over a single Ethernet hop.

For PtP for work well you need to use fiber ethernet connections and you'll need special add-on hardware which can synchronise to PtP to get a reasonable accuracy. Also PtP will lose accuracy if you pump a lot of data over the network.

The most simple way is to distribute TOD (over RS485), 1PPS and 10MHz over seperate wiring (using a distribution amplifier) and timestamp the data using that. Once the data is timestamped it doesn't matter how long it takes to travel over ethernet. If more accuracy and/or longer distances are required then using White Rabbit switches is the only option to stay below 1us of jitter (typically White Rabbit can transfer time with <1ns jitter over hundreds of kilometers when calibrated). For example CERN uses White Rabbit to synchronise test equipment used for the LHC.

[dmills]

Not been my experience with PTP at all, 1Gb/s with PTP aware switches and correctly configured traffic prioritisation and EEE turned off and I see a few hundred ns over multiple nodes even over copper and with several hundred Mb/s of other data running, go to 10 or 100Gb/s fabric and things get better again.

Sure you need to configure the gear correctly and having switches with PTP clock capability helps a lot, but sub microsecond is EASY!

Regards, Dan. 

[nctnico]

But did you do long term (days, weeks) testing with PtP using continuous monitoring? I have access to such data and have seen PtP misbehave. PtP can get upset quite easely so you need to have a good (expensive) local oscillator which is tracking the PtP slowly to average the excesses (and perhaps throw away large time jumps). Putting switches in between and using copper connections doesn't help to improve PtP synchronisation because in the end PtP still relies on packet delays. For example: copper PHYs also have internal packet buffering and re-synchronisation. This can easely add 100ns or more of time assymetry.

All in all a system will be much more stable if you can distribute the time over wires directly.

[Yansi]

10kHz is one sample every 100us.

NTP might not reach that accuracy, but PTP (IEEE1588) will keep a pair of clocks synchronised much more closely than that over a single Ethernet hop.

For PtP for work well you need to use fiber ethernet connections and you'll need special add-on hardware which can synchronise to PtP to get a reasonable accuracy. Also PtP will lose accuracy if you pump a lot of data over the network.

No, unless you try to sync for timenut nonsense precision.  .. which more then likely is not required here, neither for generic audio applications.

[OldEE]

Hi gang.

Thank you for the follow up to my mention of Dante.  I learned a lot from all you fine folks.

Also I misspoke about routing Dante.  What we actually did was to extend the Dante VLAN to another switch.
That was a glitch in the brain to keyboard link.

Larry

[nctnico]

10kHz is one sample every 100us.

NTP might not reach that accuracy, but PTP (IEEE1588) will keep a pair of clocks synchronised much more closely than that over a single Ethernet hop.

For PtP for work well you need to use fiber ethernet connections and you'll need special add-on hardware which can synchronise to PtP to get a reasonable accuracy. Also PtP will lose accuracy if you pump a lot of data over the network.

No, unless you try to sync for timenut nonsense precision.  .. which more then likely is not required here, neither for generic audio applications.

The point is that PtP can be accurate but the achieved accuracy will depend on many IFs. If the system the OP wants to build is intended for a local installation then distributing time directly is a much more cost effective and reliable solution.

[mansaxel]

For PtP for work well you need to use fiber ethernet connections and you'll need special add-on hardware which can synchronise to PtP to get a reasonable accuracy. Also PtP will lose accuracy if you pump a lot of data over the network.

The medium is irrelevant, as long as it is a cable. I have lots of end nodes connected via copper, and they behave just nicely.

The special hardware, yes. The switch needs to be talking PTP, anything else is an invitation to headache. Network cards in end nodes also need to support PTP.

If you have PTP support in the switch and end nodes, you can go near flatline full on the link without much problems. If you do not prioritize or otherwise handle the PTP traffic as important (and having a switch that is boundary clock is, in my experience, by orders of magnitude the best way to do it.), you will have problems at everything above say 40% load. 

[Evan.Cornell]

I think Analog Devices have a new series of “A2B” chips for distributing audio over twister pair in a car. Maybe some of that stuff could be repurposed?

Un-obtainium for small guys.. they've got all the good info behind NDA-walls. I was looking at this for a multi-drop data acquisition system (several nodes to one main board via 10m cables), but ended up going with individual point-to-point SerDes links instead (DS90UA101-Q1 & DS90UA102-Q1 from TI).

[Yansi]

Audio over twisted pair, like what? AES3? (AES/EBU aka professional SPDIF) - that's long known, no need for special chips for that.

DIT4192 and DIR9001 to the rescue.    (or any generic FPGA or even MCU these days)...

For multi-channel, there is the long known MADI over coax or SFP.  Not a twisted pair, but certainly could be pushed throuch any kind of physical medium, including twisted pair.  Note, that Soundcraft, those freaking bastards, use a RJ45 ports in between their mixing desks and stage boxes and call it "MADI".  How the hell can one call that MADI? Especially, when there runs also some RS485/422 withing the same cable to control the stage box. Why, Soundcraft, why calling that MADI?!

[GeorgeOfTheJungle]

If this is a dedicated network whose sole purpose is to control/communicate with these devices and not to download torrents and video streams from Netflix and youtube at the same time, I don't think he needs anymore than broadcast packets to synchronize/trigger the ADC conversions and a single UDP packet to transmit the samples back to the controller. IOW the clock signal are the broadcast packets. The smallest enet packet payload is fifty something bytes more than plenty enough to send a sample with a timestamp and more, back to the controller. Any switch can do that, even the cheapest ones are fast enough nowadays, and there won't be any dropped packets nor much jitter in a switched network that's being used well below the ports traffic limits.

[mansaxel]

One question, that I certainly missed to ask, is:

We understand that sampling needs to be precise in frequency and phase (ie wall clock, relative).  That is one thing. Another important part is: Is there a requirement to transmit the samples back to data storage/treatment immediately or can it be buffered? Because, with a lot of the discussed solutions, that is two different problems, and they might deserve different treatment.  They can be related, but must not be so.  If you can wait a little bit for the data to arrive, (you will have to anyway, to some extent, but if there is some additional leeway here) the system becomes a lot easier to build. And, clocking it is perhaps better done, then, as a separate system, leaving the Ethernet to be a very simple bulk transfer system.

Just a reality check, here.