Audio stream synchronization over asynchronous tranport medium

[Yansi]

Hi
The title might be a little unclear, so let me explain a bit:

Recently, I've had such thought: Let's have an audio data source, giving data at a frequency of 48000Hz. Then I will send these data over a non-synchronous transport medium - whatever it might be (ETH, USB, WiFi,...) - and receive the packetized data on the other side, where a data receiver sits a tries to replay the stream data at the same sampling frequency. So far, so good.

But whoa, the two sampling clocks on the transmitter and receiver are not synchronous, let the receiver be faster (or slower) a few ppm. After a while (long or short, depending on the reference clock deviation), there will be data samples missing or extra on the receiver side.

For example when the receiver is 1Hz faster (48001Hz) than the data transmitter, with a 1kB data queue in receive FIFOs I'll end up with a buffer underflow after thousand seconds.

How is situation like this usually handled in reality?

Thx, Yan

[AndyC_772]

If your clocks can't be synchronised, and it's an audio application, then just drop or duplicate samples. You may be surprised at how audible it isn't.

In more critical applications, then there's usually a single master clock somewhere in the system, and all other devices synchronise themselves to it. They might, for example, use a VCO to generate their local time base, and use an algorithm to adjust it up or down depending on whether the data fifo is tending to fill or empty.

[Yansi]

Thank you for quick reply.

I have thought the same, but there is a problem of "when" to detect and trigger the sample duplication or removal.

Let me have i.e. 1kB long receive FIFO for data.  If the data source is faster (slower) I will get the FIFO slowly overfilling (underfilling) and after some time an overflow (underflow) will occur.

So how can I manage the sample add/remove algorithm not to affect the signal quality and integrity? 

Or how is it managed in reality in real audio streams over network?  What protocols are used? (I know, I can google myself, but I have no clue what to search for.) Are there any network audio streaming standard protocols?

[amyk]

You can just synchronise to the incoming sample stream, like a PLL but with a much slower control loop.

[AndyC_772]

Set yourself some water marks in the FIFO, and use those to trigger adjustments.

So, for example, suppose you have a 1k buffer, you might reasonably divide it into three. If the FIFO is <1/3 full, duplicate a sample. If it's more than 2>3 full, then drop one, and if it's between 1/3 and 2/3 full, then that's your 'ideal' condition.

That's a simple method. You may need something more elaborate if you're trying to synchronise devices more closely, like, for example, in a surround sound system, or multi-room audio, where the output from multiple devices will be heard simultaneously. Then you'll need some sort of control loop to keep the FIFO levels in the different devices the same, on average, so that the pure delay between audio becoming available and being sent to the DAC is the same for each speaker.

It can be quite a tricky problem, not least because the time constants involved are long, so you can spend a lot of time waiting to see what happens to your buffer levels. It can be tricky to spot the dynamics of how the system is behaving. It's also important to include enough damping that the system is stable, ie. each device doesn't constantly oscillate either side of the correct data rate.

[Yansi]

That might be possible to do. But is that the right solution how to do it? What mechanisms are used in real application? Do they also drop/duplicate samples?

amyk: What you have described is not asynchronous transport. That would be "adaptive". I have nothing to force my local crystal osc to run faster or slower to adapt it for the data source. I'm just wondering how the asynchronous tranfers are usually done... Doesn't matter if USB or ETH...  just principles. If I have two free runing clock sources one for the data source (transmitter) and the data sink (receiver), how the hell I can solve the data over/under-run situation.

Let's assume I want to build a device, capable of connecting to an audio stream source (like a web radio). The streaming source provides data with a rate of A, my device can play the data with a rate of A+-100ppm. (because the clock systems are asynchronous and never the same frequency).

[Chris C]

Here's the approach I used to get this working over wifi.

Several times per second, the server sends out a separate status packet saying which sample/position it's currently playing.  Each client, upon receipt of these packets, compares the server's position against its own.

If server position is less than the client position, then the client slows down playback.  Just a bit though, a mere fraction of a percent; enough to accommodate any real clock mismatches plus a little bit more.  This is because packet delivery time over wifi is not guaranteed.  If latency increases, the clients will receive a server position that's lower than it should be because the packet was delayed, and we don't want the clients to respond strongly every time that happens.

However, if the server position is greater than the client position, it's a different story.  We know this is due to either a clock mismatch, or a DROP in latency.  In either case, the client can safely speed up playback to match the server position; right up to the limit of being noticeable to the user as a change in playback rate, or less if the mismatch isn't too severe.

This successfully handles short-term variations in latency.  They can fluctuate rapidly all over the place, even with packets sporadically being delayed over 250ms, and the system keeps on working perfectly.  But that wasn't quite sufficient.  From Wikipedia:

"The Haas Effect (or Precedence Effect) is a principle first set forth in 1949 by Helmut Haas, which established that we humans localize sounds by identifying the difference in arrival time between our two ears. The same sound arriving within 25 ms to 35 ms of itself will be suppressed and not be heard as an echo."

If we're playing the same music in several rooms while walking around the house, we don't want it to sound like we're in an echo chamber!  And despite what Wikipedia says, I personally find it necessary to keep the music synchronized within 12ms or less to avoid this effect.  To achieve that, I also had to account for long-term latency fluctuations; and even the baseline, best-case minimum latency.

Each time a client receives a status packet, it sends a response back to the server.  The server measures the time between sending the status packet and getting a response, which it divides by half.  This provides an estimate of one-way latency, and it is included in the next status packet sent out to the clients.

The clients maintain their own latency estimate, which is derived from the server's.  If the server reports a higher estimate than the client currently has, the client increases its estimate slowly.  If the server reports a lower estimate, the client decreases its estimate quickly.

And finally, each time the client receives a status packet from the server, the client adds its latency estimate to the server's position, to get an estimate of the server's true position.  That, in turn, is used for the adjustments in playback rate as described above.

I'm not sure if it's quite up to the technical standard of some other systems, which I initially tried mimicking based on whatever info I could scrounge up, and failed at.  But the method I came up with works well regardless.  Despite some spotty wifi, interference from neighbors, and transmission paths that are regularly walked through, I have not been able to detect any sync issues or echo in almost a year of frequent use.

This is all done on PCs, with a standard media player like MPlayer or VLC handling the actual playback.  My software simply controls the player, including reading the player's current position, exchanging synchronization info between server/clients, and adjusting playback rates.

On a resource-limited MCU, you can drop or duplicate individual audio samples, but do this too often and it perceptibly alters the pitch of the audio; so you're limited in how fast you can correct a sync error.  If you have more CPU power, you can do it the way MPlayer or VLC do it, with decimation.  You can Google the term, but a quick description:  This means picking out a segment of audio, that begins and ends at a zero-crossing of amplitude.  The segment is then dropped or duplicated, and a blend performed at its edges.  It has the advantage of being able to achieve greater changes in speed, without any change of pitch.

[Yansi]

Thanks Chris for your detailed reply. However I find it a little bit confusing in relation with what I am trying to ask. You wrote "client slows down playback" - what do you mean with that? Variable sample rate: Software resampling or finetuning local clock reference? Describe please exactly what do you mean by "slows down playback".

Just note, that I mean realtime audio streaming, not playback (from a known record / file) through stream - that would be very easy to solve via flow control to the data server.

For example I was told, that the simpliest protocol for audio stream that I can use is stream through HTTP. Just to send the header (with content-length unspecified) and then puke the data I have available. However the http stream does not have any flow control, so the data spewed are just spewed at the rate of a local sample clock. THEN I must solve the synchronisation of the client (receiver of the stream) which has its own and independent sample clock.

The last paragraph you wrote seems to be the most interesting. But the exact use of this to solve the situation will require more of my thought. O0 I have used already a similar (same?) trick for audio pitch shifting on a circular buffer, but I'm not sure if this won't affect the audio quality too much.

[sunnyhighway]

Timestamp the audio packets. Both sender and receiver have their own clock which are kept in sync with a common time source. Also use a buffer that takes care of the offsets created by line delay and clock skew.

[AndyC_772]

As you've correctly identified, there is a fundamental problem here, namely, the clock which ultimately governs the rate at which data is provided, is not running at quite the same speed as the one which controls the rate at which that data is consumed. It's an absolutely classic communications problem.

The very best solution, if you can do it, is to use a common clock for both. In this case, you might have a local clock running your DAC, and have this be the clock that triggers 'pull' requests for more data from the server. The server would have no sense of how fast audio was playing at all, it would simply provide blocks of data as and when required.

If the server provides data at a fixed rate, though, then your player has to be adaptive. It simply must consume audio at a rate which, on average, is exactly the same as the rate it's provided by the server.

Exactly how that's achieved may depend on your hardware architecture. If you can use one, a VCXO (voltage-controlled crystal oscillator) is a very good option, because you get a stable clock that runs continuously without interruption or glitches, and a control that you can use to tweak its frequency up and down in order to (say) keep a FIFO level within bounds.

If you're running on hardware without a VCXO, or a similarly adjustable clock, then you're back into the realm of dropping or duplicating samples.

[Yansi]

sunnyhighway: Just nooooooo. As was already pointed out, http stream has exactly 0 control of data flow. It is simply not possible to do so, unless using more sophisticated protocol. But it is still not solving anything, if the source media is not known before.

If you connect an ADC on one side of the planet with a DAC on the other side of the planet via ethernet, there's no such thing as a common clock available. Each side of the connection has its own clock, with own jitter and stability. And I'd like to find a common solution for this clocking problem.

From what I have found on the web, adding and discarding samples is very audible. (No good for me).  Seems I need an asynchronous resampler.

Then I have found this:  Some thick math stuff maybe on the right way for my goal: http://www.dspconcepts.com/sites/default/files/white-papers/Beckmann%20Stilson%20AES2005_ASRC.pdf

//No VCXO, just any uncontrollable clock source. Don't even think about DDS or PLL. That's not a solution for a man. 

[marshallh]

The way PCIE, usb3, sata et al solved this problem was to use an elastic buffer, and send slightly more symbols than you have data.

Ex: if you have 512 bytes audio data, send a 520 byte packet with dummy filler. To match clocks have the receiver selectively skip clocks on the dummy data to bring it into alignment.

[Yansi]

Exactly, that is what I was looking for. Now I will embed myself into a study how to implement that thing into my system and hope enough power will be left for processing. Thank you for the tip.

[sunnyhighway]

sunnyhighway: Just nooooooo. As was already pointed out, http stream has exactly 0 control of data flow. It is simply not possible to do so, unless using more sophisticated protocol.

In my proposal you can use just about any protocol that sends data in packets. Take your pick, there are plenty to choose from.
The packets don't even have to arrive in sequence because they are timestamped. The recipient can reorder them to the correct sequence.

If you connect an ADC on one side of the planet with a DAC on the other side of the planet via ethernet, there's no such thing as a common clock available. Each side of the connection has its own clock, with own jitter and stability. And I'd like to find a common solution for this clocking problem.

The internet is full of time servers that can give you the exact time to sync your local clock with.
PC's for example do it all the time.

From what I have found on the web, adding and discarding samples is very audible. (No good for me).  Seems I need an asynchronous resampler.

That is where sliding buffers comes in handy. On both ends of the line that is. If the recipient does not have an the audio packet (let's say) 10 seconds before it's going to blast out of the speakers, there is plenty of time to ask for that specific packet again.

Then I have found this:  Some thick math stuff maybe on the right way for my goal: http://www.dspconcepts.com/sites/default/files/white-papers/Beckmann%20Stilson%20AES2005_ASRC.pdf

//No VCXO, just any uncontrollable clock source. Don't even think about DDS or PLL. That's not a solution for a man. 

Yes, my proposal is a form of DDS. You could have mentioned in your first post that DDS is not an option. Would have saved me the trouble.

[Yansi]

Protocol was already picked, example mentioned above and noted of zero flow control possibility.

"The internet" is the problem as a whole, as with what will you sync your time, if the device can't have a connection to the outside world? 

Having 10 second long buffers doesn't solve clock syncing and solutions for lossy packet transmissions weren't in the above scope. HTTP works over TCP that naturally solves packet loss. Sorry for not mentioning the protocol in my first post - but still I don't see how the clock syncing problem is related to what protocol is used. The problem will be there regardless of what transport medium will be used.

Sliding buffers - don't know exactly what you mean by that - seems like circular to me. ?  Buffer of any length still won't solve the clock deviation issue. Asynchronous resampling probably will (hope so).

In the first post there was an introduction to the problematic of having two asy. clock sources. If I would have a DDS at one end, I probably wouldn't ask how to sync. And as long as my knowledge goes, no multimedia device uses DDS to compensate for this. Still wouldn't help if multiple (3 or more) clock systems would meet at one point.

Sorry for possible missunderstandings. I wasn't looking for an exact solution, it was more of a thought. I know it is a rather complex stuff. But one need to start somewhere to better understand the problematic. And the more one understands the stuff, the more complex and precise questions can he ask 

[Marco]

You can resample with a fixed DAC clock and "slowly" change the resampling factor based on buffer occupancy or you can can use a VCO to generate the DAC clock and "slowly" change it's frequency based on buffer occupancy, it's pretty much equivalent.

If you're doing any non trivial processing on the audio the resampling is essentially free though, so the VCO approach doesn't make a whole lot of sense.

[Stupid Beard]

I don't know if this is useful or not, quite likely it isn't, but on Linux there is PulseAudio. Among it's myriad features is one that allows you to setup an audio sink over the network. It might be worth having a look at  their code to see how they handle it?

[Howardlong]

Yansi

Could you elucidate (a) what you are trying to achieve at a high level and (b) what constraints have been placed?

There have been a good few suggestions but the requirement is still a bit vague to me if you don't mind me saying, and I mean that with all due respect to you of course!

For example, is this audio? Full duplex? Half duplex? Broadcast? Over what distance? What latency? What are the key acceptable parameters of operation? Are there any future requirements to bear in mind?

You mention Ethernet/USB/WiFi but these are very different transport mechanisms. If you're allowed to use whatever you choose to make the implementation, then what other requirements are there? Is this point to point or broadcast for example?

[lincoln]

You might look into a protocal called IEEE 1588 aka "PTP"  It is designed to transfer frequency or time and frequency over packet networks. It is used extensively in cell phone and power infrastructure, as will as physics experiments. I beleave there is a profile sponsored by AES

Link

[sunnyhighway]

Having 10 second long buffers doesn't solve clock syncing and solutions for lossy packet transmissions weren't in the above scope. HTTP works over TCP that naturally solves packet loss. Sorry for not mentioning the protocol in my first post - but still I don't see how the clock syncing problem is related to what protocol is used. The problem will be there regardless of what transport medium will be used.

The clock syncing problem is indeed NOT related to what protocol is used.

In the first post there was an introduction to the problematic of having two asy. clock sources. If I would have a DDS at one end, I probably wouldn't ask how to sync. And as long as my knowledge goes, no multimedia device uses DDS to compensate for this. Still wouldn't help if multiple (3 or more) clock systems would meet at one point.

For transmissions (regardless of medium and protocol) with a time-stamp to work, you need is an accurate time-source.

There are several accurate time-sources to sync the local clock with:
 - Public NTP servers
 - RDS time-signal on FM radio stations.
 - GPS
 - Dedicated longwave and shortwave transmitters (used by RCC's)

[Chris C]

You wrote "client slows down playback" - what do you mean with that? Variable sample rate: Software resampling or finetuning local clock reference? Describe please exactly what do you mean by "slows down playback".

Both MPlayer and VLC can be queried for their current playback position, in floating-point seconds.  And both accept a playback rate, as a floating-point percentage.  100% being "normal", apparently according to the PC's own audio sample clock, which of course runs a few ppm differently from the other PCs.  <100% is slower, >100% is faster.  When the percentage is anything other than 100%, MPlayer/VLC applies decimation to temporally compress or stretch the audio as needed.  I briefly touched on that in the last paragraph.

Everything else I described is done in my software.  The server queries the playback position and sends it to the clients.  The challenge was getting the clients to accurately correct that position to account for network latency.   And with the corrected position, the clients can query their own playback position, and generate a playback rate percentage to eliminate any error.  So long as a client's sync error is <12ms, no correction is applied, it plays at 100% rate - even if it could determine that, due to ppm differences in clock rates, continuously playing at 99.997% would be more correct.  I do this to avoid continuous decimation of the audio, which uses more CPU, and can introduce audible artifacts in some conditions.  Only when the error reaches >=12ms does it use a rate other than 100%, and it continues the correction until the error crosses zero.  So other than the initial corrections when playback is first started, corrections tend to occur in small bursts, typically every few tens of seconds.

My software uses standard TCP/IP for the synchronization packets.  I disable that protocol's Nagle algorithm, as that would add some delay.  But it's not actually necessary as my algorithm accurately corrects for it anyway.  As well as any delays caused by retransmissions.

[sunnyhighway] is right, the clock syncing problem is not related to what protocol is used for transmission of the actual audio data.  And if you can use a radio-based and latency-free common clock source like RDS/GPS/longwave/shortwave, by all means do it - it makes things SOOO much easier!  Though NTP I'm not too sure about, I don't know if it accurately accounts for for latency in receiving the response from the NTP server.

[vk6zgo]

My first thought was: "Save it in a store at the remote site---read it out again using the clock at that site."
I know it works with analog & digital video.

Maybe not,though,if the store needs incoming clock info.

If the store can recognise the start & finish of each packet,maybe it can store them without needing any clock reference.

I can't remember enough------I learnt about this stuff when it was all done in hardware, either with discrete logic or special ICs.