ISP Bandwidth Sawtooth Oscillation – Buffer Bloat!

So I just got my new 8M/8M VDSL2 office internet installed in the lab (see video).
And speedtest.net looked great of course, and everything seemed to work a treat, my connection seemed quick, and a quick youtube speed test showed it to be much faster than my old ADSL2+ connection as you’d expect.
But yesterday I tried to upload a new video to youtube and it didn’t seem to be as fast as expected. So I looked at my upload bandwidth graph, and noticed something strange:

The upload bandwidth speed (green line) looked like it was oscillating! A classic sawtooth waveform in fact. Linearly ramping up from a low of about 2Mbps, to about 6.5Mbps peak. But as soon as it reached the peak it instantly shot back down and started to ramp up again. Naturally if you take the average of that waveform you are going to get a lot less than the peak value. So my 8Mbps upload connection was looking and performing very ordinary. And this happened with both youtube uploads and FTP uploads to my server.

I hadn’t seem this before, my Telstra 100M/2.4Mbps cable connection at home always had a flat line, or a very small amount of ripple on the flat line (yes, I of course think of these graphs as electrical waveforms, I can’t help myself). And my old ADSL2+ connection in the lab also had a similar line, although being physical line limited instead of software limited like the cable modem, the line wasn’t as flat. But this oscillation thing was new.

So onto twitter I went for a collective nerd decision as to what was causing this, and everyone had their pet theory – Buffer Bloat!, bandwidth capping!, dropped packets!,  QoS bandwidth limiting effect on TCP!, typical TCP congestion control!, higher priority voice data bursts!

Well, a google search turned up nothing, couldn’t find a single similar image of this linear ramping.

So I contacted my ISP Building Connect (Oxygen Networks) and the head tech guy Paul who installed my system wasn’t entirely sure what was going on either at first. But he called me back and we experimented with a few things, like lowering my bandwidth plan, but that didn’t work. But he could see the exact same waveform on his end as on my end, so that was good. He even started to suspect it might be something at my end, Windows related, not backing off properly on bandwidth as it should when peak limit is reached.

But then he then tweaked the buffer size values on the access router at their end, and sure enough, we hit upon a magic value that made the line go completely flat:

There was still the ramp up at the start as expected, but it was flat after that. with the occasional small glitch as you’d expect. This was for an FTP upload to my server in the US. Youtube upload also showed the same flat response. Woo-Hoo! problem fixed in record time.

So it was some sort of buffer issue on the ISP access router, and maybe in combination with a buffer or other setting in my Windows 7 O/S. There seems to be a optimum total system buffer setting that causes this issue. If it’s not set correctly then the ISP router dumps packets galore and presumably reflushes the buffer, hence the linear ramp up in bandwidth again until the buffer hits maximum and the whole process starts again. Bingo, you get a classic sawtooth waveform.
So this seems to be the infamous “buffer bloat” I hear Cringely talking about all the time.

I haven’t looked into all the mechanics of this, but it certainly is a nasty little problem, and had I not gone with a small ISP were I had access to the head technical person who could tweak my connection for me, I might have stuck with this problem. Thumbs up for small ISP’s that provide personal service.