Something wrong with the server?

[firewalker]

The past two weeks is really hard to listen/download the episodes.

wget http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
--2011-10-02 12:03:09--  http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
Resolving www.theamphour.com... 74.220.207.157
Connecting to www.theamphour.com|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://theamphour.com/wp-content/uploads/TheAmpHour-62-NarquoisNerdNescience.mp3 [following]
--2011-10-02 12:03:10--  http://theamphour.com/wp-content/uploads/TheAmpHour-62-NarquoisNerdNescience.mp3
Resolving theamphour.com... 74.220.207.157
Connecting to theamphour.com|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30400604 (29M) [audio/mpeg]
Saving to: `TheAmpHour-62-NarquoisNerdNescience.mp3'

 6% [======>                                                                                                      ] 1,955,233   7.60K/s  eta 64m 37s

I will fail many times. I will take up to an hour to download the episode.

Alexander.

[McMonster]

All fine here, up to 1,7 MB/s, it must be a problem with your connection.

[firewalker]

Everything else works just fine. Maybe an ISP issue...

Alexander.

[firewalker]

This issue is really pissing me off!

[RoelA]

Code: [Select]

roel@vps:~$ wget http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
--12:20:18--  http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
           => `TheAmpHour-62-NarquoisNerdNescience.mp3'
Resolving [url=http://www.theamphour.com]www.theamphour.com[/url]... 74.220.207.157
Connecting to [url=http://www.theamphour.com]www.theamphour.com[/url]|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://theamphour.com/wp-content/uploads/TheAmpHour-62-NarquoisNerdNescience.mp3 [following]
--12:20:20--  http://theamphour.com/wp-content/uploads/TheAmpHour-62-NarquoisNerdNescience.mp3
           => `TheAmpHour-62-NarquoisNerdNescience.mp3'
Resolving theamphour.com... 74.220.207.157
Connecting to theamphour.com|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30,400,604 (29M) [audio/mpeg]

100%[============================================================================================================================>] 30,400,604     1.77M/s    ETA 00:00

12:20:37 (1.65 MB/s) - `TheAmpHour-62-NarquoisNerdNescience.mp3' saved [30400604/30400604]

Can you do a traceroute to theamphour.com?

[EEVblog]

I don't see any problem, does anyone else?

Dave.

[amspire]

It took me about 3 days to get episodes 61 and 62 on iTunes because the download would crawl and then Itunes would abort. This was starting last Friday. Tried downloading ep61 directly from the Amp Hours website and that kept aborting too. I didn't run any diagnostics.

I thought iTunes downloads actually came from iTunes, but there are about 20 other podcasts that have no problem - just the Amp Hour.

I am getting episode 63 right now and it is going a bit faster - 80 minutes to get 30MBytes, while another 24Mbyte podcast downloaded while I was typing this.

I am at Oberon, NSW, AU, and my ISP connects via the Telstra network.

Just before posting this, the iTunes download of #63 crashed at 3.9Mbytes error -3259 which means "network timeout" I gather.

Richard

[amspire]

I just downloaded Ep63 to a server in Dallas - 17 seconds, so it looks like it is probably not the hosting site.

A direct download to home is 7Kbytes/sec which is slow but an improvement.  So something on the network between your host and Telstra.

If I download 2 episodes directly at the same time, both go at 7KBytes/sec, so there is some definite throttling happening on a router somewhere.

Richard.

[firewalker]

traceroute

Code: [Select]

traceroute to theamphour.com (74.220.207.157), 30 hops max, 40 byte packets
 1   (192.168.2.1)  1.212 ms  1.144 ms  1.068 ms
 2  80.106.108.3 (80.106.108.3)  23.771 ms  9.838 ms  11.031 ms
 3  80.106.229.49 (80.106.229.49)  9.565 ms  9.265 ms  9.008 ms
 4  lari-gsra-lari7609b-1.backbone.otenet.net (79.128.231.9)  9.827 ms  9.561 ms  10.003 ms
 5  athe-crsa-lari-gsra-1.backbone.otenet.net (79.128.224.201)  20.627 ms  18.184 ms  19.438 ms
 6  ten0-0-0-0-crs02.ath.OTEGlobe.net (62.75.3.13)  101.346 ms  65.698 ms  63.834 ms
 7  62.75.4.138 (62.75.4.138)  68.858 ms  64.428 ms  64.405 ms
 8  62.157.251.41 (62.157.251.41)  66.719 ms  66.547 ms  67.075 ms
 9  194.25.6.46 (194.25.6.46)  65.911 ms  66.834 ms  66.730 ms
10  194.25.211.134 (194.25.211.134)  71.760 ms  70.547 ms  70.393 ms
11  xe-4-3-0.nyc30.ip4.tinet.net (89.149.182.114)  146.752 ms xe-9-1-0.nyc30.ip4.tinet.net (213.200.81.126)  147.053 ms  148.122 ms
12  ace-data-centers-gw.ip4.tinet.net (173.241.129.42)  154.457 ms  153.366 ms  150.470 ms
13  tg1-1.ar02.prov.acedc.net (199.58.196.41)  211.824 ms  208.905 ms  208.808 ms
14  port99.ar02.prov.bluehost.com (199.58.199.118)  208.054 ms  206.548 ms  208.254 ms
15  host157.hostmonster.com (74.220.207.157)  204.332 ms  204.079 ms  206.084 ms
Alexander.

[amspire]

The only two IPs in common with me are the last two:

14  port99.ar02.prov.bluehost.com (199.58.199.118)  208.054 ms  206.548 ms  208.254 ms
15  host157.hostmonster.com (74.220.207.157)  204.332 ms  204.079 ms  206.084 ms

Since the hostgator is in California, and I got a 20Mbit/sec download to Dallas, that makes it look like the bluehost router (199.58.199.118) is the one throttling.

Bluehost is one of Hostgators biggest competitors. I wonder why Hostgator goes through a Bluehost router? So my internet is going from New York to a Bluehost router in Utah and then to Hostgator in California.

If anyone has good download speed for AmpHours podcasts, can they do a trace (run "tracert 74.220.207.157" in a cmd window on Windows) and see if they go through IP address 199.58.199.118?

Richard.

[amspire]

My last 4 IPs in the trace route are:

14 154.54.47.174   198ms  198ms  196ms  TTL:242  (te7-3.ccr01.slc01.atlas.cogentco.com bogus rDNS: host not found [authoritative])
15 38.104.174.18   248ms  248ms  300ms  TTL:232  (No rDNS)
16 199.58.199.118  293ms  253ms  247ms  TTL:231  (port99.ar02.prov.bluehost.com bogus rDNS: host not found [authoritative])
17 74.220.207.157  254ms  255ms  254ms  TTL: 40  (host157.hostmonster.com ok)

The last two are Cogentco.  I just did media downloads from the Cogentco server (38.100.128.10) and it was fast.

Still looking like the Bluehost router in Provo, Utah may be throttling mp3's to 10Kbytes/sec, but in doing so, it often gets below 5KBytes/sec or times out.

Richard

[baljemmett]

If anyone has good download speed for AmpHours podcasts, can they do a trace (run "tracert 74.220.207.157" in a cmd window on Windows) and see if they go through IP address 199.58.199.118?

This is from one of my colo servers:

baljemmett@debvps:~$ wget http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
[...]
Downloaded: 1 files, 29M in 18s (1.65 MB/s)
baljemmett@debvps:~$ traceroute www.theamphour.com
traceroute to www.theamphour.com (74.220.207.157), 30 hops max, 40 byte packets
 1  213-229-91-195.static.as29550.net (213.229.91.195)  2.355 ms  2.295 ms  3.100 ms
 2  xe-1-2-0.lon21.ip4.tinet.net (77.67.67.137)  1.897 ms  2.111 ms  2.081 ms
 3  xe-10-1-0.nyc30.ip4.tinet.net (89.149.184.129)  74.155 ms xe-0-3-0.nyc30.ip4.tinet.net (89.149.187.142)  75.297 ms  75.314 ms
 4  ace-data-centers-gw.ip4.tinet.net (173.241.129.42)  74.063 ms  71.515 ms  71.548 ms
 5  tg1-1.ar02.prov.acedc.net (199.58.196.41)  149.066 ms  150.527 ms  149.036 ms
 6  port99.ar02.prov.bluehost.com (199.58.199.118)  149.559 ms  151.301 ms  151.334 ms
 7  host157.hostmonster.com (74.220.207.157)  149.290 ms  149.270 ms  149.243 ms

From the office, I get good download speeds but can't do a traceroute thanks to a stupid piece of router configuration that's outside of my control and gets on my wick on a regular basis:

baljemmett@office:~$ wget http://www.theamphour.com/podpress_trac/web/1054/0/TheAmpHour-62-NarquoisNerdNescience.mp3
[...]
09:40:57 (358.80 KB/s) - `TheAmpHour-62-NarquoisNerdNescience.mp3' saved [30400604/30400604]

Whereas from home, things are a bit slower (unusual, as I can usually get ten times this sort of speed):

C>wget http://www.theamphour.com/podpress_trac/web/1064/0/TheAmpHour-63-PickAndPlacePalillogy.mp3
[...]
100%[======================================>] 31,598,709  96.1K/s   in 5m 25s

2011-10-05 14:20:23 (95.0 KB/s) - `TheAmpHour-63-PickAndPlacePalillogy.mp3' saved [31598709/31598709]

C>tracert www.theamphour.com

Tracing route to theamphour.com [74.220.207.157]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  whp-router.microwavepizza.co.uk [192.168.2.1]
  2    24 ms    23 ms    24 ms  losubs.subs.dsl1.th-lon.zen.net.uk [62.3.84.17]
  3    24 ms    24 ms    23 ms  ae0-112.cr1.th-lon.zen.net.uk [62.3.84.177]
  4    31 ms    35 ms    31 ms  ge-3-0-0-0.cr2.kp-leeds.zen.net.uk [62.3.80.77]
  5    32 ms    32 ms    32 ms  so-0-0-0-0.cr1.ha-sthp.zen.net.uk [62.3.80.82]
  6   110 ms   111 ms   111 ms  so-0-0-0-0.cr1.tlx-nyc.zen.net.uk [62.3.80.86]
  7   184 ms   117 ms   131 ms  tg1-2.br01.nycm.bluehost.com [198.32.160.49]
  8   170 ms   169 ms   170 ms  tg1-1.ar02.prov.acedc.net [199.58.196.41]
  9   170 ms   170 ms   170 ms  port99.ar02.prov.bluehost.com [199.58.199.118]
 10   170 ms   169 ms   169 ms  theamphour.com [74.220.207.157]

Trace complete.

As for the bluehost router -- it's possible that one host keeps its machines in a datacentre or on connectivity owned by the other.  I know the colo facility where my servers live also houses several end-user hosting companies, including direct competitors to their own...

[amspire]

Your results seem to blow the Bluehost theory.

I wouldn't complain at all if I could download an Amp Hour podcast in 5 minutes or even 50 minutes.

I will do some tests tomorrow to see if my burst speed is high, but the packet handshake is just timing out.

Richard.

[amspire]

Did some tests this morning on the very slow mp3 download that some people are getting from "theamphour.com".

First up, I got the host wrong previously - theamphour.com is at Hostmonster at Provo Utah - the same company as Bluehost which explains when they go through the Bluehost router.

I am getting regular packet loss even for normal HTTP traffic. The loss extends to other websites on the same server, and websites on an adjacent Hostmonster server, which means that the loss is either not due to the server, or it is due to a common setting across the Hostmonster servers.

It does not appear there is any systematic throttling going on, just dropped packets. When packets are not being dropped, I can see speeds of over 200KBytes/sec. Each time there is a dropped packet, there can be a delay in the Internet transmission of around 100-500mSec and it is these accumulated delays causing the slow connection. The longer a bog download, the more out-of-sequence the packets get and so the communication slows and eventually just crashes.

I have tested the connection out from home on both Windows and Linux systems and they both have identical problems, so it is not a problem on one PC here. Downloads from most other servers (not at Bluehost) are fine.

It seems that many other connections to the same server from different locations have no problem at all. It would seem that the problem has to be in either the Bluehost router or the Hostgator server, but if so, then some connections are always slow and others are always fast.

Hostmonster and Bluehost have been implementing CPU throttling in their servers, and for some customers anyway, the CPanel has an Icon for CPUThrottling and they can check the status of the throttling.

Perhaps Dave or Chris could take a look.

One possibility is that that the server's capacity to cope with dropped packets is limited, so if you don't have drop packets, the communication flies. If you do, the communication deteriorates until it crashes. I would love to have a better explanation. I have to go through 16 routers to reach The Amp Hour server so I guess there is more chance of me loosing some packets initially then someone connecting through 4 routers.

Since I cannot tell exactly where packets are getting dropped, that is about as far as I can get right now.

Richard.

[baljemmett]

I am getting regular packet loss even for normal HTTP traffic. The loss extends to other websites on the same server, and websites on an adjacent Hostmonster server, which means that the loss is either not due to the server, or it is due to a common setting across the Hostmonster servers.

It does not appear there is any systematic throttling going on, just dropped packets. When packets are not being dropped, I can see speeds of over 200KBytes/sec. Each time there is a dropped packet, there can be a delay in the Internet transmission of around 100-500mSec and it is these accumulated delays causing the slow connection. The longer a bog download, the more out-of-sequence the packets get and so the communication slows and eventually just crashes.

Hmm.  Is there any pattern to the dropped / delayed packets, for instance is it only the biggest ones that get interfered with whilst the smaller ones (the TCP handshake, small web pages) get through unmolested?

It sounds vaguely like there might be an issue with path MTU discovery / packet fragmentation between you and Hostmonster; not certain, as it seems to work for you sometimes and not others, but if it's only larger downloads hitting trouble it might be worth investigating tweaking your MTU or applying some MSS clamping to see if that helps.  In an ideal world this sort of thing isn't a problem, but if some router along the way is failing to pass on the relevant ICMP 'fragmentation required' messages you can get this kind of issue.

[amspire]

There is not really a pattern.  There is a lost packet for every 1 to 3 full sized packets received - usually just one full sized packet.  Small packets like ACKs are more likely to get through but they get lost too.  UsuallyThis is something that has only happened in the last week or so. It was totally fine before then.

I may try slowing down my connection speed to HostMonster to see if that improves the connection, but I do not want to put more time into this.

It is not urgent, as I am able to download the podcasts to Dallas in a few seconds, and then download them from there in a few minutes.  Trying to get them from HostMonster takes about 4 hours if I am very lucky.

If HostMonster are throttling the CPU speed on their servers, then it could just be that the Amp Hour Show is increasing in popularity, and  because of some characteristic of my connection, I could be amongst the first to suffer.

So I hope Chris or Dave will actually check to see if they have the CPU Throttle icon in Cpanel.

Richard.

[ChrisGammell]

It has been getting throttled lately, quite a bit.  Today (Oct 11th) was particularly bad, to the point that I was getting <1kbps, same as some others. Ugh.

I'm an IT dummy, do you guys have suggestions on what this means in a larger context? Does that mean we'll need to switch to a different hosting platform?

I also saw in the logs that there were errors like "Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace., referer: http://www.minshenglt.com/"

Normally I wouldn't stress about it too much. But looking at our listening stats, the numbers of people listening dropped off significantly (>1000 friggin listeners!!!) right when we started seeing problems.

Any and all advice is appreciated.

[amspire]

Chris, in the CPanel for the site, do you have an icon called something like "CPU Throttling"?

I haven't heard of this before with other CPanel based providers, and so it might be something that BlueHost/HostMonster do.

Anyway, in regards to the particular error, there is some kind of bad loop going on. You definitely don't need to change 'LimitInternalRecursion'. Something is wrong that needs fixing I think.

Do you have any code or .htaccess file settings to prevent "hotlinking" (this is where you want to prevent other websites directly embedding your content on their sites.) If the code is bad, it can cause looping.

Also try and find this error in the sites error logs and access logs (if you have both, you can match them up by time).  Try and see exactly the URL being accessed on your site when the error occurred. Perhaps  something that is being requested is missing, or even something on the page that that is trying to access something on the page etc in an infinite loop.

This error you reported will have a time stamp, and there should be a matching timestamp entry in the access.log specifying the requested URL.

Richard

[EEVblog]

The latest episode is now hosted on the EEVblog dedicated server and is infinitely faster!
We'll do a trial of hosting the MP3 on this server from now on. It is doesn't pull down the forum response time then we'll keep it in place.

Dave.

[amspire]

Big improvement.

Way faster. No crashing.       

Thanks Dave.

[Zad]

It looks like it may be slowing down again. It started off at >60kB/s and is now just under 30kB/s.

Tell a lie, it has just jumped back to 60 again. Odd.

[firewalker]

wget http://www.theamphour.com/podpress_trac/web/1092/0/TheAmpHour-65-DavesDingoDystocia.mp3
--2011-10-18 14:44:34--  http://www.theamphour.com/podpress_trac/web/1092/0/TheAmpHour-65-DavesDingoDystocia.mp3
Resolving www.theamphour.com... 74.220.207.157
Connecting to www.theamphour.com|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://theamphour.com/wp-content/uploads/TheAmpHour-65-DavesDingoDystocia.mp3 [following]
--2011-10-18 14:44:35--  http://theamphour.com/wp-content/uploads/TheAmpHour-65-DavesDingoDystocia.mp3
Resolving theamphour.com... 74.220.207.157
Connecting to theamphour.com|74.220.207.157|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31200539 (30M) [audio/mpeg]
Saving to: `TheAmpHour-65-DavesDingoDystocia.mp3'

 4% [====>                                                            ] 1,546,273    211K/s  eta 2m 34s 

Everything ok now!

Alexander.

[Rasz]

http://www.theamphour.com/about/

404 Error File Not Found
 The page you are looking for might have been removed,
had its name changed, or is temporarily unavailable.
If you typed in the URL, please check the spelling
You can try to click the Back button and try another link
 

Web Hosting provided by HostMonster.Com

Seems teh blod is dead? only homepage works

[AntiProtonBoy]

yep, getting the same errors.

[EEwannabe]

I am also receiving the same error when trying to use the comment shortcut.
-Patrick