High / inconsistent network latency when using DHCP, fine with static IP address

[Delta]

I have a Linux-based media player.  It is on a 100Mbit/s wired ethernet connection to my broadband modem / router.

When I give it a static IP address, the ping times are around 6ms, and never above 10.  However when it is set to use DHCP, the ping times are all around 1000ms(!).  Transfer speeds are not affected, it just seems to be the initial latency.  This is 100% repeatable; switch back to static and all is well, back to DHCP and problem returns.

To make sense of all this, router is .1 media player is .4 laptop is .12
I am using SSH into the media player to run the ping.
The laptop is WiFi and DHCP to the router, but this link is fine.

EDIT- Pinging from media player to router now alternates between good and bad!

To try to summarise this very messy post:

Individually
Laptop to media player = ~1000ms consistent
Media player to laptop = ~1000ms consistent

Both running together
Laptop to media player = <10ms consistent
Media player to laptop = ~1000ms consistent

Then when stopping the laptop > media player ping, I see this from the media player pinging the laptop!

Code: [Select]

64 bytes from 192.168.0.12: seq=148 ttl=64 time=1004.136 ms
64 bytes from 192.168.0.12: seq=149 ttl=64 time=7.040 ms
64 bytes from 192.168.0.12: seq=150 ttl=64 time=1004.956 ms
64 bytes from 192.168.0.12: seq=151 ttl=64 time=8.162 ms
64 bytes from 192.168.0.12: seq=152 ttl=64 time=1004.103 ms
64 bytes from 192.168.0.12: seq=153 ttl=64 time=7.062 ms
64 bytes from 192.168.0.12: seq=154 ttl=64 time=1004.069 ms
64 bytes from 192.168.0.12: seq=155 ttl=64 time=6.941 ms
and after that messing around, when the media player pings the router, I get

Code: [Select]

PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: seq=0 ttl=64 time=4.518 ms
64 bytes from 192.168.0.1: seq=1 ttl=64 time=1001.566 ms
64 bytes from 192.168.0.1: seq=2 ttl=64 time=6.021 ms
64 bytes from 192.168.0.1: seq=3 ttl=64 time=1001.276 ms
64 bytes from 192.168.0.1: seq=4 ttl=64 time=5.518 ms
64 bytes from 192.168.0.1: seq=5 ttl=64 time=190.415 ms
64 bytes from 192.168.0.1: seq=6 ttl=64 time=1001.343 ms
64 bytes from 192.168.0.1: seq=7 ttl=64 time=6.342 ms
64 bytes from 192.168.0.1: seq=8 ttl=64 time=1003.160 ms
64 bytes from 192.168.0.1: seq=9 ttl=64 time=7.344 ms
64 bytes from 192.168.0.1: seq=10 ttl=64 time=31.949 ms
64 bytes from 192.168.0.1: seq=11 ttl=64 time=872.556 ms
64 bytes from 192.168.0.1: seq=12 ttl=64 time=868.554 ms
64 bytes from 192.168.0.1: seq=13 ttl=64 time=868.334 ms
64 bytes from 192.168.0.1: seq=14 ttl=64 time=1001.691 ms
64 bytes from 192.168.0.1: seq=15 ttl=64 time=96.383 ms
64 bytes from 192.168.0.1: seq=16 ttl=64 time=1001.305 ms
64 bytes from 192.168.0.1: seq=17 ttl=64 time=5.531 ms

What on earth could cause this?  Everything is fine with a static IP address!

[SeanB]

Change your DNS provider, you most likely are using the crummy overloaded box forgotten in a corner that is your ISP DNS. Try changing DNS to either openDNS, or use the 4.4.4.4 or 8.8.8.8 off Google instead, should drop the latency down considerably.

Otherwise go to GRC.com and run the DNS benchmark and see which ones local to you are best.

https://www.grc.com/dns/benchmark.htm

[grumpydoc]

DNS shouldn't figure here, especially if the box is getting the same address via DHCP as Delta is setting manualy.

It would be worth seeing what other network traffic is around, if you are able run tcpdump on the router or Wireshark on the other ends.

I'd also be interested to see the WAN traffic from the router and the firewall settings on everything.

[rollatorwieltje]

Isolate the problem first. Get rid of the wifi and test again. Wifi can be incredibly shit when dealing with small amounts of data.

[Ampera]

The answer to why I dislike networking.

[Delta]

Sean - I can't see how DNS can have any on this, as it is all within my private LAN, and all using IP addresses... 

Rollator - I hear you about the WiFi, but 1) the WiFi between the laptop and router is fine, perfect ping times, and 2) The Media Player to router is wired, and still experiences the same issues...
UPDATE - Got rid of it and tried again - I've just done a ping from my RasPi (wired) to the MP, and got this very similar results - unless the first few replies mean anything, what is (DUP!)?

Code: [Select]

PING 192.168.0.4 (192.168.0.4) 56(84) bytes of data.
64 bytes from 192.168.0.4: icmp_seq=1 ttl=64 time=0.638 ms
64 bytes from 192.168.0.4: icmp_seq=1 ttl=64 time=24.6 ms (DUP!)
64 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=25.9 ms
64 bytes from 192.168.0.4: icmp_seq=3 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=4 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=5 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=6 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=7 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=8 ttl=64 time=999 ms
64 bytes from 192.168.0.4: icmp_seq=9 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=10 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=11 ttl=64 time=1000 ms
64 bytes from 192.168.0.4: icmp_seq=12 ttl=64 time=548 ms
64 bytes from 192.168.0.4: icmp_seq=13 ttl=64 time=1009 ms

Doc, firewall settings shouldn't really affect it, as it is fine if I give it the same static IP address that DHCP assigned it...?

[senso]

What router/modem are you using?
Have you tried rebooting it?

[Monkeh]

Sean - I can't see how DNS can have any on this, as it is all within my private LAN, and all using IP addresses... 

Rollator - I hear you about the WiFi, but 1) the WiFi between the laptop and router is fine, perfect ping times, and 2) The Media Player to router is wired, and still experiences the same issues...
UPDATE - Got rid of it and tried again - I've just done a ping from my RasPi (wired) to the MP, and got this very similar results - unless the first few replies mean anything, what is (DUP!)?

Duplicate. It got a second response to the first packet. Something very wrong there.

Are you using the same IP static as you get via DHCP?

[Delta]

Are you using the same IP static as you get via DHCP?

I have used the same and different.  I have changed the DHCP range to force it to use an address that has not been used before. (.4 has not been used statically or assigned by DHCP until now)


Right, as I type this, the MP is netcatting crap from /dev/urandom to the laptop (nc into /dev/null) at a rate of 11Mbit/s.  The RasPi is pinging the MP and getting response times of ~2ms.  It seems that if a connection is held open between the router and MP then all is well.  I think I might have to do a factory reset on the router

RasPi to MP whilst MP is streaming to laptop at 11Mbits/s

Code: [Select]

64 bytes from 192.168.0.4: icmp_seq=717 ttl=64 time=0.716 ms
64 bytes from 192.168.0.4: icmp_seq=718 ttl=64 time=0.925 ms
64 bytes from 192.168.0.4: icmp_seq=719 ttl=64 time=3.11 ms
64 bytes from 192.168.0.4: icmp_seq=720 ttl=64 time=2.20 ms
64 bytes from 192.168.0.4: icmp_seq=721 ttl=64 time=0.717 ms
64 bytes from 192.168.0.4: icmp_seq=722 ttl=64 time=1.53 ms
64 bytes from 192.168.0.4: icmp_seq=723 ttl=64 time=2.06 ms
64 bytes from 192.168.0.4: icmp_seq=724 ttl=64 time=2.37 ms
64 bytes from 192.168.0.4: icmp_seq=725 ttl=64 time=1.81 ms
64 bytes from 192.168.0.4: icmp_seq=726 ttl=64 time=1.79 ms
64 bytes from 192.168.0.4: icmp_seq=727 ttl=64 time=3.44 ms
64 bytes from 192.168.0.4: icmp_seq=728 ttl=64 time=2.80 ms
64 bytes from 192.168.0.4: icmp_seq=729 ttl=64 time=3.55 ms


[grumpydoc]

Sean - I can't see how DNS can have any on this, as it is all within my private LAN, and all using IP addresses... 

Time-outs trying to reverse lookup address can do odd things, however I did say that I didn't think DNS was the problem

Is it loosing ARP?

That DUP looks suspicious.

Doc, firewall settings shouldn't really affect it, as it is fine if I give it the same static IP address that DHCP assigned it...?

Broadly speaking no, but stateful firewalls sometimes do odd things.

The output of  ifconfig (or ipconfig /all if on Windows) would also be useful.

[Delta]

Surely once DHCP has assigned an address, it shouldn't have any influence at all on ongoing network activity...
The "bad" pings being all nearly exactly 1000ms seems suspicious too...

Doc - ifconfig output from Media Mplayer...

Code: [Select]

eth0      Link encap:Ethernet  HWaddr 00:15:C0:4A:01:81 
          inet addr:192.168.0.4  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::215:c0ff:fe3a:781%4933360/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:482879 errors:0 dropped:6 overruns:0 frame:0
          TX packets:961136 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:32687040 (31.1 MiB)  TX bytes:1362710443 (1.2 GiB)


[grumpydoc]

Surely once DHCP has assigned an address, it shouldn't have any influence at all on ongoing network activity...

Unless there is something wrong with the leases and it is trying to reacquire continuously

Doc - ifconfig output from Media Mplayer...

Nothing obvious - and on the other boxes?

Some tcpdump output would be very useful.

[hendorog]

Perhaps you have you created a route loop by accident.

What is the default gateway set to?

Also show the output of:

Code: [Select]

route -n
and

Code: [Select]

traceroute -n 192.168.0.1


[Delta]

Laptop

Code: [Select]

wlan0     Link encap:Ethernet  HWaddr 18:4d:a7:34:a6:91 
          inet addr:192.168.0.12  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2473738 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2459360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2995315922 (2.9 GB)  TX bytes:2042206924 (2.0 GB)

RasPi

Code: [Select]

eth0      Link encap:Ethernet  HWaddr b8:27:fb:07:a9:c2 
          inet addr:192.168.0.15  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::ba27:ebff:fe07:e5b1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1907950806 errors:0 dropped:1417 overruns:0 frame:0
          TX packets:2480464166 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2768426024 (2.7 GB)  TX bytes:261132587 (261.1 MB)

I've got to pop out now, but when I get a chance I'll install Wireshark...  Thanks for your help.

[Delta]

Perhaps you have you created a route loop by accident.

What is the default gateway set to?

Also show the output of:

Code: [Select]

route -n
and

Code: [Select]

traceroute -n 192.168.0.1


(On Media Player)
Gateway is set to 192.168.0.1

route -n

Code: [Select]

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

traceroute doesn't do anything...

Code: [Select]

traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 38 byte packets
 1  *  *  *
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *^C


[grumpydoc]

traceroute doesn't do anything...

Code: [Select]

traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 38 byte packets
 1  *  *  *
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *^C


So, the router certainly has a firewall enabled which eats packets on the LAN otherwise it would reply with 1 hop eg if I do this to a random host on my LAN I get

traceroute aa.bb.cc.140
traceroute to aa.bb.cc.140 (aa.bb.cc.140), 30 hops max, 60 byte packets
 1  aa.bb.cc.140 (aa.bb.cc.140)  33.558 ms  33.514 ms  33.487 ms

[hendorog]

traceroute doesn't do anything...

Code: [Select]

traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 38 byte packets
 1  *  *  *
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *^C


So, the router certainly has a firewall enabled which eats packets on the LAN otherwise it would reply with 1 hop eg if I do this to a random host on my LAN I get

traceroute aa.bb.cc.140
traceroute to aa.bb.cc.140 (aa.bb.cc.140), 30 hops max, 60 byte packets
 1  aa.bb.cc.140 (aa.bb.cc.140)  33.558 ms  33.514 ms  33.487 ms

ping and traceroute both use ICMP packets - so if traceroute doesn't work then ping doesn't either.

[grumpydoc]

ping and traceroute both use ICMP packets - so if traceroute doesn't work then ping doesn't either.

No, traceroute uses udp packets with a limited TTL, then listens for the ICMP TTL exceeded message.

If the packet or reply is swallowed then you get the sort of trace that was posted.

there is a -I option on linux traceroute which uses pings  but it doesn't use ICMP echo by default.

It is worth thinking about what needs to happen when you ping a host. The machine that you are pinging from needs to know the MAC address of the host that you are trying to reach so it first send out an ARP "who-has" request which is a broadcast packet. The remote host then replies with a unicast packet containing an ARP reply. Only then can you send out the original packet.

Anything which filters, throttles or blocks ARP can slow transmission.

[hendorog]

ping and traceroute both use ICMP packets - so if traceroute doesn't work then ping doesn't either.

No, traceroute uses udp packets with a limited TTL, then listens for the ICMP TTL exceeded message.

If the packet or reply is swallowed then you get the sort of trace that was posted.

there is a -I option on linux traceroute which uses pings  but it doesn't use ICMP echo by default.

It is worth thinking about what needs to happen when you ping a host. The machine that you are pinging from needs to know the MAC address of the host that you are trying to reach so it first send out an ARP "who-has" request which is a broadcast packet. The remote host then replies with a unicast packet containing an ARP reply. Only then can you send out the original packet.

Anything which filters, throttles or blocks ARP can slow transmission.

Oops yes you are right, thanks for the correction. 

Another thought - which explains some but not all - is that the network driver is going into low power mode - we found an issue yesterday with slow pings on a wireless interface by turning off power management.

[Delta]

Well success, but I kind of feel I've waste mine and everyone's time.

Whilst faffing around behind the telly, I accidentally knocked the (hard) power switch on the media player.  When it restarted, all network stuff now seems fine.  Ping times of <10ms.  And yes, it's still using DHCP, and still being assigned .4

I feel this is a bit of a  - I should have done a hard restart on it earlier, the network adapted had been restarted numerous times, and the cable had been in and out, but I can't understand how this was happening.

Anyway, it seems fine now, and I've had a nice lesson in networking, so thanks to everyone who chipped in.

  but