Raspberry PI networking issue - no response to ARP requests

[paulca]

So I have a couple of PIs on my network.  One of which receives data from various nodes.

Twice now I have caught it in a state where I can ping it and I can ssh to it and it's alive and working, but nothing else can see it.

Hub:  10.0.0.3
Probe: 10.0.0.101

I can log into both from my desktop.  However 101 cannot ping 3.  I checked with TCPDump from 101 and I see it constantly broadcasting for the hw address of 3, but nothing answers it.

15:56:56.478274 ARP, Request who-has 10.0.0.3 tell 10.0.0.101, length 28
15:56:57.518252 ARP, Request who-has 10.0.0.3 tell 10.0.0.101, length 28
15:57:02.315659 ARP, Request who-has 10.0.0.3 tell 10.0.0.101, length 28
15:57:03.358279 ARP, Request who-has 10.0.0.3 tell 10.0.0.101, length 28

I have found similar, related issues around the web, but no real solution.

I thought I'd ask the wealth of intelligent people in here who may be using RPIs before I resort to more brute force methods of static arp tables or keep alives.

Note, the PIs are on Wifi, my desktop is wired.  If I reboot the Wifi router everything is fine again for another while.

[madires]

Note, the PIs are on Wifi, my desktop is wired.  If I reboot the Wifi router everything is fine again for another while.

Try another WiFi router

[WaveyDipole]

Since you can 'see' both from your desktop, it would seem unlikely that this is down to the WiFi dropping out.
What netmask do you have set at each end? Are all devices set to the same netmask? And is it the same on the router?
Do all devices have the same gateway address set (i.e. the router)?

[paulca]

Since you can 'see' both from your desktop, it would seem unlikely that this is down to the WiFi dropping out.
What netmask do you have set at each end? Are all devices set to the same netmask? And is it the same on the router?
Do all devices have the same gateway address set (i.e. the router)?

I agree.  Netmask, is 255.255.255.0 on all of them.  They all use the same router.  It's a straight flat single subnet network.

It has something to do with either the Wifi on 101 not sending ARP broadcast, 3 not receiving them or more likely that 101 does not receive the ARP responses.  At that time, my desktop, 199 had an arp entry and could have responded, but didn't.

When it happens again I will do some more diagnostics on other hosts to see if the arps are getting out of the PI.

There are quite a few references to this being an issue, particularly on PI 2s.  I might have to upgrade it.  Unfortunately the different threads on the issues are quite ranging and hard to determine if there is or is not an under lying problem or what exactly it is.  Nor did I find a solution after reading about 4 different threads.

[WaveyDipole]

Does .3 have iptables or a firewall running on it? If so, it may be blocking requests?

You say that .101 cannot ping .3. Can .3 ping .101?

Can both of them ping the router?
Can the router (assuming it has a ping facility) ping both .3 and .101?

[paulca]

No duplicate IPs, DHCP is consigned to the 200-240 range.

I'll diagnose more next occurrence.

Can't be IPTables as it is fine for a while with correct arp table entries, even if I delete them, a quick ping and the ARP exchange happens fine.  Then suddenly 101 can't see 3 again.

To be honest, 101 is an old PI 2B and it's been annoying me for other reasons.  Maybe I should just replace it with a PI3.  I happen to have one lying around as I replaced it with an ESP8266.

On other thoughts, check this little exchange out that happens a lot, commands are run meer seconds apart:

Code: [Select]

paul@localhost ~ $ ssh pi@10.0.0.101
ssh: connect to host 10.0.0.101 port 22: No route to host
paul@localhost ~ $ ping 10.0.0.101
PING 10.0.0.101 (10.0.0.101) 56(84) bytes of data.
[b]64 bytes from 10.0.0.101: icmp_seq=1 ttl=64 time=1031 ms[/b]
64 bytes from 10.0.0.101: icmp_seq=2 ttl=64 time=29.1 ms
64 bytes from 10.0.0.101: icmp_seq=3 ttl=64 time=1.15 ms
^C
--- 10.0.0.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.158/353.988/1031.705/479.354 ms, pipe 2
paul@localhost ~ $ ssh pi@10.0.0.101
pi@10.0.0.101's password:

Frequently 101 will not answer to SSH.  Even when I couldn't remember it's IP I scanned the whole LAN with NMap looking for SSH ports and it didn't find the PI.  But if you ping it, it suddenly appears.

I'm thinking either 101 has dodgy networking of some kind or a dodgy USB Wifi dongle.

Another reason to replace it with the PI 3.

[WaveyDipole]

That does seem to rule out IP tables.

Was the above exchange done from .3 or the desktop? I take it that at the same time you can communicate with other hosts without any problem? Assuming that is the case, then looking at this exchange does seem to suggest that the network connection between the PI2 and router is intermittently failing.

If the WiFi signal is intermittently dropping out on the PI2 then presumably you should see the PI2 disappear from the router status or connected devices page? Or possibly you might see the WiFi signal drop out on the PI itself. You should be able to use cat /proc/net/wireless' or 'watch -n 1 cat /proc/net/wireless' to view WiFi status.

Presumably you could move the SSD card to the PI3 to see if it makes a difference. You could also swap the WiFi dongles around to see whether that solves the problem or whether the problem moves from one PI to the other.

[paulca]

PI3 has onboard Wifi.

Swapping the memory, would need it copied from an SD to a MicroSD, no biggy with "dd".  The trouble is moving the sensor birds nest over.  I'd be compelled to tidy it up.    It's not that bad, but I have about 3 things sharing 3 way header pins I made to split the 5V and GND pins.  The dupont wires keep falling off if you look at it funny.  Really do need a better alternative.

I suppose it does need done. 

[Avacee]

You mention this happens a while after booting. Could the Pi going into power-saving mode?
iwconfig will display if power-saving is enabled.

Destructions on how to turn off power-saving to remove it as a possibility:
https://www.modmypi.com/blog/disable-wifi-power-management
http://www.thelowercasew.com/disabling-wifi-power-management-permanently-for-raspberry-pi-3-with-raspbian-jessie

[acecase]

I would suspect a power saving issue as well. Try running a constant arping and see if it is occasionally not responding to arp requests. Also see if a constant ping will keep it alive. Beyond that, I would say these things are so inexpensive these days that a problematic unit could be retired without hurting too bad.

[cdev]

Thank you for these links.

You mention this happens a while after booting. Could the Pi going into power-saving mode?
iwconfig will display if power-saving is enabled.

Destructions on how to turn off power-saving to remove it as a possibility:
https://www.modmypi.com/blog/disable-wifi-power-management
http://www.thelowercasew.com/disabling-wifi-power-management-permanently-for-raspberry-pi-3-with-raspbian-jessie

[paulca]

Oddly the issue has not resurfaced and the PI has been running fine since this post.

Well... I say "fine".

Last Sunday evening it locked up.  When rebooted the ssh server has not restarted.  Repeatedly rebooting it and still no ssh.  Yet everything else is functioning.  I still haven't bothered to connect a HDMI cable and keyboard to it to find out what has upset it.  I wondered about disk space, some log file eating the disc, but it had a 16Gb memory card that was only about 25% full and I don't think I am using anything that would log that much.

I'm sure it will be something simple, but a the greatest sin a headless appliance device can do is fail to start it's ssh server

[paulca]

The cause is simple the solution is also simple.  It's just annoying.

SD corruption.  The disc reports fine with fsck, but I have found multiple binaries with issues.

"tar" for example has a corrupted symbol table and is trying to load libraries such as:

ld-linux-crm(fo.3

As opposed to

ld-linux-arm.so.3

Obviously with tar broken the OS is fairly broken.  I can't install anything.  I "could" copy the tar command off another PI with a similar version of Rasbian, but the root cause is the memory card is old and dying.  It has already spend 3 years in a dash cam.  So time to save my stuff off it, reflash OS onto a new card.

[bson]

Sounds like a power saving mode, and the ICMP ECHO requests wake it, while TCP SYNs and oddly ARP won't.

[cdev]

The RPI NIC defaults to power saving mode which makes sense if its running as a client. That may be it.

But there may be some security related reason to want the power saving to be disabled. It may not matter if you have updated the software very recently.

I think I found it in the documentation and mailing list for hostapd and wpa_supplicant.

I will try to find it again and post it here.  There is something called sleep proxy, there is also wake on lan.

[TomS_]

Do all devices have the same gateway address set (i.e. the router)?

Gateway doesnt matter when all devices are in the same subnet. They can all have different or no gateway and they will still be able to talk.

Gateway only matters when you need to reach destinations beyond the local subnet, e.g. the Internet, or if you had a larger more complex network with multiple subnets.

[paulca]

You can gateway the local subnet if you want.

route add host 192.168.0.254 via dev eth0
route add net 192.168.0.0/24 via 192.168.0.254

Though why you would want to do that....  I suppose it allows you to firewall LAN<->LAN communications within a subnet.

From memory of trying this, it might be a bit more tricky than the above route commands suggest.

[tinalee]

I had this issue with two units (Rasperry pi Zero-W) Using built in wifi wlan0. It seems to be a definate bug. I have tried different wifi drivers and my only solution was to write a script that checks the arp cache every few seconds, identifies resovled IPs, inturn then attempts a ping to each. For any that do not respond, that arp entry is then deleted and re-PINGed. Not had issue since, now several weeks.
It seems that the ARP mechanism is getting stuck when devices go stale. The above workaround is the only solution i have found. I hope this helps others because it was doing my head in.
Here is my probably not so elegant script as a guide: (Change filename obviously to where you want to store temporary ARP output)

Code: [Select]


#!/bin/sh

filename="/mnt/sdcard/tmp/arp-entries.txt"
while true; do
  arp -n > "$filename"
  while read -r line || [ -n "$line" ]; do
    name=$(echo "$line" | tr -d '\r')
    if echo $name | grep "wlan"; then
        name=$(echo $name | cut -d " " -f1)
        ping -c1 $name > /dev/null
        if [ $? != 0 ]; then
          # PING failed, force re-ARP
#          echo "Failed"
          arp -d $name
          ping -c1 $name > /dev/null
        fi
    fi
  done < "$filename"
  sleep 120
done



[paulca]

This is interesting.  Thanks for the idea.

In my case I ran an RJ-45 cable into the living room, so both PIs are on a wired connection now.

I'm moving house (hopefully) in a month or so, so I may have to run a PI on a wifi connection, although I think the main use case can be covered with an ESP8266 which in my experience have been completely bullet proof reliable in terms of holding their Wifi connection and re-establishing it if the access point goes down.  I even replaced the access point completely with a new one, set the SID and PSK key to the same values and the ESPs just carried on working.