It usually isn’t the problem, but do be aware that ICMP PING isn’t exactly a priority for any network stack so it may be that packets are just ignored.
If you want to be 110% sure you have a problem, measure TCP retransmission rates or something along those lines.