-
Rigol DS2072 Lock-Ups from Network Interface
Posted by
motocoder
on 30 Oct, 2014 22:27
-
Changing the thread title now that the issue has been determined not to be specific to my scope. The issue seems to be a bug in many Rigol models, and is triggered when the scopes network interface is connected.
Starting this thread to document my support experience with Rigol, so that others who are considering a purchase from them can know what sort of support they provide (good or bad).
My DS2072 has been locking up since I got it a little more than a year ago, but lately the problem was so severe that the scope was pretty much unusable.
Where things are at right now: I contacted Rigol via Tequipment.net, after a few emails discussing what I had already done to troubleshoot things, they gave me an RMA number and an address in Oregon to ship the unit (at my expense). They should have received the scope this Monday, so I assume it's in their repair pipeline now.
So far the experience has been a good one. We'll see how long the repair takes.
-
#1 Reply
Posted by
marshallh
on 30 Oct, 2014 23:26
-
In my experience it took 8 weeks and then one day it showed up my doorstep via DHL. No tracking, notification
-
#2 Reply
Posted by
motocoder
on 30 Oct, 2014 23:34
-
In my experience it took 8 weeks and then one day it showed up my doorstep via DHL. No tracking, notification
Wow, that's depressing, but thank you for letting me know. I've got a project on hold right now due to no scope, so I suppose I'll have to see if I can find one at work to borrow for a day.
-
#3 Reply
Posted by
motocoder
on 02 Nov, 2014 16:07
-
I discovered that their warranty policy says thry will provide a loaner if the repair will take more than 10 business days. I emailed the Rigol rep about that last week. So far he has not responded, but will give him another day or two before escalating.
-
#4 Reply
Posted by
tequipment
on 06 Nov, 2014 17:41
-
Any issues message me. I will make sure its a fast turn around.
Evan
VP and Co-founder
TEquipment.NET
-
#5 Reply
Posted by
motocoder
on 12 Nov, 2014 18:00
-
Status update:
I've been dealing with Steve Huss from Rigol. They have had my scope since October 27th, approximately 17 calendar days and 13 business days. The last update I got was that they were still trying to reproduce the lock-up problem (which I was seeing approximately every couple of minutes) before sending the scope off for repair.
Evan from TEquipment.net was nice enough to contact Rigol last Friday (Nov-7), and it did appear that things were moving (at least they were responding to my emails) for the rest of the day on Friday. On Friday Steve asked if the scope was connected to anything (network, USB), as a possible clue as to how to reproduce the problem. I told him it was connected to a network switch, and he said he was going to connect it to their network and leave it running over the weekend to see if it locked up. He also told me "it is possible to get a loaner. We can look into that on Monday".
However, since that time, I have received no updates from Steve and he isn't responding to my emails.
-
#6 Reply
Posted by
motocoder
on 20 Nov, 2014 00:39
-
Ok, I've held off posting for quite some time, because I was working with Rigol while they tried to troubleshoot the issue. It's been slow going, but there is some progress.
Rigol has now had my oscilloscope in their hands for 3.5 weeks. After a little more than 2 weeks, they told me they were finally able to repro the lock-up problem (for me, it was happening very frequently). They only saw the problem after connecting the scope to a network, so their theory was that the problem was related to this. However, they were not able to repro the problem a second time. They came up with this theory that the problem was somehow triggered by my network.
After some discussion about the Rigol guys coming up to my house (they are in Oregon) to troubleshoot, they decided instead to send me another unit to test. They said if the problem did not repro with this unit, we could discuss a swap, and if it did repro, we could discuss further troubleshooting.
So today the unit arrives. It's a demo unit that has physical damage, is hardware version 1.0 (mine is/was 2.0), and has the anti-tampering sticker on the bottom broken. So regardless of what happens with the troubleshooting, I am not going to be keeping this one.
So I've plugged it in, and am looking to see if the problem repros.
-
#7 Reply
Posted by
FrankenPC
on 20 Nov, 2014 00:51
-
Mine had a bad PWM on the fan controller. Nice whine sound. It was bad out of the box so Tequipment.net just replaced it with no hassle. I've never dealt directly with Rigol.
-
#8 Reply
Posted by
motocoder
on 20 Nov, 2014 01:02
-
Mine had a bad PWM on the fan controller. Nice whine sound. It was bad out of the box so Tequipment.net just replaced it with no hassle. I've never dealt directly with Rigol.
TEquipment, in my experience, is great. And I think if it had been within 30 days of purchase, they would have swapped it out no issues. I will gladly do business with them in the future.
BTW - the demo unit has already locked up once. I am thinking maybe there is something on my network here that is triggering it.
-
#9 Reply
Posted by
Bud
on 20 Nov, 2014 01:08
-
They only saw the problem after connecting the scope to a network, so their theory was that the problem was related to this.
This may be half true, my 2072A locked up today when I was downloading the screen capture to my PC and accidentally unplugged the router power. They better look at their LAN part of the firmware overall.
-
#10 Reply
Posted by
Monkeh
on 20 Nov, 2014 01:11
-
They only saw the problem after connecting the scope to a network, so their theory was that the problem was related to this.
This may be half true, my 2072A locked up today when I was downloading the screen capture to my PC and accidentally unplugged the router power. They better look at their LAN part of the firmware overall.
They can look at the USB part (on the 1000Z, anyway) which barely works, while they're at it.
-
#11 Reply
Posted by
motocoder
on 20 Nov, 2014 01:13
-
They only saw the problem after connecting the scope to a network, so their theory was that the problem was related to this.
This may be half true, my 2072A locked up today when I was downloading the screen capture to my PC and accidentally unplugged the router power. They better look at their LAN part of the firmware overall.
That is good information. I suspect they are monitoring this thread, but I will pass that info along either way.
-
#12 Reply
Posted by
motocoder
on 20 Nov, 2014 01:32
-
They only saw the problem after connecting the scope to a network, so their theory was that the problem was related to this.
This may be half true, my 2072A locked up today when I was downloading the screen capture to my PC and accidentally unplugged the router power. They better look at their LAN part of the firmware overall.
I just tried power cycling the network switch. The scope didn't completely lock up, but it started doing all sorts of whacky things like artifacts on the screen, controls not responding as expected, etc.
-
#13 Reply
Posted by
motocoder
on 20 Nov, 2014 01:36
-
I would appreciate it if someone else with a DS2072 or DS2072A (or similar model) could try this experiment:
Plug the scope into a network with a DHCP server so the scope can get an IP address and come up on the network.
Boot the scope and verify it is working correctly.
Power cycle the network switch to which the scope is attached.
Check the controls on the scope - try enabling and disabling Ch1 and Ch2 by pushing the channel buttons.
Note any strange behavior (or not) and post your results to the thread here.
-
#14 Reply
Posted by
Bud
on 20 Nov, 2014 04:03
-
DS2072A, confirmed intermittent wacky behavior after power cycling the home LAN router. Did test two times, first time the menus gone unresponsive. Second time everything was normal. Mine has a static IP, not DHCP.
Using Wireshark I captured that in first test the 2072A sent two ARP packets before going crazy. Second time it first still sent same two ARP packets but spit out bunch of other packets after that.
So indeed 2072A has problems with LAN.
-
#15 Reply
Posted by
motocoder
on 20 Nov, 2014 04:22
-
DS2072A, confirmed intermittent wacky behavior after power cycling the home LAN router. Did test two times, first time the menus gone unresponsive. Second time everything was normal. Mine has a static IP, not DHCP.
Using Wireshark I captured that in first test the 2072A sent two ARP packets before going crazy. Second time it first still sent same two ARP packets but spit out bunch of other packets after that.
So indeed 2072A has problems with LAN.
Thanks, Bud. This should be easy for their enigineers to repro now.
-
#16 Reply
Posted by
apelly
on 20 Nov, 2014 05:42
-
Power cycled the switch a dozen or so times. No problem.
Pulled the network cable for a few seconds and replaced it. Your symptoms exactly.
It's not hung though. It looks like part of the UI is running very very slowly. Can adjust the timebase no problem.
After a while it couldn't tell if the LAN was connected or not.
Firmware is within the last 2 releases I'm pretty sure. Web UI reports 00.03.00.SP1
-
#17 Reply
Posted by
motocoder
on 20 Nov, 2014 06:03
-
Power cycled the switch a dozen or so times. No problem.
Pulled the network cable for a few seconds and replaced it. Your symptoms exactly.
It's not hung though. It looks like part of the UI is running very very slowly. Can adjust the timebase no problem.
After a while it couldn't tell if the LAN was connected or not.
Firmware is within the last 2 releases I'm pretty sure. Web UI reports 00.03.00.SP1
Thanks - after waiting, did it ever come back and act normally, or did you have to restart it?
-
#18 Reply
Posted by
apelly
on 21 Nov, 2014 00:59
-
I only had time to muck around for 5 minutes so I didn't wait more than a few. It did not come back. I'll have another fool around for you later.
-
#19 Reply
Posted by
motocoder
on 21 Nov, 2014 02:40
-
I only had time to muck around for 5 minutes so I didn't wait more than a few. It did not come back. I'll have another fool around for you later.
That's probably sufficient. 5 minutes is a pretty long time.
Thanks again - this forum is such a great resource
-
#20 Reply
Posted by
jlmoon
on 21 Nov, 2014 02:47
-
Starting this thread to document my support experience with Rigol, so that others who are considering a purchase from them can know what sort of support they provide (good or bad).
My DS2072 has been locking up since I got it a little more than a year ago, but lately the problem was so severe that the scope was pretty much unusable.
Where things are at right now: I contacted Rigol via Tequipment.net, after a few emails discussing what I had already done to troubleshoot things, they gave me an RMA number and an address in Oregon to ship the unit (at my expense). They should have received the scope this Monday, so I assume it's in their repair pipeline now.
So far the experience has been a good one. We'll see how long the repair takes.
Hello Motocoder,
I have a DS2202 for about 8 months now and have experienced in the past a couple of times display freezing and no control response. I don't have my equipment connected to any network or usb devices attached to my unit. I don't remember the test state I was in when encountering this strange behavior but will start taking notes. It has happened to me only about twice now and the only resolve was to power cycle the scope.
-
#21 Reply
Posted by
motocoder
on 21 Nov, 2014 06:16
-
Starting this thread to document my support experience with Rigol, so that others who are considering a purchase from them can know what sort of support they provide (good or bad).
My DS2072 has been locking up since I got it a little more than a year ago, but lately the problem was so severe that the scope was pretty much unusable.
Where things are at right now: I contacted Rigol via Tequipment.net, after a few emails discussing what I had already done to troubleshoot things, they gave me an RMA number and an address in Oregon to ship the unit (at my expense). They should have received the scope this Monday, so I assume it's in their repair pipeline now.
So far the experience has been a good one. We'll see how long the repair takes.
Hello Motocoder,
I have a DS2202 for about 8 months now and have experienced in the past a couple of times display freezing and no control response. I don't have my equipment connected to any network or usb devices attached to my unit. I don't remember the test state I was in when encountering this strange behavior but will start taking notes. It has happened to me only about twice now and the only resolve was to power cycle the scope.
Yes, please take notes. I've shared this thread with Rigol, ao hopefully this info will help their enigineers.
-
#22 Reply
Posted by
Rasz
on 21 Nov, 2014 07:05
-
setup dhcp server on your second computer/laptop, connect scope directly to it (crossed cable, no network switches), play with cable/disabling dhcp server
you can also generate arbitrary arp traffic with nping or
http://www.netscantools.com/nstpro_arpping.htmlthis will eliminate your home lan setup/particular switch
-
#23 Reply
Posted by
motocoder
on 21 Nov, 2014 08:06
-
setup dhcp server on your second computer/laptop, connect scope directly to it (crossed cable, no network switches), play with cable/disabling dhcp server
you can also generate arbitrary arp traffic with nping or http://www.netscantools.com/nstpro_arpping.html
this will eliminate your home lan setup/particular switch
Rigol told me there are known issues with crossover cables, so I think you cant eliminate the switch. Also, you can see from the thread that static IP setup shows issue too, so no need for dhcp server.
-
#24 Reply
Posted by
Murray
on 21 Nov, 2014 15:50
-
You might want to check the DHCP lease time in your router. If there is traffic during the last half of the lease, the lease will be renewed. If your lease time is quite low, and you left the scope on and connected but did no network activity for half the lease time, your DHCP lease might expire, and would have to be re-negotiated. If the scope does not handle the re-negotiation properly, this might be enough, like the other network disturbances that have been tested, to fubar the scope. If you increase you lease time to more than twice the longest time you expect to leave the scope on and connected to the net with no network activity, it MIGHT reduce the lockups.
-
#25 Reply
Posted by
motocoder
on 21 Nov, 2014 16:29
-
You might want to check the DHCP lease time in your router. If there is traffic during the last half of the lease, the lease will be renewed. If your lease time is quite low, and you left the scope on and connected but did no network activity for half the lease time, your DHCP lease might expire, and would have to be re-negotiated. If the scope does not handle the re-negotiation properly, this might be enough, like the other network disturbances that have been tested, to fubar the scope. If you increase you lease time to more than twice the longest time you expect to leave the scope on and connected to the net with no network activity, it MIGHT reduce the lockups.
interesting theory. I think my next step will be to set up a static ip and see if it locks up. If so, I am going to unplug everything else from the switch and see if it still happens. I also want to try another switch. If it really is switch related, I will just send this one to Rigol.
-
#26 Reply
Posted by
motocoder
on 21 Nov, 2014 21:12
-
The Rigol reps in Oregon have shared this thread with Rigol engineering, so I'm going to continue to record my findings here. Everyone else feel free to add any detail you might have.
This morning I swapped out the network cable and verified that the unit still locks up.
I then configure the scope with a static IP address, and turned off "Auto IP" (whatever that is). The scope still locks up.
I'm going to try a few more things over the weekend, but I think we can rule cable issues and DHCP out.
-
#27 Reply
Posted by
Tyrian
on 21 Nov, 2014 21:37
-
I recently got an MSO2072A scope from TEquipment a few days ago and began experiencing the same screen freezing bug. However, my scope is not connected to a LAN. We're discussing it in another thread here:
https://www.eevblog.com/forum/testgear/rigol-mso2072a-problem-display-stops-updating/I've also contacted Rigol tech support about this and have been talking to Jason (out of Ohio, I believe). I've been posting about my experience in the above thread. Have you tried resetting the FRAM during boot? The theory in the other thread is that there may be a software bug that corrupts a section of the FRAM, resulting in the screen freeze, and bumping a control knob makes the screen update again.
-
#28 Reply
Posted by
motocoder
on 21 Nov, 2014 23:05
-
I recently got an MSO2072A scope from TEquipment a few days ago and began experiencing the same screen freezing bug. However, my scope is not connected to a LAN. We're discussing it in another thread here: https://www.eevblog.com/forum/testgear/rigol-mso2072a-problem-display-stops-updating/
I've also contacted Rigol tech support about this and have been talking to Jason (out of Ohio, I believe). I've been posting about my experience in the above thread. Have you tried resetting the FRAM during boot? The theory in the other thread is that there may be a software bug that corrupts a section of the FRAM, resulting in the screen freeze, and bumping a control knob makes the screen update again.
Yes, resetting the FRAM is SOP after doing a BIOS update - I always do it. And when I got the demo unit from Rigol, this is one of the first things I did before starting testing. The scope does not lock up if it's not connected to anything.
However, note that one of the other posters on this thread mentioned that he has seen some issues with USB, so if you're scope is connected via USB, you might want to try disconnecting that as well.
Thanks for the link to the other thread - I'll take a look and see if there are any additional clues there.
-
#29 Reply
Posted by
motocoder
on 22 Nov, 2014 01:20
-
Locks up with a static IP. I have connected it to a 10MBps hub now (by itself). If it locks up on the hub, it will be useful to use for wireshark as every device on the hub receives every packet (not so with a switch).
-
#30 Reply
Posted by
motocoder
on 22 Nov, 2014 18:01
-
Ok, so last night I ran through several test scenarios. I reduced the things connected to my network switch, starting with the scope by itself, and gradually adding other computers.
What I found is that the lock-up only seems to occur when my primary desktop computer (which runs Windows 8.1) is connected to the switch along with the scope. I can remove everything except the scope and that computer from the switch, and see the lock-up, but if I remove the Windows computer and add the other machines, I don't see the lock-up.
I have an old hub that someone at work loaned me. I am trying to repro the lock-up with the scope and my computer connected to that. The hub will let me use Wireshark and see all the packets coming from the scope, so I'd like to go that route if possible. If not, I can run Wireshark on my desktop and at least capture broadcast packets and packets between the desktop and the scope.
Update: no lock-up with the hub. Swapping out the switch.
-
#31 Reply
Posted by
motocoder
on 23 Nov, 2014 19:59
-
So yesterday I tried a number of things, and thought I had discovered what was triggering the bug. I still may have, as the scope is no longer locking up, but it's not conclusive. Here are the details:
I swapped out the switch - no change, i.e. the scope still locks up. I also determined that the scope would lock up when connected to a switch along with my Windows desktop computer and nothing else, but would not lock up even if every computer in my house other than my Windows desktop were also connected. So, clearly whatever the issue is, it was being triggered by some traffic originating from my Windows desktop.
At that point, I decided to set up WireShark and get some network captures. Unfortunately, my switch does not support port mirroring, and I was not able to reproduce the lock-up with the scope connected to a 10MB hub (the only hub that I have). So I decided to just put the scope and my desktop computer on the switch, and run WireShark from my desktop. I captured a number of lock-ups this way. The attached capture shows one that was especially interesting. In this capture, the scope booted and locked up almost immediately; the whole process from first network packet from the scope until lock-up took less than 30 seconds.
Now note that this capture is pretty brief, and does not include one type of network packet that the scope may have been receiving, namely broadcast packets. So I did some more captures using broadcast captures. I don't want to post those here, as the likelihood that there may be some security-related or personal info goes up and I don't have the time to review every packet to make sure that's not an issue. However, I can tell you that there were ARP broadcast packets, some broadcasts for that same scanner software, some NETBIOS broadcast packets, and some dropbox local sync broadcast packet. I also see some HTTP exchange between the scope and the PC, specifically it is some LXI identification query. I don't think this is causing any issues, because I also see these when the scope is stable with no lock-ups for 12+ hours. Also note that when the scope locks up, it always right after it sends 3 ARP queries to the desktop, each of which asking what MAC address owns its IP address.
So, looking at this capture, and the others, I decided the only thing that looked unique was these strange scanner software packets. In WireShark they are labelled as protocol "BJNP", but they are actually UDP packets to some particular port. The UDP point-to-point packets are sent to port 8612, and the UDP broadcast packets are sent to port 8612. The BJNP broadcast packets seem to be some mechanism for the Canon printer/scanner drivers to identify Canon printers and scanners on the network. I have no idea why the driver also sends such packets directly to specific IP addresses. But in any case, I no longer own any Canon printers, so I uninstalled this driver software.
And then after rebooting everything, including the scope, wonder of wonders, the lock-ups stopped. This is a plausible explanation, as this computer is the only computer on the network that still had the Canon software installed on it, and would explain why the scope only locks up when this computer is on the same LAN with it.
However, that's not the end of story. I decided to write a small "UdpPing" program to allow Rigol to recreate this scenario without having to install old Canon printer drivers. I did that, but so far I have not been able to recreate the lock-up. It's possible that there is some network traffic that I didn't capture (multicast packets maybe?), or perhaps my program isn't faithfully recreating the exact packets that I did capture. I'm going to poke around with it a little bit later today, and will post back here if I find out more.
-
#32 Reply
Posted by
tautech
on 23 Nov, 2014 20:19
-
motocoder from now on you shall be known as Sherlock Holmes.
Great detective work.
-
#33 Reply
Posted by
marmad
on 23 Nov, 2014 20:29
-
Hey motocoder - you never responded to my question in the other thread, so I'll just ask it again here:
Based on the behavior of my RUU utility, there were some major changes to the LAN code in the DS2000 FW, beginning with, I believe, v.3 (although maybe it was v.2). Have you downgraded to FW 02.01.00.03 or FW 01.01.00.02 to see if the lock-ups persist?
Not that it solves the problem, but if the bug was introduced with the changes to the LAN in that specific FW#, it might help Rigol locate it more quickly.
-
#34 Reply
Posted by
motocoder
on 23 Nov, 2014 20:56
-
Hey motocoder - you never responded to my question in the other thread, so I'll just ask it again here:
Based on the behavior of my RUU utility, there were some major changes to the LAN code in the DS2000 FW, beginning with, I believe, v.3 (although maybe it was v.2). Have you downgraded to FW 02.01.00.03 or FW 01.01.00.02 to see if the lock-ups persist?
Not that it solves the problem, but if the bug was introduced with the changes to the LAN in that specific FW#, it might help Rigol locate it more quickly.
Hi marmad - sorry, I haven't been monitoring that thread. You raise a good point. I was running firmware 02.01.00.03, and only upgraded to 03.01.00.04 in troubleshooting this (before contacting Rigol). I haven't tried 01.01.00.02, although I think that may be what was on the scope when I got it. So it is definitely possible that the bug was introduced in the 02.* firmware.
At the moment, I'm not able to recreate the lock-up anymore, though. I just modified my UdpPing program to allow you to specify the payload and the TTL value, so that I can hopefully recreate exactly the packets that the Canon software was sending.
-
-
I'm probably missed something - but if you want to use a DHCP server then just tell it that the scope's MAC address always is given the same IP - i.e. reserved. There are other solutions but on a network this usually a good solution.
-
#36 Reply
Posted by
motocoder
on 23 Nov, 2014 22:13
-
I'm probably missed something - but if you want to use a DHCP server then just tell it that the scope's MAC address always is given the same IP - i.e. reserved. There are other solutions but on a network this usually a good solution.
I'm not sure what problem it is you are trying to solve. The IP address of the scope isn't an issue, and I aware of ways to ensure the scope always gets the same IP, with DHCP or without.
-
#37 Reply
Posted by
motocoder
on 23 Nov, 2014 22:16
-
Ok, I recreated the exact sequence of Canon printer driver broadcast and unicast (to the scope) packets from one of the captures I had, and this isn't recreating the lock-up. Also, I have tried to reinstall the Canon drivers, but I'm not able to get that set up correctly without actually having the Canon printer. So it seems I won't easily be able to recreate the lock-up.
At this point, I think I've donated several thousand dollars worth of engineering time to help Rigol troubleshoot this - several times the cost of the scope for sure, so I am going to just ask them again to send me my scope back.
-
#38 Reply
Posted by
Rasz
on 24 Nov, 2014 11:08
-
causing any issues, because I also see these when the scope is stable with no lock-ups for 12+ hours. Also note that when the scope locks up, it always right after it sends 3 ARP queries to the desktop, each of which asking what MAC address owns its IP address.
So, looking at this capture, and the others, I decided the only thing that looked unique was these strange scanner software packets. In WireShark they are labelled as protocol "BJNP", but they are actually UDP packets to some particular port. The UDP point-to-point packets are sent to port 8612, and the UDP broadcast packets are sent to port 8612. The BJNP broadcast packets seem to be some mechanism for the Canon printer/scanner drivers to identify Canon printers and scanners on the network. I have no idea why the driver also sends such packets directly to specific IP addresses. But in any case, I no longer own any Canon printers, so I uninstalled this driver software. And then after rebooting everything, including the scope, wonder of wonders, the lock-ups stopped.
This is a plausible explanation, as this computer is the only computer on the network that still had the Canon software installed on it, and would explain why the scope only locks up when this computer is on the same LAN with it.
However, that's not the end of story. I decided to write a small "UdpPing" program to allow Rigol to recreate this scenario without having to install old Canon printer drivers. I did that, but so far I have not been able to recreate the lock-up. It's possible that there is some network traffic that I didn't capture (multicast packets maybe?), or perhaps my program isn't faithfully recreating the exact packets that I did capture. I'm going to poke around with it a little bit later today, and will post back here if I find out more.
This is why I pinpointed arp broadcast in the first place:
https://news.ycombinator.com/item?id=8565161look at your captures again and you will find this shitty Canon software spamming mDNS crap all over your network
its not only type of packet, but also amount per second.
I read similar story on one of the bofh forums some time ago where one computer was able to start arp flood in 30 computer office, it started with one, but 100ms later every computer in the office (with mdns support) was spamming that shit
100Mbit flood on every port in the building.
edit: btw to not give Rigol stupid ideas (like charging you for shipping) - this is TOTALLY their fault, they have shit in low level network stack.
-
#39 Reply
Posted by
Kjelt
on 24 Nov, 2014 12:07
-
Probably just me, but looking on the wireshark capture, I don't get it.
AFAICT the Rigol scope is on an IPv6 ...........:01:8a address and your computer on an IPv6 .........:30:8f address.
So who or what is on the 192.168.1.120 and 192.168.1.9 IPv4 addresses, one of them is probably still your computer, but what is the other
address they all want to discover?
-
#40 Reply
Posted by
motocoder
on 24 Nov, 2014 14:54
-
causing any issues, because I also see these when the scope is stable with no lock-ups for 12+ hours. Also note that when the scope locks up, it always right after it sends 3 ARP queries to the desktop, each of which asking what MAC address owns its IP address.
So, looking at this capture, and the others, I decided the only thing that looked unique was these strange scanner software packets. In WireShark they are labelled as protocol "BJNP", but they are actually UDP packets to some particular port. The UDP point-to-point packets are sent to port 8612, and the UDP broadcast packets are sent to port 8612. The BJNP broadcast packets seem to be some mechanism for the Canon printer/scanner drivers to identify Canon printers and scanners on the network. I have no idea why the driver also sends such packets directly to specific IP addresses. But in any case, I no longer own any Canon printers, so I uninstalled this driver software. And then after rebooting everything, including the scope, wonder of wonders, the lock-ups stopped.
This is a plausible explanation, as this computer is the only computer on the network that still had the Canon software installed on it, and would explain why the scope only locks up when this computer is on the same LAN with it.
However, that's not the end of story. I decided to write a small "UdpPing" program to allow Rigol to recreate this scenario without having to install old Canon printer drivers. I did that, but so far I have not been able to recreate the lock-up. It's possible that there is some network traffic that I didn't capture (multicast packets maybe?), or perhaps my program isn't faithfully recreating the exact packets that I did capture. I'm going to poke around with it a little bit later today, and will post back here if I find out more.
This is why I pinpointed arp broadcast in the first place:
https://news.ycombinator.com/item?id=8565161
look at your captures again and you will find this shitty Canon software spamming mDNS crap all over your network
its not only type of packet, but also amount per second.
I read similar story on one of the bofh forums some time ago where one computer was able to start arp flood in 30 computer office, it started with one, but 100ms later every computer in the office (with mdns support) was spamming that shit 100Mbit flood on every port in the building.
edit: btw to not give Rigol stupid ideas (like charging you for shipping) - this is TOTALLY their fault, they have shit in low level network stack.
Very perceptive, Rasz! I think "arp flood" may be what happened here. I looked at the packet capture again,and actually it is my desktop computer that was sending those last 3 arp packets to the scope, not the scope sending the last 3 arp packets, right before or after the lock-up. Last night I used some software (bit-twist) to simulate that, but it didn't trigger a lock-up even if I sent hundreds of them.
Also, I didn't have capture of multicast packets enabled - only broadcast unicast to/from my desktop - so I may have missed whatever the Canon software was doing.
Can you elaborate on any theories you might have as to how the Canon software might be triggering the issue? Now that I know how to use bit-twist to send arbitrary packets, it would be great if I could exactly reproduce the lock-up so that Rigol can start working on a fix.
-
#41 Reply
Posted by
motocoder
on 24 Nov, 2014 15:01
-
Probably just me, but looking on the wireshark capture, I don't get it.
AFAICT the Rigol scope is on an IPv6 ...........:01:8a address and your computer on an IPv6 .........:30:8f address.
So who or what is on the 192.168.1.120 and 192.168.1.9 IPv4 addresses, one of them is probably still your computer, but what is the other
address they all want to discover?
Those are MAC addresses, not IPv6 addresses.
-
#42 Reply
Posted by
Kjelt
on 24 Nov, 2014 15:31
-
Those are MAC addresses, not IPv6 addresses.
ok so the scope asks who has 192.168.1.9 but wants a reply to 0.0.0.0 never seen that how do you explain that?
-
#43 Reply
Posted by
motocoder
on 24 Nov, 2014 15:35
-
Those are MAC addresses, not IPv6 addresses.
ok so the scope asks who has 192.168.1.9 but wants a reply to 0.0.0.0 never seen that how do you explain that?
I think it is how arp works. When the scope starts up, it sends a broadcast arp request to make sure some other device doesn't have the IP address it wants to use. The arp broadcast packets are a normal thing. It is the way a device determines the ethernet mac address to send traffic to for a particular IP.
I think the unicast arp requests (from the Asustek... to RigolTec...) were actually being generated by the Canon software.
-
#44 Reply
Posted by
Kjelt
on 24 Nov, 2014 15:43
-
Ok well not to rock the boat for nothing just interested, he're is an ARP wireshark cap from me.
You can see that here that the Hewlett device that asks for the IP address from the Cisco already has a valid IP address on its own.
It might well be that 0.0.0.0 is used if the device has no IP address yet.
-
#45 Reply
Posted by
Monkeh
on 24 Nov, 2014 15:50
-
This is a standard method used by most DHCP clients to determine if the address they're about to use is already in use on the network. It's easier to check than to just assume and try and cope with the 'hilarity' of two devices claiming the same IP.
The client SHOULD perform a check on the suggested address to ensure that the address is not already in use. For example, if the client is on a network that supports ARP, the client may issue an ARP request for the suggested request. When broadcasting an ARP request for the suggested address, the client must fill in its own hardware address as the sender's hardware address, and 0 as the sender's IP address, to avoid confusing ARP caches in other hosts on the same subnet. If the network address appears to be in use, the client MUST send a DHCPDECLINE message to the server.
-
#46 Reply
Posted by
motocoder
on 24 Nov, 2014 16:01
-
Ok well not to rock the boat for nothing just interested, he're is an ARP wireshark cap from me.
You can see that here that the Hewlett device that asks for the IP address from the Cisco already has a valid IP address on its own.
It might well be that 0.0.0.0 is used if the device has no IP address yet.
There are both unicast and broadcast arp exchanges. It looks like the unicast are typically between the default gateway/router and some other machine, and is just a way to refresh the cached arp info. The broadcast messages are when the device sending them really has no clue who owns a particular IP address.
-
#47 Reply
Posted by
Rasz
on 25 Nov, 2014 11:31
-
Those are MAC addresses, not IPv6 addresses.
ok so the scope asks who has 192.168.1.9 but wants a reply to 0.0.0.0 never seen that how do you explain that?
https://tools.ietf.org/html/rfc5227#section-2.1.1I think "arp flood" may be what happened here. I looked at the packet capture again,and actually it is my desktop computer that was sending those last 3 arp packets to the scope, not the scope sending the last 3 arp packets, right before or after the lock-up. Last night I used some software (bit-twist) to simulate that, but it didn't trigger a lock-up even if I sent hundreds of them.
it probably only works if you spam with same IP scope just requested
from your brief screenshot it looks like scope replies like it already owns that particular IP _while it still waits for DHCP request reply_
Can you elaborate on any theories you might have as to how the Canon software might be triggering the issue? Now that I know how to use bit-twist to send arbitrary packets, it would be great if I could exactly reproduce the lock-up so that Rigol can start working on a fix.
canon loves polluting local networks, quick google shows another garbage packet type
https://ask.wireshark.org/questions/5178/why-gratuitous-arps-for-0000I never had to deal with it personally, so all I got is guesses, would be easier with a live example - too bad you fixed yours.
-
#48 Reply
Posted by
motocoder
on 25 Nov, 2014 18:19
-
it probably only works if you spam with same IP scope just requested
from your brief screenshot it looks like scope replies like it already owns that particular IP _while it still waits for DHCP request reply_
Yes, I changed the IP address before replaying the packets.
-
#49 Reply
Posted by
motocoder
on 29 Nov, 2014 16:50
-
Got my scope back from Rigol yesterday. Since they sent no shipping notice or tracking number, I wasn't expecting it, and it sat outside in the inclement weather (fortunately under the overhang of my porch) for some indeterminate amount of time. I brought indoors and let the condensation dry out overnight, and it seems to be working.
So in summary: Rigol had my scope for 4 weeks. There was actually nothing physically wrong with my scope; the lock-up issue I encountered is caused by a bug in their network implementation that was triggered by some network traffic on my LAN.
Positives- The Rigol reps I communicated with were generally pleasant to deal with.
- The Rigol reps generally seemed to have an interest in determining the root cause of the bug, or at least gathering as much info as they could to assist the engineers in investigation.
Negatives- No notification or tracking number on return shipments. My scope sat outside for some time, and could easily have been stolen or damaged by rain.
- I had to pay for shipping, not just once, but twice.
- Rigol does not seem to be set up to communicate with the customer on the status of the repair process. Plan on sending your equipment into an 8 week black hole
- Rigol does not live up to their written warranty commitments, specifically to provide a loaner or replacement if the repair will take more than 10 days.
-
#50 Reply
Posted by
motocoder
on 29 Nov, 2014 20:57
-
- Rigol does not live up to their written warranty commitments, specifically to provide a loaner or replacement if the repair will take more than 10 days.
I had good RMA service with:
A prepaid shipping label to Rigol
Tracked, shipping from Wash. to Ohio: Rigol
36 hours to repair
Tracking # of the shipment(gnd) back to me.
I have No Problem with that service
But I did have to drive to another Country to drop off & pick it up 
Interesting. Maybe this is a regression since they relocated to Oregon. Or maybe I was not dealing with the actual Rigol service people? In any event, this will be the last piece of Rigol test equipment I buy. Ever. Scope is going to be up for sale on Monday if you're interested.
-
#51 Reply
Posted by
motocoder
on 29 Nov, 2014 21:38
-
Oh, and one more thing. The scope went to them with lots of time left on the trial options. It came back with no time left on any of them.
-
#52 Reply
Posted by
marmad
on 29 Nov, 2014 21:43
-
Oh, and one more thing. The scope went to them with lots of time left on the trial options. It came back with no time left on any of them.
After a year of owning it? So basically, you never used the DSO.
-
#53 Reply
Posted by
motocoder
on 29 Nov, 2014 21:44
-
Oh, and one more thing. The scope went to them with lots of time left on the trial options. It came back with no time left on any of them.
After a year of owning it? So basically, you never used the DSO.
No, in my case it was because I had the Riglol options installed, and when I uninstalled those, it restored whatever time was there before installation. It isn't an issue for me, but my point is, for someone with a new scope, it might be a big deal.
-
#54 Reply
Posted by
marmad
on 29 Nov, 2014 21:47
-
No, in my case it was because I had the Riglol options installed, and when I uninstalled those, it restored whatever time was there before installation. It isn't an issue for me, but my point is, for someone with a new scope, it might be a big deal.
Well, perhaps they knew it was a little suspicious that you had time left on the trials after a year. In any case, I didn't have any problem whatsoever getting a new trial code from Rigol when I lost my trial options prematurely.
-
#55 Reply
Posted by
motocoder
on 29 Nov, 2014 21:48
-
No, in my case it was because I had the Riglol options installed, and when I uninstalled those, it restored whatever time was there before installation. It isn't an issue for me, but my point is, for someone with a new scope, it might be a big deal.
Well, perhaps they knew it was a little suspicious that you had time left on the trials after a year. In any case, I didn't have any problem whatsoever getting a new trial code from Rigol when I lost my trial options prematurely.
Fair enough. I'd like to hear from someone with a
recent repair experience, to see if this level of shit service is now the norm, or whether I just got "special" treatment.
-
#56 Reply
Posted by
marmad
on 29 Nov, 2014 21:57
-
Fair enough. I'd like to hear from someone with a recent repair experience, to see if this level of shit service is now the norm, or whether I just got "special" treatment.
I got excellent service from the dealer I bought my Rigol scope from. I'm not sure why you would believe you can buy cheap tech from the lowest-cost online dealer, and then expect wonderful service over a year after the purchase date. You generally have to pay more up front - either for more expensive equipment or for after-sales support - and this is true for virtually all technology.
-
#57 Reply
Posted by
motocoder
on 29 Nov, 2014 22:03
-
Fair enough. I'd like to hear from someone with a recent repair experience, to see if this level of shit service is now the norm, or whether I just got "special" treatment.
I got excellent service from the dealer I bought my Rigol scope from. I'm not sure why you would believe you can buy cheap tech from the lowest-cost online dealer, and then expect wonderful service over a year after the purchase date. You generally have to pay more up front - either for more expensive equipment or for after-sales support - and this is true for virtually all technology.
Because 1) They offer a 3 year warranty and I expect them to live up to it, and 2) they are trying to compete with the big boys. A poor support experience is a factor that people should consider before buying. I spent a significant amount of time helping them troubleshoot this bug, which since I'm selling the scope I really have nothing personal to gain, and no personal stake in this. I'm just trying to put some info out there that may let people make a more informed purchase decision.
But I have to ask, why are you so defensive about this? What's your interest in it?
Also, for anyone that's interested, I just noticed that they updated my scope firmware. I think this might be a new version not publicly available yet: 00.03.02.SP3
-
#58 Reply
Posted by
motocoder
on 29 Nov, 2014 22:35
-
Also, for anyone that's interested, I just noticed that they updated my scope firmware.
I think this might be a new version not publicly available yet: 00.03.02.SP3
Interesting.
Will You be testing the new Firmware if it fixes the AC coupled triggering bugs? ( jitter and offset)
I got a E-mail from Rigol NA, to test a new FW for the fixes, but He did not realize in was not released for the DS2000 yet!
Do you have have it now? !!!
Yes, am happy to do that. Would you review my set-up to make sure I am testing this properly?
- Connect function generator with 1kHz sine wave, 2V peak-to-peak
- Channel 1 DC coupling
- AC triggering (Trigger menu, Setting, Coupling = AC
-
#59 Reply
Posted by
motocoder
on 29 Nov, 2014 22:53
-
Also, for anyone that's interested, I just noticed that they updated my scope firmware.
I think this might be a new version not publicly available yet: 00.03.02.SP3
Interesting.
Will You be testing the new Firmware if it fixes the AC coupled triggering bugs? ( jitter and offset)
I got a E-mail from Rigol NA, to test a new FW for the fixes, but He did not realize in was not released for the DS2000 yet!
Do you have have it now? !!!
Yes, am happy to do that. Would you review my set-up to make sure I am testing this properly?
- Connect function generator with 1kHz sine wave, 2V peak-to-peak
- Channel 1 DC coupling
- AC triggering (Trigger menu, Setting, Coupling = AC
A Square wave is better, if it has fast risetime.
The risetime needs to fast to show 20nsec and not be a slope.
trigger level set to 50% (Min + Max)
Well, I don't have a good fast rise time square wave generator here. However, I have it set up like what I see in Dave's video, and I don't see any jitter. Attaching a screenshot.
-
#60 Reply
Posted by
marmad
on 29 Nov, 2014 22:54
-
But I have to ask, why are you so defensive about this? What's your interest in it?
I think you have that backwards - I have nothing to defend. I'm just wondering what are the reasonable expectations one can have (in terms of warranty service) given the circumstances (cost of item, length of time since purchase, etc).
Your experience does not sound very good, but I've had a lot of stuff serviced under warranty in the last 30 years - from virtually every kind of manufacturer you can think of - Rigol, HP, Canon, WD, Amazon (Kindle), etc, etc, - and honestly, your experience sounds about middle of the pack.
-
#61 Reply
Posted by
motocoder
on 29 Nov, 2014 23:02
-
But I have to ask, why are you so defensive about this? What's your interest in it?
I think you have that backwards - I have nothing to defend. I'm just wondering what are the reasonable expectations one can have (in terms of warranty service) given the circumstances (cost of item, length of time since purchase, etc).
Your experience does not sound very good, but I've had a lot of stuff serviced under warranty in the last 30 years - from virtually every kind of manufacturer you can think of - Rigol, HP, Canon, WD, Amazon (Kindle), etc, etc, - and honestly, your experience sounds about middle of the pack.
I think it's reasonable to expect they live up to the terms in their warranty, and that they provide some reasonable expectation as to turnaround time. I recently sent in a radar detector for service, and the rep told me worst case and typical turnaround times before I sent the unit off. Also, they were clearly avoiding the topic of a loaner unit. They would reply to me, but not answer my question about that. This went on for multiple iterations. Since a loaner unit is clearly spelled out in their written warranty statement, I think it's reasonable to expect they live up to that. Had they provided a loaner unit, I wouldn't have cared if it took them 6 months to fix the issue.
Also, quite frankly, it bothers me that someone else got a 3 day turnaround with tracking numbers and shipping paid both ways. Why were these experiences so different? Should we try and deal with the Ohio service center and avoid these guys in Oregon? That's info that would be useful to have.
-
#62 Reply
Posted by
marmad
on 29 Nov, 2014 23:35
-
I think it's reasonable to expect they live up to the terms in their warranty, and that they provide some reasonable expectation as to turnaround time.
I've just read their US warranty - and I'm not quite sure how they didn't live up to the terms. It's certainly nice to have an idea of the turnaround time, but I've not found it to be standard behavior when sending things for repair.
Also, they were clearly avoiding the topic of a loaner unit. They would reply to me, but not answer my question about that. This went on for multiple iterations. Since a loaner unit is clearly spelled out in their written warranty statement, I think it's reasonable to expect they live up to that. Had they provided a loaner unit, I wouldn't have cared if it took them 6 months to fix the issue.
Sorry, but you're misstating what it actually says:
"For service that will exceed this time Rigol will
attempt to provide loaner units to be used during the repair cycle.
Final availability of loaner units will be at Rigol’s discretion."
If you think that guarantees you a loaner unit, well... you've got a lot of heartache coming in your warranty repair future.
But I would agree that the Rigol rep(s) should have discussed it with you - even if it was to tell you one wasn't available.
Also, quite frankly, it bothers me that someone else got a 3 day turnaround with tracking numbers and shipping paid both ways. Why were these experiences so different? Should we try and deal with the Ohio service center and avoid these guys in Oregon? That's info that would be useful to have.
Shipping paid both ways is fairly unusual with warranty service - and is certainly not stated in the warranty. But not getting the return tracking number is bizarre (since it certainly had one - nothing worth more than a few bucks is sent without one these days) - that seems like an utter failure of the support staff to relay it to you.
-
#63 Reply
Posted by
Teneyes
on 29 Nov, 2014 23:43
-
For full disclosure:
I have tried to developed a good relationship with Rigol /support with reporting clear,concise bugs over the last few years. So yes the prepaid shipping, was asked for and supplied, That was unusual.
-
#64 Reply
Posted by
motocoder
on 30 Nov, 2014 00:00
-
I think it's reasonable to expect they live up to the terms in their warranty, and that they provide some reasonable expectation as to turnaround time.
I've just read their US warranty - and I'm not quite sure how they didn't live up to the terms. It's certainly nice to have an idea of the turnaround time, but I've not found it to be standard behavior when sending things for repair.
Also, they were clearly avoiding the topic of a loaner unit. They would reply to me, but not answer my question about that. This went on for multiple iterations. Since a loaner unit is clearly spelled out in their written warranty statement, I think it's reasonable to expect they live up to that. Had they provided a loaner unit, I wouldn't have cared if it took them 6 months to fix the issue.
Sorry, but you're misstating what it actually says:
"For service that will exceed this time Rigol will attempt to provide loaner units to be used during the repair cycle. Final availability of loaner units will be at Rigol’s discretion."
If you think that guarantees you a loaner unit, well... you've got a lot of heartache coming in your warranty repair future. But I would agree that the Rigol rep(s) should have discussed it with you - even if it was to tell you one wasn't available.
Also, quite frankly, it bothers me that someone else got a 3 day turnaround with tracking numbers and shipping paid both ways. Why were these experiences so different? Should we try and deal with the Ohio service center and avoid these guys in Oregon? That's info that would be useful to have.
Shipping paid both ways is fairly unusual with warranty service - and is certainly not stated in the warranty. But not getting the return tracking number is bizarre (since it certainly had one - nothing worth more than a few bucks is sent without one these days) - that seems like an utter failure of the support staff to relay it to you.
I looked up the warranty before I sent off the unit, and I do not recall any qualifications about the loaner unit. But it is possible that I just missed it. I checked the wayback machine to see if that page was updated recently, but I don't see any changes there in the last 3 months.
I agree about the shipping - but since they sent me a demo unit and asked me to test it on my network, I would have expected them to at least pick up the return shipping cost on that. They could have sent me a return shipping label with the scope. In fairness, they probably would have picked up the cost on that if I had pressed them, but quite frankly, as the "repair" time was approaching 1 month, I was ready to get my scope back and be done with it. Once I determined that this wasn't a defect in my scope, but rather just a bug in their firmware, I really wanted my scope back. After spending several days testing, buying a replacement switch to eliminate that as a source of the issue, etc., I really felt like it was not unreasonable for them to send me my scope back when requested. They were dragging their feet about sending that back, and actually said they were going to keep and do some more testing. I sent the demo unit back to try and force the issue.
Also, I'm pretty sure you don't, but in the interest of full disclosure, please confirm whether or not you work for Rigol.
-
#65 Reply
Posted by
marmad
on 30 Nov, 2014 02:11
-
I agree about the shipping - but since they sent me a demo unit and asked me to test it on my network, I would have expected them to at least pick up the return shipping cost on that. They could have sent me a return shipping label with the scope.
Absolutely - I agree fully with you about this. But what was the distinction between a loaner and a demo unit? Was the demo non-functional in some way (or not a DS2000)?
They were dragging their feet about sending that back, and actually said they were going to keep and do some more testing.
That IS odd. I wonder if that service center happened to be without functional DS2000s to continue testing with?
Also, I'm pretty sure you don't, but in the interest of full disclosure, please confirm whether or not you work for Rigol.
No, not at all. I'm plenty critical of Rigol from time to time - especially about their lack of communication, information, and support. I certainly don't think you got
good warranty service - I'm just not sure how much it differs with other Chinese equipment manufacturers - as well as a slew of low-cost manufacturers in other tech fields.
-
#66 Reply
Posted by
motocoder
on 30 Nov, 2014 02:38
-
Absolutely - I agree fully with you about this. But what was the distinction between a loaner and a demo unit? Was the demo non-functional in some way (or not a DS2000)?
The only reason they sent me the loaner was because I agreed to run some tests on it on my network. I am quite positive they would never have sent it otherwise. I would have been fine with the unit they sent had they sent it to me 1.5 weeks earlier. The loaner offer in their warranty statement is complete BS, IMHO.
That IS odd. I wonder if that service center happened to be without functional DS2000s to continue testing with?
Could be. If so, tell me that. The most frustrating aspect of this was sending them an email, and waiting a week or more to hear back. Anyone in the customer service business should know that an easy way to make a customer happier is to spend the extra 30 seconds to communicate status to them.