Author Topic: Hacking the Rigol MSO5000 series oscilloscopes  (Read 179314 times)

BobRyan, Sparky, w5cpu and 3 Guests are viewing this topic.

Offline delfinom

  • Contributor
  • Posts: 42
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1175 on: August 20, 2019, 04:45:34 am »
Here's a new bspatch that should disable two callbacks to rigol. But I could only test that it stopped storing the response in /tmp/firmware.xml right now.

Interestingly both callbacks are two different urls and both don't work.

This should be applied on a clean .04.08 appEntry, final md5sum edb0207efa63aec9d801036521b5452a


Most likely the freeze is due to a connection issue back to the rigol chinese site (and the randomness with the great firewall and peering) and from what I can tell its executing it awkwardly on the ui thread without much concern for what happens if the http connection stalls.




On the plus side, given how...crappy the update check is, I'm pretty sure another update/attack vector in the future is simply MITMing the http requests asking for version.
« Last Edit: August 20, 2019, 04:53:02 am by delfinom »
 
The following users thanked this post: thm_w, tcottle, Shodge, NoisyBoy, serg_77

Offline NoisyBoy

  • Regular Contributor
  • *
  • Posts: 167
  • Country: us
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1176 on: August 20, 2019, 05:03:03 am »
Thanks delfinom, you rock  :-+

You think perhaps something erroneous got stored in the firmware.xml response file on the affected scopes and that's what caused the hang?  And if so, you think there's a way to copy over a good copy and fix the problem without going through re-patching?  Once the freeze occurs, even the original 04.04 and 04.08 without patch would cause the freeze, that's what leads me to think some erroneous data got stored somewhere.

If not, there's no big deal.  It is easy to re-patch just to remove the phone home "features", especially given how infrequent patches are released.

Again, thanks for all your hard work to look into this matter, it is much appreciated..
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1177 on: August 20, 2019, 08:30:27 am »
I think its pretty certain that the scope phones home everytime it can.

One thing it does in that phone call, i think, is to send a RSA encrypted pack that contains some relevant data from the /data dir. Personal data: keys, licenses,etc

If someone wants to put a wireshark to work we can verify this info.

Mabl, no pissing contest here, but i released the SCPI commands in the general Rigol all SCPI commands thread. I'll try and check with yours.
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1178 on: August 20, 2019, 08:33:22 am »
 Maybe after a successful home call it sets a flag until further changes demand a new phone call or a online update is triggered
 

Offline ebastler

  • Super Contributor
  • ***
  • Posts: 3040
  • Country: de
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1179 on: August 20, 2019, 08:39:17 am »
I think its pretty certain that the scope phones home everytime it can.

One thing it does in that phone call, i think, is to send a RSA encrypted pack that contains some relevant data from the /data dir. Personal data: keys, licenses,etc

If someone wants to put a wireshark to work we can verify this info.

There is also a non-technical solution to this, which would help to work around the RSA encryption: Since Rigol sell this scope in Europe, they have to adhere to the GDPR data protection/privacy regulations. They can't collect any personal data without informing you about the data they use, the purpose of that exercise, the storage duration etc.. And yes, if they collect a unique identifier like the scope's serial number, and potentially link that back to the buyer via sales records from their distributors, that's personal information.

Could someone who bought the MSO5000 in Europe file an information request with Rigol Europe?
 

Offline Mr. Scram

  • Super Contributor
  • ***
  • Posts: 8153
  • Country: 00
  • Display aficionado
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1180 on: August 20, 2019, 08:44:08 am »
An oscilloscope calling home. Sometimes I hate the era we live in.  :palm:
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1181 on: August 20, 2019, 08:49:28 am »
We can also change the key... and the info will be worthless.
 

Offline Fungus

  • Super Contributor
  • ***
  • Posts: 10178
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1182 on: August 20, 2019, 08:52:09 am »
I think its pretty certain that the scope phones home everytime it can.

I dunno about "every time it can" but it looks like it does something at bootup.

If someone wants to put a wireshark to work we can verify this info.

Maybe it's just checking for updates. Has anybody seen any sort of "new updates are available" message on one of these?

Tracking how much hacking is going on seems pointless (it's probably in the "90%" region) but that doesn't mean they aren't doing it.

Making the 'scope hang at bootup is a bug though, it needs fixing. A wireshark session would be very informative, if only to figure out what it's doing and how to tweak routers to block it so it boots up as fast as possible.


 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1183 on: August 20, 2019, 09:03:11 am »
We can patch out the home calling but i don't appreciate patching very much. Not future proof.  ;)
 

Offline delfinom

  • Contributor
  • Posts: 42
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1184 on: August 20, 2019, 11:17:37 am »
Thanks delfinom, you rock  :-+

You think perhaps something erroneous got stored in the firmware.xml response file on the affected scopes and that's what caused the hang?  And if so, you think there's a way to copy over a good copy and fix the problem without going through re-patching?  Once the freeze occurs, even the original 04.04 and 04.08 without patch would cause the freeze, that's what leads me to think some erroneous data got stored somewhere.

If not, there's no big deal.  It is easy to re-patch just to remove the phone home "features", especially given how infrequent patches are released.

Again, thanks for all your hard work to look into this matter, it is much appreciated..

A good one won't fix it because it's placed in /tmp which is nuked on reboot. The file itself is most likely not causing it, but rather the http connection going outbound through the great firewall of china.


I think its pretty certain that the scope phones home everytime it can.

I dunno about "every time it can" but it looks like it does something at bootup.

If someone wants to put a wireshark to work we can verify this info.

Maybe it's just checking for updates. Has anybody seen any sort of "new updates are available" message on one of these?

Tracking how much hacking is going on seems pointless (it's probably in the "90%" region) but that doesn't mean they aren't doing it.

Making the 'scope hang at bootup is a bug though, it needs fixing. A wireshark session would be very informative, if only to figure out what it's doing and how to tweak routers to block it so it boots up as fast as possible.

Well there are two callbacks.
One is definitely an update check, and doesn't send anything other than your scope serial number.

The other is a "upload" function but  it's not set to upload anything that would reveal it's hacked since that is only in appEntry with no file system modifications.
More than likely it's a diagnostic/troubleshooting function. But I can't even find where it's triggered and is most likely not in use.
« Last Edit: August 20, 2019, 11:37:55 am by delfinom »
 

Offline Mr. Scram

  • Super Contributor
  • ***
  • Posts: 8153
  • Country: 00
  • Display aficionado
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1185 on: August 20, 2019, 11:30:59 am »
Having a device in the network independently upload data or being able to do so is definitely a concern.
 

Offline Fungus

  • Super Contributor
  • ***
  • Posts: 10178
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1186 on: August 20, 2019, 11:50:42 am »
Well there are two callbacks.
One is definitely an update check, and doesn't send anything other than your scope serial number.

Does it do that during boot, shortly after power-on?

 

Offline NoisyBoy

  • Regular Contributor
  • *
  • Posts: 167
  • Country: us
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1187 on: August 20, 2019, 02:37:53 pm »
At boot, a few seconds after all the UI are up, after you get the network attached message.  Troubleshooting the bug that froze my scope while it connects to Rigol.com is what made me discover this phone-home behavior.
 

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1188 on: August 20, 2019, 03:34:13 pm »
Mabl, no pissing contest here, but i released the SCPI commands in the general Rigol all SCPI commands thread. I'll try and check with yours.

Oh okay. I did not see or find that thread. It was just a side product of the python wrapper that I'm working on.

EDIT: Found it https://www.eevblog.com/forum/testgear/lists-of-rigol-scpi-commands/
« Last Edit: August 20, 2019, 04:40:43 pm by mabl »
 

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1189 on: August 20, 2019, 04:35:05 pm »
Well there are two callbacks.
One is definitely an update check, and doesn't send anything other than your scope serial number.

The other is a "upload" function but  it's not set to upload anything that would reveal it's hacked since that is only in appEntry with no file system modifications.
More than likely it's a diagnostic/troubleshooting function. But I can't even find where it's triggered and is most likely not in use.

Hmm I still don't get this.

The function at 0x000c4fe4 downloads from  http://www.rigol.com/Support/ProductUpgradeStatue?SN=%1&result=1 to also ask for a firmware.xml.   It downloads the content via HTTP  to /tmp/firmware.xml. For me, currently the downloaded content is

Code: [Select]
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<h1>301 Moved Permanently</h1>
<p>The requested resource has been assigned a new permanent URI.</p>
<hr/>Powered by Tengine</body>
</html>

EDIT: Note the typo and the change of URL to what skip reported:

[...]  scope does a regular http (not https) get of "/Support/ProductUpgradeFile?sn=MS5xxxxxxxxxx&hardware=1.0&behaviour=soft&software=00.01.01.02.03 HTTP/1.1". 

Function at 0x000c46f4: Triggers on manual update, triggers the dowload of firmware.xml via 0x000c4fe4 via an unclear mechanism. It also generates the hardware/behaviour/software strings for the URI which gets extended to the above URI somewhere. It contains references to http://www.rigol.com/up.aspx?act=%1&filename=%2.dat. with the %1="up". It is unclear when this triggers. It looks like an XML path lookup of "storage/firmware" is involved somehow.

EDIT2: Yep it is XML. It actually loads the URLs from /rigol/resource/dsometa.xml, and these sound far more up-to-date.

Code: [Select]
<firmware>http://www.rigol.com/Support/ProductUpgradeFile?sn=%1$hardware=%2$behaviour=%3$software=%4</firmware>
<uploadurl>http://www.rigol.com/up.aspx?act=%1$filename=%2</uploadurl>

That upload-url makes me a bit squeezy.


In any case, it fails to parse this firmware.xml, probably due to its wrong content, and fails with "ASSERT: "false" in file cmetanode.cpp, line 121" on stdout. Since appEntry exits, the GUI appears to hang.

I would guess that on some machines, the update check triggers automatically without manual update check, gets a "mal-formed" XML and then appEntry dies. I don't get why my scope doesn't do it however...


The backtrace to 0x000c4fe4 when executing a manual update (after breakpoint at 0x000c46f4 has triggered)

Code: [Select]
#0  0x000c4fe4 in ?? ()
#1  0xb62fec28 in QMetaObject::activate(QObject*, int, int, void**) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#2  0x000d42e8 in CHttp::sigFinish(QNetworkReply::NetworkError) ()
#3  0x000cf3f0 in ?? ()
#4  0x000cf1bc in ?? ()
#5  0x000d3e88 in ?? ()
#6  0xb62fec28 in QMetaObject::activate(QObject*, int, int, void**) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#7  0xb6630770 in ?? () from target:/rigol/Qt5.5/lib/libQt5Network.so.5
#8  0xb66907b4 in ?? () from target:/rigol/Qt5.5/lib/libQt5Network.so.5
#9  0xb62fc8c4 in QMetaCallEvent::placeMetaCall(QObject*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#10 0xb630006c in QObject::event(QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#11 0xb6bb76e8 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Widgets.so.5
#12 0xb6bbcccc in QApplication::notify(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Widgets.so.5
#13 0xb62ce5bc in QCoreApplication::notifyInternal(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#14 0xb62d151c in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#15 0xb6325ec0 in QEventDispatcherUNIX::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#16 0xb5914ba0 in QUnixEventDispatcherQPA::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/plugins/platforms/libqlinuxfb.so
#17 0xb62cc3f0 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#18 0xb62cc81c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#19 0xb62d43e0 in QCoreApplication::exec() () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#20 0x0004334c in ?? ()
#21 0xb5da7874 in __libc_start_main () from target:/lib/libc.so.6
#22 0x00042d98 in ?? ()


BTW. The backtrace of the assert is the following
Code: [Select]
#0  0xb5dbfb48 in raise () from target:/lib/libc.so.6
#1  0xb5dc4ed8 in abort () from target:/lib/libc.so.6
#2  0xb60f1018 in QMessageLogger::fatal(char const*, ...) const () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#3  0xb60ec660 in qt_assert(char const*, char const*, int) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#4  0x0087256c in ?? ()
#5  0x0087065c in ?? ()
#6  0x000c5b28 in ?? ()
#7  0x000c50a0 in ?? ()
#8  0x000d2544 in ?? ()
#9  0xb62fec28 in QMetaObject::activate(QObject*, int, int, void**) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#10 0x000d42e8 in CHttp::sigFinish(QNetworkReply::NetworkError) ()
#11 0x000cf3f0 in ?? ()
#12 0x000cf1bc in ?? ()
#13 0x000d3e88 in ?? ()
#14 0xb62fec28 in QMetaObject::activate(QObject*, int, int, void**) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#15 0xb6630770 in ?? () from target:/rigol/Qt5.5/lib/libQt5Network.so.5
#16 0xb66907b4 in ?? () from target:/rigol/Qt5.5/lib/libQt5Network.so.5
#17 0xb62fc8c4 in QMetaCallEvent::placeMetaCall(QObject*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#18 0xb630006c in QObject::event(QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#19 0xb6bb76e8 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Widgets.so.5
#20 0xb6bbcccc in QApplication::notify(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Widgets.so.5
#21 0xb62ce5bc in QCoreApplication::notifyInternal(QObject*, QEvent*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#22 0xb62d151c in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#23 0xb6325ec0 in QEventDispatcherUNIX::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#24 0xb5914ba0 in QUnixEventDispatcherQPA::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/plugins/platforms/libqlinuxfb.so
#25 0xb62cc3f0 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#26 0xb62cc81c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#27 0xb62d43e0 in QCoreApplication::exec() () from target:/rigol/Qt5.5/lib/libQt5Core.so.5
#28 0x0004334c in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#29 0xb5da7874 in __libc_start_main () from target:/lib/libc.so.6
#30 0x00042d98 in ?? ()

Where 0x000c5b28 is inside the firmware.xml parser at 0x000c5aac, which extracts firmware comments, version and url for download.

« Last Edit: August 20, 2019, 07:50:35 pm by mabl »
 
The following users thanked this post: thm_w, luma, NoisyBoy, whatisthis

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1190 on: August 20, 2019, 06:25:11 pm »
mabl, i think yours isnt dying because the update check isnt being triggered. Or did you try it manually? Mine also doesnt hang when i trigger it manually.
 
The following users thanked this post: NoisyBoy

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1191 on: August 20, 2019, 06:27:20 pm »
Mine (firmware 00.01.01.04.08) hangs on manual update trigger, with that wrong content in the firmware.xml.  The old path to download firmware.xml still works though.
 
The following users thanked this post: NoisyBoy

Offline delfinom

  • Contributor
  • Posts: 42
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1192 on: August 20, 2019, 06:41:20 pm »
Quote
I would guess that on some machines, the update check triggers automatically without manual update check, gets a "mal-formed" XML and then appEntry dies. I don't get why my scope doesn't do it however...

Yea, I triggered it manually and didn't have it die, same exact file contents you have and that same typoed url.


I also wonder if the freeze at bootup has to do with the fact the online update button press is also persisted, if you press the button it greys out, and if you were to reboot the scope, it stays greyed out for some time still.
But I did try and it had no effect on it freezing.



I see you are much further along by running gdb on the scope :P
I am probably just going to leave it at just patching out the http requests and calling it a day. They serve no useful function once you start using the changes.
« Last Edit: August 20, 2019, 06:43:43 pm by delfinom »
 
The following users thanked this post: NoisyBoy

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1193 on: August 20, 2019, 06:58:17 pm »
Yea, I triggered it manually and didn't have it die, same exact file contents you have and that same typoed url.
Strange.


I see you are much further along by running gdb on the scope :P

Oh that was surprisingly simple. I just compiled gdbserver in gdb-7.12 with the xilinx toolchain (attached to this post). On the scope I just run

Code: [Select]
$ ./gdbserver localhost:30000 /rigol/appEntry -run

And on the client, using a cross-toolchain (gdb version is 8.2, so its is backward compatible) I use

Code: [Select]
$ arm-linux-gnueabihf-gdb

target remote ip:30000
b *0xaddress
layout asm
c
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1194 on: August 20, 2019, 07:09:25 pm »
 Can i use IDA Pro as client, mabl?
 

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1195 on: August 20, 2019, 07:13:05 pm »
I do not know, but it looks like it, see https://www.hex-rays.com/products/ida/support/idadoc/1343.shtml.
Unfortunately, Ghidra  does not yet have support. I just used the command line manually.
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1196 on: August 20, 2019, 07:19:07 pm »
That's a PITA. Can you single step, step into, step over? All my previous mso5000 investigations were done with jtag but i dont want to open mine for now...

Of course the specs say IDA work but i wanted full testimony in this particular case.

Thankx for the gdbserver.
 

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1197 on: August 20, 2019, 07:22:25 pm »
I only tried step (command stepi) on gdb and that works. Backtrace (bt) also works. I did not yet try step-over(nexti). I'm no gdb wizard :-D
 

Offline mabl

  • Regular Contributor
  • *
  • Posts: 100
  • Country: 00
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1198 on: August 20, 2019, 07:41:13 pm »
I also wonder if the freeze at bootup has to do with the fact the online update button press is also persisted, if you press the button it greys out, and if you were to reboot the scope, it stays greyed out for some time still.
But I did try and it had no effect on it freezing.

Interesting. The Power-On state set to "Last" now restores the scope state. I thought that did not work in the last firmware version?  :-+

It looks like the scope writes the current state (a 1K field of memory) approximately every minute to /rigol/data/stat.dat  (independently of the power on state setting).  That mount point has the realtime option:

Code: [Select]
/dev/ubi1_0 on /rigol/data type ubifs (rw,sync,relatime)

I wonder how long this small ubi partition will survive? And there are the calibration files and licenses on it....
« Last Edit: August 20, 2019, 07:59:49 pm by mabl »
 

Offline tv84

  • Frequent Contributor
  • **
  • Posts: 925
  • Country: pt
Re: Hacking the Rigol MSO5000 series oscilloscopes
« Reply #1199 on: August 20, 2019, 08:03:06 pm »
It looks like the scope writes the current state (a 1K field of memory) approximately every minute to /rigol/data/stat.dat  (independently of the power on state setting).  That mount point has the realtime option:

It's been a long time since I saw that but I have the impression that all those constant writings are in the FRAM copy and not in the NAND.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf