Author Topic: Trying to work out a checksum in a 8 byte data stream..  (Read 22237 times)

0 Members and 1 Guest are viewing this topic.

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Trying to work out a checksum in a 8 byte data stream..
« on: November 10, 2015, 12:41:39 am »
This has been bugging me for a while. I have a piece of hardware that I want to control from a MCU that currently works over a simple serial connection from a piece of software (that wont work in win10!). In order for me to really do this I need to figure out how the checksum works so that I don't have to manually store tons of data when it could easily be calculated on the fly.

Data is sent as blocks of 8 bytes, the last two are checksum. If I alter those the hardware does not respond, if I alter the other 6 bytes the hardware does also not respond and in both cases an 8bit corrupt data counter is increased by 1.

So here is an example of valid data:
01 00 20 01 01 01 7b 38
01 00 20 02 02 02 1b 04
01 00 20 03 03 03 3b 10
01 00 20 04 04 04 db 7c
01 00 20 05 05 05 fb 68
01 00 20 06 06 06 9b 54
01 00 20 07 07 07 bb 40
01 00 20 08 08 08 6a be
01 00 20 09 09 09 4a aa
01 00 20 0a 0a 0a 2a 96
01 00 20 0b 0b 0b 0a 82
01 00 20 0c 0c 0c ea ee
01 00 20 0d 0d 0d ca fa
01 00 20 0e 0e 0e aa c6
01 00 20 0f 0f 0f 8a d2

Somehow the last two bytes are calculated from the first 6. I would have thought it cannot be *that* complicated as the hardware is based on a 16f871 PIC. I have tried some CRC ideas but basically, I'm looking for any ideas that may help me! I have also tried looking for patterns in binary (nothing obvious). Thing is this isn't my strong point hence I post here in the hope that someone with more experience may help me.
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #1 on: November 10, 2015, 12:52:05 am »
This does not appear to be a standard CRC variant.

What sort of control do you have over the data? Can you generate arbitrary payload and get CRC for it?
Alex
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #2 on: November 10, 2015, 12:55:29 am »
No, I think that's the problem. I can create valid data from the program and record it, or record responses from the hardware. But I have no way to generate a checksum for any given bit of data.

Here if it helps is another example, this time responses from the hardware rather than data sent from the software (post above)

00 01 70 01 22 01 14 d3
00 01 60 00 80 00 9f 85
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 58 00 3a 4f
00 01 00 00 00 00 e4 f5
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #3 on: November 10, 2015, 12:58:14 am »
I'm asking if you can ask the program to send any values. Can you generate this sum for 00 00 00 00 00 00 and 01 00 00 00 00 00?
Alex
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #4 on: November 10, 2015, 01:01:11 am »
No, sadly not. The best I can do is something like this:

    00 01 30 00 01 00 3c eb
    00 01 30 00 02 00 6f be
    00 01 30 00 03 00 5e 8d
    00 01 30 00 04 00 c9 14
    00 01 30 00 05 00 f8 27
    00 01 30 00 06 00 ab 72
    00 01 30 00 07 00 9a 41

 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #5 on: November 10, 2015, 01:07:20 am »
If you look at the difference that each bit makes (e.g. from 01 00 20 02 02 02 to 01 00 20 03 03 03)

It looks to always be
20 14

Likewise the difference between these two messages are

01 00 20 04 04 04 db 7c
01 00 20 06 06 06 9b 54
                              40  28

So a message of

00 00 00 02 02 02 has a checksum of 40 28.

So it does look to be some sort of shift and XOR checksum CRC thingy.

You could probably just analyse a few more messages to get the polynomial... really easy if you could get a message with just the last bit that differs.

Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #6 on: November 10, 2015, 01:07:42 am »
One way to get the CRC for 00 00 00 00 00 00 is to send a bunch (65K) of requests like "00 00 00 00 00 00 AA BB", where AA and BB run from 0x00 to 0xff. Find the one your device likes. A simple script will do that in seconds.

This will also let you generate packets at will, which will help a lot.
Alex
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #7 on: November 10, 2015, 01:15:16 am »
That's a clever idea. I may write a quick arduino script to do that and see what happens... I just have to keep hitting it until the bad message counter does not increment.

This has however been so frustrating, on and off I have been trying to get this for a few years, only just occurred to me to ask here now.

hamster_nz, im going to try sending 00 00 00 02 02 02 40 28 now and see if it thinks its valid.

Will post results in a sec.
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #8 on: November 10, 2015, 01:19:56 am »
That's a clever idea.
I did this many times for reverse engineering things.

If you can automate this process and can generate checksums for simple patterns with 1 bit set in different positions, I'm sure we'll find what this CS is.
Alex
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #9 on: November 10, 2015, 01:30:11 am »
Ok so:

[10/11/2015 01:25:00] Written data (COM6)
    01 00 30 00 00 00 fc 37  - message status request
[10/11/2015 01:25:00] Read data (COM6)
    00 01 30 01 0d 00 40 88 - message status response, 1 bad msg, 13 good.
[10/11/2015 01:26:08] Written data (COM6)
    00 00 00 02 02 02 40 28 - hamster_nz data (no response as expected)
[10/11/2015 01:26:28] Written data (COM6)
    01 00 30 00 00 00 fc 37   - message status request
[10/11/2015 01:26:28] Read data (COM6)
    00 01 30 02 0e 00 43 84  - message status response, 2 bad msg, 14 good.

So on that basis, it didn't like that...
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #10 on: November 10, 2015, 01:39:33 am »
If you send "01 00 30 00 00 00 fc 37" ( message status request)  a few times, you will get a reply back that will at some time have one bit different (the number of good messages).

Should be easy to analyse the checksum from there!

Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #11 on: November 10, 2015, 01:49:22 am »
Right doing that now...
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #12 on: November 10, 2015, 01:54:49 am »
Here we go:

[10/11/2015 01:52:33] Written data (COM6)
    01 00 30 00 00 00 fc 37                           ..0...ü7         
[10/11/2015 01:52:33] Read data (COM6)
    00 01 30 00 ff 00 f2 da                           ..0.ÿ.òÚ

Counter rolls over:
         
[10/11/2015 01:52:33] Written data (COM6)
    01 00 30 00 00 00 fc 37                           ..0...ü7         
[10/11/2015 01:52:33] Read data (COM6)
    00 01 30 00 00 00 0d d8                           ..0....Ø         


[10/11/2015 01:52:34] Written data (COM6)
    01 00 30 00 00 00 fc 37                           ..0...ü7         
[10/11/2015 01:52:34] Read data (COM6)
    00 01 30 00 01 00 3c eb                           ..0...<ë         

1st good message.. and so on...

[10/11/2015 01:52:34] Written data (COM6)
    01 00 30 00 00 00 fc 37                           ..0...ü7         
[10/11/2015 01:52:34] Read data (COM6)
    00 01 30 00 02 00 6f be                           ..0...o¾         
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #13 on: November 10, 2015, 02:12:11 am »
I'm having a go now (really should be asleep) of quickly making a program in processing to send serial data to the hardware directly from the pc, rather than arduino. That way its easy for me to see the data sent/received, and to try ataradovs idea.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #14 on: November 10, 2015, 09:45:53 am »
Prediction:

Code: [Select]
00 01 30 00 08 00  => a4 51
00 01 30 00 09 00  => 95 62
00 01 30 00 0a 00  => c6 37
00 01 30 00 0b 00  => f7 04
00 01 30 00 0c 00  => 60 9d
00 01 30 00 0d 00  => 51 ae
00 01 30 00 0e 00  => 02 fb
00 01 30 00 0f 00  => 33 c8

It looks like CRC16, but there seems to be some thing else going on. Could there be extra bytes in the message? or could the first three bytes be packed somehow?
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #15 on: November 10, 2015, 10:21:41 am »
This has been bugging me for a while. I have a piece of hardware that I want to control from a MCU that currently works over a simple serial connection from a piece of software (that wont work in win10!). In order for me to really do this I need to figure out how the checksum works so that I don't have to manually store tons of data when it could easily be calculated on the fly.

Data is sent as blocks of 8 bytes, the last two are checksum. If I alter those the hardware does not respond, if I alter the other 6 bytes the hardware does also not respond and in both cases an 8bit corrupt data counter is increased by 1.

So here is an example of valid data:
01 00 20 01 01 01 7b 38
01 00 20 02 02 02 1b 04
01 00 20 03 03 03 3b 10
01 00 20 04 04 04 db 7c
01 00 20 05 05 05 fb 68
01 00 20 06 06 06 9b 54
01 00 20 07 07 07 bb 40
01 00 20 08 08 08 6a be
01 00 20 09 09 09 4a aa
01 00 20 0a 0a 0a 2a 96
01 00 20 0b 0b 0b 0a 82
01 00 20 0c 0c 0c ea ee
01 00 20 0d 0d 0d ca fa
01 00 20 0e 0e 0e aa c6
01 00 20 0f 0f 0f 8a d2

Somehow the last two bytes are calculated from the first 6. I would have thought it cannot be *that* complicated as the hardware is based on a 16f871 PIC. I have tried some CRC ideas but basically, I'm looking for any ideas that may help me! I have also tried looking for patterns in binary (nothing obvious). Thing is this isn't my strong point hence I post here in the hope that someone with more experience may help me.

Strange...it looks like bit 3 of byte 4 somehow gets xored with bit 1 or byte 5...

This is my code with the test vectors slightly munged by XORing with 00 00 00 00 01 00 for the last eight... I get the right checksums, but can't explain why...

01 00 20 01 01 01  => 7b 38 should be 7b 38, error 00 00
01 00 20 02 02 02  => 1b 04 should be 1b 04, error 00 00
01 00 20 03 03 03  => 3b 10 should be 3b 10, error 00 00
01 00 20 04 04 04  => db 7c should be db 7c, error 00 00
01 00 20 05 05 05  => fb 68 should be fb 68, error 00 00
01 00 20 06 06 06  => 9b 54 should be 9b 54, error 00 00
01 00 20 07 07 07  => bb 40 should be bb 40, error 00 00
01 00 20 08 09 08  => 6a be should be 6a be, error 00 00
01 00 20 09 08 09  => 4a aa should be 4a aa, error 00 00
01 00 20 0a 0b 0a  => 2a 96 should be 2a 96, error 00 00
01 00 20 0b 0a 0b  => 0a 82 should be 0a 82, error 00 00
01 00 20 0c 0d 0c  => ea ee should be ea ee, error 00 00
01 00 20 0d 0c 0d  => ca fa should be ca fa, error 00 00
01 00 20 0e 0f 0e  => aa c6 should be aa c6, error 00 00
01 00 20 0f 0e 0f  => 8a d2 should be 8a d2, error 00 00
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline BradC

  • Super Contributor
  • ***
  • Posts: 2104
  • Country: au
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #16 on: November 10, 2015, 11:17:46 am »
If it were me I'd step through the Windows app in IDA and just watch what it does.. but that's a totally different ballgame.
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #17 on: November 10, 2015, 12:00:46 pm »
If it were me I'd step through the Windows app in IDA and just watch what it does.. but that's a totally different ballgame.

Is it able to reverse at the C level, or just assembly?
 

Online mikerj

  • Super Contributor
  • ***
  • Posts: 3233
  • Country: gb
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #18 on: November 10, 2015, 03:28:13 pm »
If it were me I'd step through the Windows app in IDA and just watch what it does.. but that's a totally different ballgame.

Is it able to reverse at the C level, or just assembly?

IDA=Interactive DisAssembler.  You can only debug at the C level if you have the source code and the appropriate debug files.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26751
  • Country: nl
    • NCT Developments
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #19 on: November 10, 2015, 04:26:34 pm »
There are several ways to tackle this. If the software is written in .Net or Microsoft C chances are you can decompile it back to source. If not even then a decompiler can provide some insight into how a program works.

What works in many cases is using a debugger and see where the checksum gets calculated. It may be they are using a non-zero seed value. Besides IDA there is also olydbg ( http://www.ollydbg.de/ )
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #20 on: November 10, 2015, 08:29:07 pm »
Cool, I had to call it a day and get some sleep, just got home and its half 8pm, and going to have another play with it. A friend of mine who is a bit of a reverse engineering wiz is also having a look, but don't let that put you off.. I'll tease him lots if I can figure it out with help from you all here first ;)
 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #21 on: November 10, 2015, 08:35:06 pm »
Prediction:

Code: [Select]
00 01 30 00 08 00  => a4 51
00 01 30 00 09 00  => 95 62
00 01 30 00 0a 00  => c6 37
00 01 30 00 0b 00  => f7 04
00 01 30 00 0c 00  => 60 9d
00 01 30 00 0d 00  => 51 ae
00 01 30 00 0e 00  => 02 fb
00 01 30 00 0f 00  => 33 c8

It looks like CRC16, but there seems to be some thing else going on. Could there be extra bytes in the message? or could the first three bytes be packed somehow?

Results:
    00 01 30 00 08 00 85 40
    00 01 30 00 09 00 b4 73
    00 01 30 00 0a 00 e7 26
    00 01 30 00 0b 00 d6 15
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #22 on: November 10, 2015, 08:58:17 pm »
Prediction:

Code: [Select]
00 01 30 00 08 00  => a4 51
00 01 30 00 09 00  => 95 62
00 01 30 00 0a 00  => c6 37
00 01 30 00 0b 00  => f7 04
00 01 30 00 0c 00  => 60 9d
00 01 30 00 0d 00  => 51 ae
00 01 30 00 0e 00  => 02 fb
00 01 30 00 0f 00  => 33 c8

It looks like CRC16, but there seems to be some thing else going on. Could there be extra bytes in the message? or could the first three bytes be packed somehow?

Results:
    00 01 30 00 08 00 85 40
    00 01 30 00 09 00 b4 73
    00 01 30 00 0a 00 e7 26
    00 01 30 00 0b 00 d6 15

Humm all out by 21 11 ... eg a4 51 xor 21 11 = 85 40.....
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Online nali

  • Frequent Contributor
  • **
  • Posts: 656
  • Country: gb
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #23 on: November 10, 2015, 09:37:54 pm »
This has been bugging me for a while. I have a piece of hardware that I want to control from a MCU that currently works over a simple serial connection from a piece of software (that wont work in win10!). In order for me to really do this I need to figure out how the checksum works so that I don't have to manually store tons of data when it could easily be calculated on the fly.

Data is sent as blocks of 8 bytes, the last two are checksum. If I alter those the hardware does not respond, if I alter the other 6 bytes the hardware does also not respond and in both cases an 8bit corrupt data counter is increased by 1.

So here is an example of valid data:
01 00 20 01 01 01 7b 38
01 00 20 02 02 02 1b 04
01 00 20 03 03 03 3b 10
01 00 20 04 04 04 db 7c
01 00 20 05 05 05 fb 68
01 00 20 06 06 06 9b 54
01 00 20 07 07 07 bb 40
01 00 20 08 08 08 6a be
01 00 20 09 09 09 4a aa
01 00 20 0a 0a 0a 2a 96
01 00 20 0b 0b 0b 0a 82
01 00 20 0c 0c 0c ea ee
01 00 20 0d 0d 0d ca fa
01 00 20 0e 0e 0e aa c6
01 00 20 0f 0f 0f 8a d2

Somehow the last two bytes are calculated from the first 6. I would have thought it cannot be *that* complicated as the hardware is based on a 16f871 PIC. I have tried some CRC ideas but basically, I'm looking for any ideas that may help me! I have also tried looking for patterns in binary (nothing obvious). Thing is this isn't my strong point hence I post here in the hope that someone with more experience may help me.

Well if you map out the checksums in binary there are definitely some bit patterns in there, you can see some of the binary counting in some of the individual bits i.e. 01010101 00110011 00001111 etc. The one exception to that is bit 4 of the checksum which changes pattern on the 4th row  :-//


     
01 00 20 01 01 01 7b 38 0111101100111000
01 00 20 02 02 02 1b 04 0001101100000100
01 00 20 03 03 03 3b 10 0011101100010000
01 00 20 04 04 04 db 7c 1101101101111100
01 00 20 05 05 05 fb 68 1111101101101000
01 00 20 06 06 06 9b 54 1001101101010100
01 00 20 07 07 07 bb 40 1011101101000000
01 00 20 08 08 08 6a be 0110101010111110
01 00 20 09 09 09 4a aa 0100101010101010
01 00 20 0a 0a 0a 2a 96 0010101010010110
01 00 20 0b 0b 0b 0a 82 0000101010000010
01 00 20 0c 0c 0c ea ee 1110101011101110
01 00 20 0d 0d 0d ca fa 1100101011111010
01 00 20 0e 0e 0e aa c6 1010101011000110
01 00 20 0f 0f 0f 8a d2 1000101011010010

 

Offline beyondhelpTopic starter

  • Contributor
  • Posts: 46
Re: Trying to work out a checksum in a 8 byte data stream..
« Reply #24 on: November 10, 2015, 09:50:33 pm »

Humm all out by 21 11 ... eg a4 51 xor 21 11 = 85 40.....

On that basis do you get the same checksum with this:

00 01 70 01 22 01 14 d3

or

00 01 00 00 00 00 e4 f5
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf