Author Topic: Trying to work out a checksum in a 8 byte data stream.. (Read 22386 times)

ataradov · « **Reply #25 on:** November 10, 2015, 09:54:05 pm »

Have you gotten anywhere with generating arbitrary packets? It would help a lot and will eliminate a lot of guessing. Required patterns:

00 00 00 00 00 00
00 00 00 00 00 01
01 00 00 00 00 00
80 00 00 00 00 00
00 00 00 00 00 80

beyondhelp · « **Reply #26 on:** November 10, 2015, 10:13:34 pm »

just working out how to send and receive data in processing. That way I can record the results easier than just via an arduino. I;m going to spend a couple of hours on it now and I'll let you know how I get on using those bytes.

beyondhelp · « **Reply #27 on:** November 11, 2015, 02:59:59 am »

finally its 3am and i have a test running. Should have the first checksum for 01 00 00 00 00 00 soon I hope

beyondhelp · « **Reply #28 on:** November 11, 2015, 03:32:13 am »

GOT ONE!!!

01 00 00 00 00 00 15 1a

Interestingly I get a reply as well

00 01 00 00 00 00 e4 f5

No idea what that means yet, but I have had that response from other requests.

Shall keep looking but I'll let it run as im off to bed now.

ataradov · « **Reply #29 on:** November 11, 2015, 03:34:28 am »

Quote from: beyondhelp on November 11, 2015, 03:32:13 am

GOT ONE!!!

Now run the rest of my combinations, especially all 00's.

You now have a very powerful tool in your hands.

beyondhelp · « **Reply #30 on:** November 11, 2015, 03:39:17 am »

Thanks, yes its very exciting... Its rather slow however took 20 mins to get that one. I'm sending packets of data every 300ms, so could take as long as 6 hours, but it means my serial port program on the pc (requests are being sent from the arduino) format nicely and so spotting the change in counts when it responds is easy.

Working on 00 00 00 00 00 01 now. but I have to goto bed again ,so shall update some point tomorrow!

Thanks for the help so far...

JimRemington · « **Reply #31 on:** November 11, 2015, 04:07:14 am »

I've had good luck with the CRC reverse engineering program called "reveng", download here: http://sourceforge.net/projects/reveng/

It can decode an arbitrary CRC and also reverses the bitstreams, etc. I gave reveng a couple of tries (with the presumed 16 bit CRC in both byte orders) and it works for three of the messages, but not with four messages. However, it is a bad sign if more than one polynomial is discovered, or if the byte order of the CRC checksum doesn't matter, which may mean that the input data are linearly dependent. I'll keep playing with it a bit, but here is a log of a few attempts, using the first three messages you posted, and finally adding one of the latter messages. In each attempt, the byte order of the presumed checksum is swapped.

Note: in one particularly difficult case I had to swap the NIBBLES of each byte in order to solve the problem!

Code: [Select]

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  010020010101387b 010
020020202041b 010020030303103b
width=16  poly=0x1021  init=0x0a56  refin=false  refout=false  xorout=0x0000  ch
eck=0x95f4  name=(none)

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  0100200101017b38 010
0200202021b04 0100200303033b10
width=16  poly=0x726f  init=0x4be1  refin=true  refout=true  xorout=0x0000  chec
k=0x99ee  name=(none)
width=16  poly=0xadd7  init=0x2e11  refin=true  refout=true  xorout=0x0000  chec
k=0x8e4c  name=(none)

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  010020010101387b 010
020020202041b 010020030303103b 0100200f0f0fd28a
reveng: no models found

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  0100200101017b38 010
0200202021b04 0100200303033b10 0100200f0f0f8ad2
reveng: no models found

beyondhelp · « **Reply #32 on:** November 11, 2015, 08:53:43 am »

I have another one... This took some time to recover:

00 00 00 00 00 01 94 4f

Shall leave if finding 00 00 00 00 00 00 now

Andy Watson · « **Reply #33 on:** November 11, 2015, 01:15:04 pm »

Quote from: beyondhelp on November 11, 2015, 03:32:13 am

GOT ONE!!!

01 00 00 00 00 00 15 1a

Interestingly I get a reply as well

00 01 00 00 00 00 e4 f5

Just one? I'd count that as TWO

beyondhelp · « **Reply #34 on:** November 11, 2015, 05:36:17 pm »

I have another!

00 00 00 00 00 00 b5 5f - took over 4 hours to get that one!

So far, form the list I needed above:
01 00 00 00 00 00 15 1a
00 00 00 00 00 01 94 4f
00 00 00 00 00 00 b5 5f

ataradov · « **Reply #35 on:** November 11, 2015, 05:38:29 pm »

That's great. I'll play with those numbers later today and we'll see what can be done here.

beyondhelp · « **Reply #36 on:** November 11, 2015, 05:48:25 pm »

Awesome, I have double checked and send those, they are all good. Now working on 80 00 00 00 00 00. Hopefully by the time I'm home that will be done as well.

hamster_nz · « **Reply #37 on:** November 11, 2015, 06:52:46 pm »

Sorry... long post....

For all the test vectors I have got from the thread, I've got something that is almost there..

Code: [Select]

00 01 00 00 00 00 => 00 01 00 00 00 00  => e4 f4 should be e4 f5, error 00 01
01 00 20 01 01 01 => 01 00 20 01 01 01  => 7b 38
01 00 20 02 02 02 => 01 00 20 02 02 02  => 1b 04
01 00 20 03 03 03 => 01 00 20 03 03 03  => 3b 10
01 00 20 04 04 04 => 01 00 20 04 04 04  => db 7c
01 00 20 05 05 05 => 01 00 20 05 05 05  => fb 68
01 00 20 06 06 06 => 01 00 20 06 06 06  => 9b 54
01 00 20 07 07 07 => 01 00 20 07 07 07  => bb 40
01 00 20 08 08 08 => 01 00 20 08 09 08  => 6a be
01 00 20 09 09 09 => 01 00 20 09 08 09  => 4a aa
01 00 20 0a 0a 0a => 01 00 20 0a 0b 0a  => 2a 96
01 00 20 0b 0b 0b => 01 00 20 0b 0a 0b  => 0a 82
01 00 20 0c 0c 0c => 01 00 20 0c 0d 0c  => ea ee
01 00 20 0d 0d 0d => 01 00 20 0d 0c 0d  => ca fa
01 00 20 0e 0e 0e => 01 00 20 0e 0f 0e  => aa c6
01 00 20 0f 0f 0f => 01 00 20 0f 0e 0f  => 8a d2
00 01 70 01 22 01 => 00 01 70 01 23 00  => 14 d2 should be 14 d3, error 00 01
00 01 60 00 80 00 => 00 01 60 00 81 00  => 9f 85
00 01 b0 00 20 00 => 00 01 b0 01 21 01  => f3 17 should be f3 16, error 00 01
00 01 b0 01 20 00 => 00 01 b0 00 21 01  => c3 20 should be c3 21, error 00 01
00 01 b0 02 53 00 => 00 01 b0 03 52 01  => 99 24
00 01 b0 03 65 00 => 00 01 b0 02 64 01  => 9a bc
00 01 b0 04 71 00 => 00 01 b0 05 70 01  => bd f6
00 01 b0 05 75 00 => 00 01 b0 04 74 01  => 49 0d
00 01 b0 06 65 00 => 00 01 b0 07 64 01  => 6a 57
00 01 b0 07 6e 00 => 00 01 b0 06 6f 00  => 81 ac should be 81 ad, error 00 01
00 01 b0 08 63 00 => 00 01 b0 09 63 00  => dd c5
00 01 b0 09 65 00 => 00 01 b0 08 65 00  => 4b 58
00 01 b0 0a 72 00 => 00 01 b0 0b 72 00  => ff 9b
00 01 b0 0b 20 00 => 00 01 b0 0a 20 00  => 12 c4 should be 12 c5, error 00 01
00 01 b0 0c 36 00 => 00 01 b0 0d 36 00  => 57 e8 should be 57 e9, error 00 01
00 01 b0 0d 34 00 => 00 01 b0 0c 34 00  => 05 b9 should be 05 b8, error 00 01
00 01 b0 0e 20 00 => 00 01 b0 0f 20 00  => e2 2f should be e2 2e, error 00 01
00 01 b0 0f 20 00 => 00 01 b0 0e 20 00  => d2 18 should be d2 19, error 00 01
00 01 30 00 58 00 => 00 01 30 00 58 01  => 3a 4f
00 01 30 00 01 00 => 00 01 30 00 01 00  => 3c eb
00 01 30 00 02 00 => 00 01 30 00 02 00  => 6f be
00 01 30 00 03 00 => 00 01 30 00 03 00  => 5e 8d
00 01 30 00 04 00 => 00 01 30 00 04 00  => c9 14
00 01 30 00 05 00 => 00 01 30 00 05 00  => f8 27
00 01 30 00 06 00 => 00 01 30 00 06 00  => ab 72
00 01 30 00 07 00 => 00 01 30 00 07 00  => 9a 41
00 01 30 00 08 00 => 00 01 30 00 08 01  => 85 41 should be 85 40, error 00 01
00 01 30 00 09 00 => 00 01 30 00 09 01  => b4 72 should be b4 73, error 00 01
00 01 30 00 0a 00 => 00 01 30 00 0a 01  => e7 27 should be e7 26, error 00 01
00 01 30 00 0b 00 => 00 01 30 00 0b 01  => d6 14 should be d6 15, error 00 01
01 00 00 00 00 00 => 01 00 00 00 00 00  => 15 1b should be 15 1a, error 00 01

It is some sort of obscuring transform in front of a CRC16...

Code: [Select]

short int my_crc(unsigned char *data, short unsigned length)
{
  int i;
  short unsigned crc = 0, crc_low, crc_high;
  for(i = 0; i < length; i++)
     printf("%02x ",data[i]);
  printf("=> ",data[i]);

  if((data[2] & 0x80)) {
    data[3] ^= 0x01;
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }
  if((data[2] & 0x70) == 0x70) {
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }
  if((data[3] & 0x08) ^ (data[4] & 0x80))
    data[4] ^= 0x01;
  if((data[2] & 0x08) ^ (data[3] & 0x08) ^ (data[4] & 0x08))
    data[5] ^= 0x01;

  crc = 0x0000;
  for(i = 0; i < length*8;  i++)
  {
    int mask, index,bit,wrap;
    mask = 0x80>>(i%8);
    index = i/8;
    bit = (data[index] & mask) ? 1 : 0;
    wrap = (crc & 0x8000) ? 1 : 0;
    crc <<= 1;
    if(wrap ^ bit) crc ^= 0x1021;
  }
  /* Display the data) */
  for(i = 0; i < length; i++)
     printf("%02x ",data[i]);
#if 1
  if( data[i] || data[i+1])
     crc ^=  0x1b15 ^ 0x45a0;
#endif
  crc_low  = (crc>>0)&0xFF;
  crc_high = (crc>>8)&0xFF;

  if( crc_low == data[i] && crc_high == data[i+1])
    printf(" => %02x %02x\r\n", data[i], data[i+1]);
  else
    printf(" => %02x %02x should be %02x %02x, error %02x %02x\r\n",
           crc_low, crc_high,
           data[i], data[i+1],
           crc_low^data[i], crc_high^data[i+1]);

  return 0;
}

beyondhelp · « **Reply #38 on:** November 11, 2015, 07:14:24 pm »

Absolutely top work. Honestly I'm way out of my depth with checksums etc, but so grateful for the help I'm being given here. I'm actually amazed at how much work it appears to be to crack this.

Gotta love a challenge eh!

Is there anything more I can do to help? I have a load more captured data I can share.

ataradov · « **Reply #39 on:** November 11, 2015, 07:25:09 pm »

Quote from: beyondhelp on November 11, 2015, 07:14:24 pm

Is there anything more I can do to help? I have a load more captured data I can share.

Capturing more simple cases. Like 01 and 80 in various positions may be helpful.

I don't think it is that complex, it is just something else. Why on earth anyone would do obfuscation like that?

Edit: And may be more useful:
02 00 00 00 00 00
04 00 00 00 00 00
08 00 00 00 00 00
10 00 00 00 00 00
20 00 00 00 00 00
40 00 00 00 00 00

And the same with numbers in the last byte.

This will tell us how individual bits affect the result if they are at the beginning and at the end.

beyondhelp · « **Reply #40 on:** November 11, 2015, 08:13:10 pm »

Ok well its ticking away, a few more hours potentially on this one

hamster_nz · « **Reply #41 on:** November 11, 2015, 08:19:16 pm »

Quote from: beyondhelp on November 11, 2015, 07:14:24 pm

Is there anything more I can do to help? I have a load more captured data I can share.

Yes please, the more test vectors the quicker the result (however, only if you are 100% sure that they are valid - if a bad test slips in it it can be a real bother to weed out!

It definitely has a CRC nature to it:
- You can flip bits in the input and it usually flips the same same set of bits in the checksum
- The bits that are flipped are usually the same that would be flipped in an vanilla CRC16 checksum
- So far when the CRC bits flipped they aren't the same as the simple CRC16 checksum, it can be solved by also flipping some of the following LSBs e.g:

Code: [Select]

  if((data[2] & 0x80)) {
    data[3] ^= 0x01;
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }

and

Code: [Select]

  if((data[3] & 0x08) ^ (data[4] & 0x80))
    data[4] ^= 0x01;
  ....

That transform seems just too structured to be a random coincidence - I just don't have enough data at the moment to get the insight to determine the original pattern.

So I guess the original thinking went along the line of "I want to use a CRC16, but don't want anybody to reverse engineering the protocol. I'll apply a simple 'random' transform to data before I CRC it - that way I have the benefits of a standard CRC & standard polynomial (i.e. can use a CRC library) but it isn't obvious on the wire".

beyondhelp · « **Reply #42 on:** November 11, 2015, 08:26:13 pm »

ok another one to add to the list:

80 00 00 00 00 00 71 53

beyondhelp · « **Reply #43 on:** November 11, 2015, 08:29:53 pm »

These are all valid, as they were generated by the hardware and captured.

Code: [Select]

00 01 70 01 22 01 14 d3
00 01 60 00 f3 01 b4 c9
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 2a 00 01 20
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 d7 01 96 03
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 13 00 2d 8e
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 80 00 9f 85
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 58 00 3a 4f
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 7e 00 51 b4
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 41 00 f0 e7
00 01 00 00 00 00 e4 f5

beyondhelp · « **Reply #44 on:** November 12, 2015, 12:10:21 am »

Ok the last one on that list:

00 00 00 00 00 80 3d cf

Interestingly the bad message counter just doesn't count anything when this checksum is correct. No good or bad messages. It just totally ignores it, however change any bit on the checksum and it then flags a bad message.

So there complete list!

01 00 00 00 00 00 15 1a
00 00 00 00 00 01 94 4f
00 00 00 00 00 00 b5 5f
80 00 00 00 00 00 71 53
00 00 00 00 00 80 3d cf

I shall start it doing some of the other suggestions now.

ataradov · « **Reply #45 on:** November 12, 2015, 12:16:45 am »

Can't wait to get home and start working on this

I love practical brain teasers like that.

beyondhelp · « **Reply #46 on:** November 12, 2015, 01:17:43 am »

haha I can't wait for you to do it as well!!

I've just been looking, but don't really know where to start :s

apelly · « **Reply #47 on:** November 12, 2015, 01:26:46 am »

It's not something like check-digit + crc is it?

beyondhelp · « **Reply #48 on:** November 12, 2015, 01:39:48 am »

Quote from: hamster_nz on November 11, 2015, 08:19:16 pm

That transform seems just too structured to be a random coincidence - I just don't have enough data at the moment to get the insight to determine the original pattern.

So I guess the original thinking went along the line of "I want to use a CRC16, but don't want anybody to reverse engineering the protocol. I'll apply a simple 'random' transform to data before I CRC it - that way I have the benefits of a standard CRC & standard polynomial (i.e. can use a CRC library) but it isn't obvious on the wire".

That makes perfect sense, and is all I can really think it could be. I mean why would you want to make it even more complex?

bson · « **Reply #49 on:** November 12, 2015, 02:46:47 am »

Quote from: hamster_nz on November 11, 2015, 06:52:46 pm

It is some sort of obscuring transform in front of a CRC16...

Maybe it starts at something other than 0... like a simple mod 255 sum of the bytes in the message, or something.


EEVblog Main Site	EEVblog on Youtube	EEVblog on Twitter	EEVblog on Facebook	EEVblog on Odysee

Author Topic: Trying to work out a checksum in a 8 byte data stream.. (Read 22386 times)

Share me