Trying to work out a checksum in a 8 byte data stream..

• #25 Reply
  Posted by ataradov on 10 Nov, 2015 21:54
• Have you gotten anywhere with generating arbitrary packets? It would help a lot and will eliminate a lot of guessing. Required patterns:
  
  00 00 00 00 00 00
  00 00 00 00 00 01
  01 00 00 00 00 00
  80 00 00 00 00 00
  00 00 00 00 00 80
• #26 Reply
  Posted by beyondhelp on 10 Nov, 2015 22:13
• just working out how to send and receive data in processing. That way I can record the results easier than just via an arduino. I;m going to spend a couple of hours on it now and I'll let you know how I get on using those bytes.
• #27 Reply
  Posted by beyondhelp on 11 Nov, 2015 02:59
• finally its 3am and i have a test running. Should have the first checksum for 01 00 00 00 00 00 soon I hope
• #28 Reply
  Posted by beyondhelp on 11 Nov, 2015 03:32
• GOT ONE!!!
  
  01 00 00 00 00 00 15 1a
  
  Interestingly I get a reply as well
  
  00 01 00 00 00 00 e4 f5
  
  No idea what that means yet, but I have had that response from other requests.
  
  Shall keep looking but I'll let it run as im off to bed now.
• #29 Reply
  Posted by ataradov on 11 Nov, 2015 03:34
• Quote from: beyondhelp on 11 Nov, 2015 03:32

  GOT ONE!!!

  Now run the rest of my combinations, especially all 00's.
  
  You now have a very powerful tool in your hands.
• #30 Reply
  Posted by beyondhelp on 11 Nov, 2015 03:39
• Thanks, yes its very exciting... Its rather slow however took 20 mins to get that one. I'm sending packets of data every 300ms, so could take as long as 6 hours, but it means my serial port program on the pc (requests are being sent from the arduino) format nicely and so spotting the change in counts when it responds is easy.
  
  Working on 00 00 00 00 00 01 now. but I have to goto bed again ,so shall update some point tomorrow!
  
  Thanks for the help so far...
• #31 Reply
  Posted by JimRemington on 11 Nov, 2015 04:07
• I've had good luck with the CRC reverse engineering program called "reveng", download here: http://sourceforge.net/projects/reveng/
  
  It can decode an arbitrary CRC and also reverses the bitstreams, etc.  I gave reveng a couple of tries (with the presumed 16 bit CRC in both byte orders) and it works for three of the messages, but not with four messages. However, it is a bad sign if more than one polynomial is discovered, or if the byte order of the CRC checksum doesn't matter, which may mean that the input data are linearly dependent. I'll keep playing with it a bit, but here is a log of a few attempts, using the first three messages you posted, and finally adding one of the latter messages. In each attempt, the byte order of the presumed checksum is swapped.
  
  Note: in one particularly difficult case I had to swap the NIBBLES of each byte in order to solve the problem!
  
  

  Code: [Select]

  C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  010020010101387b 010
020020202041b 010020030303103b
width=16  poly=0x1021  init=0x0a56  refin=false  refout=false  xorout=0x0000  ch
eck=0x95f4  name=(none)

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  0100200101017b38 010
0200202021b04 0100200303033b10
width=16  poly=0x726f  init=0x4be1  refin=true  refout=true  xorout=0x0000  chec
k=0x99ee  name=(none)
width=16  poly=0xadd7  init=0x2e11  refin=true  refout=true  xorout=0x0000  chec
k=0x8e4c  name=(none)

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  010020010101387b 010
020020202041b 010020030303103b 0100200f0f0fd28a
reveng: no models found

C:\Users\Justme\Desktop\Current\REVENG~1.0A>reveng -s -w16  0100200101017b38 010
0200202021b04 0100200303033b10 0100200f0f0f8ad2
reveng: no models found
• #32 Reply
  Posted by beyondhelp on 11 Nov, 2015 08:53
• I have another one... This took some time to recover:
  
  00 00 00 00 00 01 94 4f
  
  Shall leave if finding 00 00 00 00 00 00 now
• #33 Reply
  Posted by Andy Watson on 11 Nov, 2015 13:15
• Quote from: beyondhelp on 11 Nov, 2015 03:32

  GOT ONE!!!
  
  01 00 00 00 00 00 15 1a
  
  Interestingly I get a reply as well
  
  00 01 00 00 00 00 e4 f5
  

  Just one? I'd count that as TWO 
• #34 Reply
  Posted by beyondhelp on 11 Nov, 2015 17:36
• I have another!
  
  00 00 00 00 00 00 b5 5f -  took over 4 hours to get that one!
  
  So far, form the list I needed above:
  01 00 00 00 00 00 15 1a
  00 00 00 00 00 01 94 4f
  00 00 00 00 00 00 b5 5f
  
• #35 Reply
  Posted by ataradov on 11 Nov, 2015 17:38
• That's great. I'll play with those numbers later today and we'll see what can be done here.
• #36 Reply
  Posted by beyondhelp on 11 Nov, 2015 17:48
• Awesome, I have double checked and send those, they are all good. Now working on 80 00 00 00 00 00. Hopefully by the time I'm home that will be done as well.
• #37 Reply
  Posted by hamster_nz on 11 Nov, 2015 18:52
• Sorry... long post....
  
  For all the test vectors I have got from the thread, I've got something that is almost there..
  
  

  Code: [Select]

  00 01 00 00 00 00 => 00 01 00 00 00 00  => e4 f4 should be e4 f5, error 00 01
01 00 20 01 01 01 => 01 00 20 01 01 01  => 7b 38
01 00 20 02 02 02 => 01 00 20 02 02 02  => 1b 04
01 00 20 03 03 03 => 01 00 20 03 03 03  => 3b 10
01 00 20 04 04 04 => 01 00 20 04 04 04  => db 7c
01 00 20 05 05 05 => 01 00 20 05 05 05  => fb 68
01 00 20 06 06 06 => 01 00 20 06 06 06  => 9b 54
01 00 20 07 07 07 => 01 00 20 07 07 07  => bb 40
01 00 20 08 08 08 => 01 00 20 08 09 08  => 6a be
01 00 20 09 09 09 => 01 00 20 09 08 09  => 4a aa
01 00 20 0a 0a 0a => 01 00 20 0a 0b 0a  => 2a 96
01 00 20 0b 0b 0b => 01 00 20 0b 0a 0b  => 0a 82
01 00 20 0c 0c 0c => 01 00 20 0c 0d 0c  => ea ee
01 00 20 0d 0d 0d => 01 00 20 0d 0c 0d  => ca fa
01 00 20 0e 0e 0e => 01 00 20 0e 0f 0e  => aa c6
01 00 20 0f 0f 0f => 01 00 20 0f 0e 0f  => 8a d2
00 01 70 01 22 01 => 00 01 70 01 23 00  => 14 d2 should be 14 d3, error 00 01
00 01 60 00 80 00 => 00 01 60 00 81 00  => 9f 85
00 01 b0 00 20 00 => 00 01 b0 01 21 01  => f3 17 should be f3 16, error 00 01
00 01 b0 01 20 00 => 00 01 b0 00 21 01  => c3 20 should be c3 21, error 00 01
00 01 b0 02 53 00 => 00 01 b0 03 52 01  => 99 24
00 01 b0 03 65 00 => 00 01 b0 02 64 01  => 9a bc
00 01 b0 04 71 00 => 00 01 b0 05 70 01  => bd f6
00 01 b0 05 75 00 => 00 01 b0 04 74 01  => 49 0d
00 01 b0 06 65 00 => 00 01 b0 07 64 01  => 6a 57
00 01 b0 07 6e 00 => 00 01 b0 06 6f 00  => 81 ac should be 81 ad, error 00 01
00 01 b0 08 63 00 => 00 01 b0 09 63 00  => dd c5
00 01 b0 09 65 00 => 00 01 b0 08 65 00  => 4b 58
00 01 b0 0a 72 00 => 00 01 b0 0b 72 00  => ff 9b
00 01 b0 0b 20 00 => 00 01 b0 0a 20 00  => 12 c4 should be 12 c5, error 00 01
00 01 b0 0c 36 00 => 00 01 b0 0d 36 00  => 57 e8 should be 57 e9, error 00 01
00 01 b0 0d 34 00 => 00 01 b0 0c 34 00  => 05 b9 should be 05 b8, error 00 01
00 01 b0 0e 20 00 => 00 01 b0 0f 20 00  => e2 2f should be e2 2e, error 00 01
00 01 b0 0f 20 00 => 00 01 b0 0e 20 00  => d2 18 should be d2 19, error 00 01
00 01 30 00 58 00 => 00 01 30 00 58 01  => 3a 4f
00 01 30 00 01 00 => 00 01 30 00 01 00  => 3c eb
00 01 30 00 02 00 => 00 01 30 00 02 00  => 6f be
00 01 30 00 03 00 => 00 01 30 00 03 00  => 5e 8d
00 01 30 00 04 00 => 00 01 30 00 04 00  => c9 14
00 01 30 00 05 00 => 00 01 30 00 05 00  => f8 27
00 01 30 00 06 00 => 00 01 30 00 06 00  => ab 72
00 01 30 00 07 00 => 00 01 30 00 07 00  => 9a 41
00 01 30 00 08 00 => 00 01 30 00 08 01  => 85 41 should be 85 40, error 00 01
00 01 30 00 09 00 => 00 01 30 00 09 01  => b4 72 should be b4 73, error 00 01
00 01 30 00 0a 00 => 00 01 30 00 0a 01  => e7 27 should be e7 26, error 00 01
00 01 30 00 0b 00 => 00 01 30 00 0b 01  => d6 14 should be d6 15, error 00 01
01 00 00 00 00 00 => 01 00 00 00 00 00  => 15 1b should be 15 1a, error 00 01

  It is some sort of obscuring transform in front of a CRC16...
  
  

  Code: [Select]

  short int my_crc(unsigned char *data, short unsigned length)
{
  int i;
  short unsigned crc = 0, crc_low, crc_high;
  for(i = 0; i < length; i++)
     printf("%02x ",data[i]);
  printf("=> ",data[i]);

  if((data[2] & 0x80)) {
    data[3] ^= 0x01;
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }
  if((data[2] & 0x70) == 0x70) {
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }
  if((data[3] & 0x08) ^ (data[4] & 0x80))
    data[4] ^= 0x01;
  if((data[2] & 0x08) ^ (data[3] & 0x08) ^ (data[4] & 0x08))
    data[5] ^= 0x01;

  crc = 0x0000;
  for(i = 0; i < length*8;  i++)
  {
    int mask, index,bit,wrap;
    mask = 0x80>>(i%8);
    index = i/8;
    bit = (data[index] & mask) ? 1 : 0;
    wrap = (crc & 0x8000) ? 1 : 0;
    crc <<= 1;
    if(wrap ^ bit) crc ^= 0x1021;
  }
  /* Display the data) */
  for(i = 0; i < length; i++)
     printf("%02x ",data[i]);
#if 1
  if( data[i] || data[i+1])
     crc ^=  0x1b15 ^ 0x45a0;
#endif
  crc_low  = (crc>>0)&0xFF;
  crc_high = (crc>>8)&0xFF;

  if( crc_low == data[i] && crc_high == data[i+1])
    printf(" => %02x %02x\r\n", data[i], data[i+1]);
  else
    printf(" => %02x %02x should be %02x %02x, error %02x %02x\r\n",
           crc_low, crc_high,
           data[i], data[i+1],
           crc_low^data[i], crc_high^data[i+1]);

  return 0;
}

• #38 Reply
  Posted by beyondhelp on 11 Nov, 2015 19:14
• Absolutely top work. Honestly I'm way out of my depth with checksums etc, but so grateful for the help I'm being given here. I'm actually amazed at how much work it appears to be to crack this.
  
  Gotta love a challenge eh!
  
  Is there anything more I can do to help? I have a load more captured data I can share.
• #39 Reply
  Posted by ataradov on 11 Nov, 2015 19:25
• Quote from: beyondhelp on 11 Nov, 2015 19:14

  Is there anything more I can do to help? I have a load more captured data I can share.

  Capturing more simple cases. Like 01 and 80 in various positions may be helpful.
  
  I don't think it is that complex, it is just something else. Why on earth anyone would do obfuscation like that?
  
  Edit: And may be more useful:
  02 00 00 00 00 00
  04 00 00 00 00 00
  08 00 00 00 00 00
  10 00 00 00 00 00
  20 00 00 00 00 00
  40 00 00 00 00 00
  
  And the same with numbers in the last byte.
  
  This will tell us how individual bits affect the result if they are at the beginning and at the end.
• #40 Reply
  Posted by beyondhelp on 11 Nov, 2015 20:13
• Ok well its ticking away, a few more hours potentially on this one
• #41 Reply
  Posted by hamster_nz on 11 Nov, 2015 20:19
• Quote from: beyondhelp on 11 Nov, 2015 19:14

  Is there anything more I can do to help? I have a load more captured data I can share.
  

  
  Yes please, the more test vectors the quicker the result (however, only if you are 100% sure that they are valid - if a bad test slips in it it can be a real bother to weed out!
  
  It definitely has a CRC nature to it:
  - You can flip bits in the input and it usually flips the same same set of bits in the checksum
  - The bits that are flipped are usually the same that would be flipped in an vanilla CRC16 checksum
  - So far when the CRC bits flipped they aren't the same as the simple CRC16 checksum, it can be solved by also flipping some of the following LSBs e.g:
  

  Code: [Select]

    if((data[2] & 0x80)) {
    data[3] ^= 0x01;
    data[4] ^= 0x01;
    data[5] ^= 0x01;
  }

  and
  
  

  Code: [Select]

    if((data[3] & 0x08) ^ (data[4] & 0x80))
    data[4] ^= 0x01;
  ....
That transform seems just too structured to be a random coincidence - I just don't have enough data at the moment to get the insight to determine the original pattern.
  
  So I guess the original thinking went along the line of "I want to use a CRC16, but don't want anybody to reverse engineering the protocol. I'll apply a simple 'random' transform to data before I CRC it - that way I have the benefits of a standard CRC & standard polynomial (i.e. can use a CRC library) but it isn't obvious on the wire".
  
• #42 Reply
  Posted by beyondhelp on 11 Nov, 2015 20:26
• ok another one to add to the list:
  
  80 00 00 00 00 00 71 53
  
  
  
• #43 Reply
  Posted by beyondhelp on 11 Nov, 2015 20:29
• These are all valid, as they were generated by the hardware and captured.
  
  

  Code: [Select]

  00 01 70 01 22 01 14 d3
00 01 60 00 f3 01 b4 c9
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 2a 00 01 20
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 d7 01 96 03
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 13 00 2d 8e
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 80 00 9f 85
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 58 00 3a 4f
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5
00 01 00 00 00 00 e4 f5

00 01 70 01 22 01 14 d3
00 01 60 00 7e 00 51 b4
00 01 b0 00 20 00 f3 16
00 01 b0 01 20 00 c3 21
00 01 b0 02 53 00 99 24
00 01 b0 03 65 00 9a bc
00 01 b0 04 71 00 bd f6
00 01 b0 05 75 00 49 0d
00 01 b0 06 65 00 6a 57
00 01 b0 07 6e 00 81 ad
00 01 b0 08 63 00 dd c5
00 01 b0 09 65 00 4b 58
00 01 b0 0a 72 00 ff 9b
00 01 b0 0b 20 00 12 c5
00 01 b0 0c 36 00 57 e9
00 01 b0 0d 34 00 05 b8
00 01 b0 0e 20 00 e2 2e
00 01 b0 0f 20 00 d2 19
00 01 30 00 41 00 f0 e7
00 01 00 00 00 00 e4 f5

• #44 Reply
  Posted by beyondhelp on 12 Nov, 2015 00:10
• Ok the last one on that list:
  
  00 00 00 00 00 80 3d cf
  
  Interestingly the bad message counter just doesn't count anything when this checksum is correct. No good or bad messages. It just totally ignores it, however change any bit on the checksum and it then flags a bad message.
  
  So there complete list!
  
  01 00 00 00 00 00 15 1a
  00 00 00 00 00 01 94 4f
  00 00 00 00 00 00 b5 5f
  80 00 00 00 00 00 71 53
  00 00 00 00 00 80 3d cf
  
  I shall start it doing some of the other suggestions now.
• #45 Reply
  Posted by ataradov on 12 Nov, 2015 00:16
• Can't wait to get home and start working on this I love practical brain teasers like that.
• #46 Reply
  Posted by beyondhelp on 12 Nov, 2015 01:17
• haha I can't wait for you to do it as well!! 
  
  I've just been looking, but don't really know where to start :s
• #47 Reply
  Posted by apelly on 12 Nov, 2015 01:26
• It's not something like check-digit + crc is it?
• #48 Reply
  Posted by beyondhelp on 12 Nov, 2015 01:39
• Quote from: hamster_nz on 11 Nov, 2015 20:19

  That transform seems just too structured to be a random coincidence - I just don't have enough data at the moment to get the insight to determine the original pattern.
  
  So I guess the original thinking went along the line of "I want to use a CRC16, but don't want anybody to reverse engineering the protocol. I'll apply a simple 'random' transform to data before I CRC it - that way I have the benefits of a standard CRC & standard polynomial (i.e. can use a CRC library) but it isn't obvious on the wire".
  

  
  That makes perfect sense, and is all I can really think it could be. I mean why would you want to make it even more complex?
• #49 Reply
  Posted by bson on 12 Nov, 2015 02:46
• Quote from: hamster_nz on 11 Nov, 2015 18:52

  It is some sort of obscuring transform in front of a CRC16...
  

  Maybe it starts at something other than 0... like a simple mod 255 sum of the bytes in the message, or something.