Author Topic: Resolving contention - create a unique/random sequence from a 96bit ID? use CRC?  (Read 3790 times)

0 Members and 1 Guest are viewing this topic.

Online incfTopic starter

  • Regular Contributor
  • *
  • Posts: 154
  • Country: us
  • ASCII > UTF8
I've got a dozens of MCUs that don't have a hardware random number generator.

These dozens of MCUs operate power supplies that all "fight" for their share of power from a current limited supply by monitoring current and voltage. (Perhaps a bad idea, but this post is about resolving that)

Each MCU has a unique 96-bit ID. They cannot communicate with one another.

I'm curious what are some "good" ways to use the unique ID to generate a random and completely unique sequences of numbers?


In my specific application, I might have each power supply have a random restart/startup/soft-start delay + random brown out reset voltage somehow based on the unique ID (and if the power rail voltage went too low, some of the power supplies would cut out first, and would restart more slowly than others). The goal is to make each unit behave differently at different times. Maybe ADC noise might help to make things random.

I feel like this problem has been solved before.

I do have a CRC generator. Maybe a running CRC each millisecond based on the unique ID, XORed with some ADC noise? (the 4-least significant bits of all the ADC channels XORed together overtop of different portions of the ID)
« Last Edit: January 29, 2025, 11:24:21 pm by incf »
 

Online westfw

  • Super Contributor
  • ***
  • Posts: 4376
  • Country: us
Re: Resolving contention - create a unique/random sequence from a 96-bit ID?
« Reply #1 on: January 29, 2025, 11:01:26 pm »
Quote
what are some "good" ways to use the unique ID to generate a random and completely unique sequences of numbers?
In theory, you can just use your unique ID to seed a standard  Pseudo-Random Number Generator.

The usual problems are that the "unique IDs" are not the same size as most PRNGs, so you can't use the whole ID, and the IDs are NOT "random", but instead include things like wafer number and die position and such (so picking a subset of the ID for the RNG seed is "fraught".)

You can probably run the ID through some sort of hash function that reduces the size (a CRC would be one choice; various crypto algorithms another.  But you'd need to do some experimenting to see what sort of collision rate you get, especially after whatever your code does to reduce the PRNG to a manageble size.  ("x" having few collisions doesn't necessarily mean that "x mod 100" has few collisions.)
Or you could just look for a 96bit PRNG (though it might have the same "reduction problem."
 

Online westfw

  • Super Contributor
  • ***
  • Posts: 4376
  • Country: us
This doc contains LFSR tap info for RNGs up to 168 bits (!)
https://docs.amd.com/v/u/en-US/xapp052
 
The following users thanked this post: incf

Online incfTopic starter

  • Regular Contributor
  • *
  • Posts: 154
  • Country: us
  • ASCII > UTF8
This doc contains LFSR tap info for RNGs up to 168 bits (!)
https://docs.amd.com/v/u/en-US/xapp052
That document  "knocked my socks off" - I would have never thought to search for that from Xilinx/AMD. This seems a lot easier than pursuing CRCZoo looking for max length polynomials.

(While I don't know anything about LFSRs except for having read a story involving old gambling machines, I'm sure I can figure it out)
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 16098
  • Country: fr
Yes my first approach would also be to seed a decent PRNG with the unique ID.
If you can't customize the PRNG to use a 96-bit seed but don't want to truncate the UID arbitrarily, you can reduce the latter to say, 64 bits or 32 bits, with a *much* better result than truncating, using a Fletcher checksum on the UID.
This is exactly what I do personally to generate 32-bit UIDs for a communication protocol from 96-bit UIDs (coming from STM32 MCUs).

https://en.wikipedia.org/wiki/Fletcher%27s_checksum
 
The following users thanked this post: woofy

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 7441
  • Country: fi
    • My home page and email address
Apologies to those who have heard me describe this before, but:

Xorshift64* is a very simple LFSR pseudo-random number generator, whose upper 32-40 bits pass all known randomness tests, and is thus better than e.g. the industry standard, Mersenne Twister (MT19937).

It is not cryptographically secure, but it is random enough even for scientific simulations and numerical work; again, both faster and "more random" than the industry standard.  Just do not assume that the next number in a sequence cannot be predicted, as that would require a cryptographically secure PRNG.

It uses 64 bit state, and has a period of 2⁶⁴-1.  The only disallowed state is all zeros.  (No other state will ever lead to the all-zeros state, so all other seed states will give the full sequence of 2⁶⁴-1 numbers.)

The state is updated using three exclusive OR operations and binary shifts:
    state = state ^ (state >> 12);
    state = state ^ (state << 25);
    state = state ^ (state >> 27);
To generate the PRNG corresponding to the (old or new) state, it is multiplied by a 64-bit value 2685821657736338717, and bits 32..63 (or 24..63) are used:
    number = (uint64_t)(state * UINT64_C(2685821657736338717)) >> 32;

To use a 96-bit seed, you can use the first 32 bits and fixed (nonzero) lower 32 bits as the initial seed, then update the state roughly dozen times.  Then, exclusive-OR the state with the next 32 bits (combined with fixed low 32 bits to form a 64-bit value), and update the state some dozen times or so again.  Repeat that with the last 32 bits, too.  The result is suitable for use as a Xorshift64* PRNG state, yielding the full 2⁶⁴-1 sequence from that point forwards.

On a 32-bit architecture, each state update involves nine bit shifts (by 5, 20, or 25 bits left/up; or by 7, 12, or 27 bits right/down), six exclusive-OR operations, and three OR operations, using four 32-bit registers.  The multiplication is fast if you have a 32×32=64-bit or 32×32=high/low 32-bit operation, but slower if you only have a truncating 32×32=32-bit multiplication.  (On ARMv6-m/Cortex-M0/M0+ it is slow, but on ARMv7e-m/Cortex-M3/M4/M7 and better it is just four instructions: muls, mla, umull, and adds.)

Example C code:
Code: [Select]
#include <stdint.h>

uint32_t  prng_u32(uint64_t *state) {
    uint64_t  x = *state;
    x ^= x >> 12;
    x ^= x << 25;
    x ^= x >> 27;
    *state = x;
    return ((x * UINT64_C(2685821657736338717))) >> 32;
}

uint64_t  hash_96to64(const uint32_t seed[3], const uint32_t fixed[3], const uint8_t rounds[3]) {
    union {
        uint64_t  u64;
        uint32_t  u32[2];
    } state = { .u32 = { 0, 0 } };

    uint_fast8_t  w = 2;
    do {
        state.u32[0] = state.u32[0] ^ fixed[w];
        state.u32[1] = state.u32[1] ^ seed[w];

        uint_fast8_t  r = rounds[w];
        do {
            state.u64 ^= state.u64 >> 12;
            state.u64 ^= state.u64 << 25;
            state.u64 ^= state.u64 >> 27;
        } while (r-->0);
    } while (w-->0);

    return state.u64;
}
The hash_96to32() is a parametrized hash function, letting you tune each use case using the fixed and rounds parameters.  I've found that something like a dozen iterations suffices for a full avalanche effect (at least half the bits differing, whenever the seed value differs by one bit).  Each of the fixed values should be nonzero, to ensure the state update rounds have full effect; preferably with both set and clear bits in the least and most significant bits (instead of all-zeros or all-ones).

While I am not a mathematician, nor a cryptologist, a lot of stuff I do relies on "random" pseudorandom number sequences, and me understanding the details, so I've had to learn to work with the various methods.  Linear feedback shift registers are definitely the fastest.  As Xorshift and Xoroshiro variants show, they can be extremely random, too.  The two industry benchmarks for empirical randomness testing are the TestU01 framework's BigCrush test suite (which includes over a hundred individual tests), and the Diehard tests by Marsaglia et al. and its derivatives like Dieharder.  Using only 32 to 40 high bits of Xorshift64* is not the absolute best (least "weak" results in Dieharder while passing all BigCrush tests), nor the fastest, but it really has no "poor" seeds (except for zero, which is not allowed, and will not occur by itself during normal operation), while the better ones tend to suffer from some "poor" seed states generating less-random sequences so that such seeds need to be "avoided" or worked around.  The relative density of poor seeds is not a problem; it is really only the initial sequence that is impacted by "poor" seeds – but we tend to use those initial sequences the most.

Also, apologies for the wall of text: I had a lot to say.
« Last Edit: January 30, 2025, 08:32:12 am by Nominal Animal »
 
The following users thanked this post: bingo600, ksjh, incf

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 7441
  • Country: fi
    • My home page and email address
Note that for hashing keywords (ASCII strings and similar), I still use the XOR variant of DJB2 hash (DJBX33X), which is simply
Code: [Select]
uint32_t  djb2_hash(const char *src) {
    if (!src)
        return 0;  // NULL hashes to 0, while "" hashes to 5381.  NA's arbitrary choice.

    uint32_t  result = 5381;
    while (*src)
        result += (result << 5) ^ (unsigned char)(*(src++));

    return result;
and is not at all random; but the distribution of values is excellent, and is very suitable for use as a hash table hash function.

Similarly to a hash table, if you use the 96-bit value as a seed to some good PRNG to get your random sequence, you do not necessarily need an unpredictable or random mapping from the 96-bit value to the seed; you only need most/all 96-bit values to generate "good" seeds.  For a hash function, it means that typical values hashed won't produce the same indexes while not producing other indexes.  Randomness is not needed, just non-periodicity.

In that sense, my reply #5 above is way overkill.  In real life, you'd likely get quite good results if you convert the 96-bit value to a seed by using something as simple as
    uint64_t  seed = (uint64_t)(word[0]) * UINT64_C(odd1)
                   ^ (uint64_t)(word[1]) * UINT64_C(odd2)
                   ^ (uint64_t)(word[2]) * UINT64_C(odd3);
making sure the resulting seed is nonzero, perhaps exclusive-ORing the low bits with 0xAAAAAAAA and the high bits with 0x55555555 if they have fewer than 5 or more than 27 bits set, and just discarding the first hundred or so numbers from the generated sequence.
« Last Edit: January 30, 2025, 08:52:14 am by Nominal Animal »
 
The following users thanked this post: ksjh, incf

Online peter-h

  • Super Contributor
  • ***
  • Posts: 4502
  • Country: gb
  • Doing electronics since the 1960s...
CPU IDs tend to be very predictable, but they are sure to all be different :) The change from one chip to the next might be just one bit.

When I had to do this sort of stuff in the past, I just encrypted (with DES, nowadays AES256 which is much smaller especially if you use Tiny AES) the CPU ID (or whatever unique number was to hand) with itself, and just kept repeating that, getting 64 bits of entirely (and very provably) random data each time.

Then if say you need just 16 random bits, just use 16 of the 64 bits.

« Last Edit: January 31, 2025, 01:09:07 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online incfTopic starter

  • Regular Contributor
  • *
  • Posts: 154
  • Country: us
  • ASCII > UTF8
I didn't know AES was suitably random when repeatedly called upon only a few bytes of (zero padded?) input.

AES( AES( AES( [0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x34, 0x56, 0x78] ) ) ) etc.

I suppose random number generation is much more forgiving than encryption.
« Last Edit: January 31, 2025, 02:08:15 pm by incf »
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 7441
  • Country: fi
    • My home page and email address
It's the "avalanche effect" required from cryptographic hash functions.  A single bit difference in the plaintext needs to change about/roughly half the bits in the ciphertext.

Note that even though that uses AES256, the sequence generated is not cryptographically safe.  Anyone knowing any full hash, will be able to generate the rest of the sequence following that hash.  Fortunately, I don't think you have any reason to want a cryptographically secure sequence here!
 
The following users thanked this post: ksjh

Offline ksjh

  • Regular Contributor
  • *
  • Posts: 63
  • Country: de

Xorshift64* is a very simple LFSR pseudo-random number generator, whose upper 32-40 bits pass all known randomness tests, and is thus better than e.g. the industry standard, Mersenne Twister (MT19937).


In my job as a computer scientist doing research in network performance evaluation, we also use a lot of (discrete event) simulations and also need PRNGs. The PCG family of PRNGs is also very promising, see https://www.pcg-random.org/
This web site also provides useful details of different random number generators and their implementation, especially in the linked PCG paper. I see no real need for a cryptographically secure generator in this use case.
 
The following users thanked this post: Nominal Animal

Online peter-h

  • Super Contributor
  • ***
  • Posts: 4502
  • Country: gb
  • Doing electronics since the 1960s...
The STM CPUs, for example, 32F417, use a bit of hardware which makes some noise. But you can fake it; the algorithm which consumes this has no way of testing the randomness.

For your purpose you just need data which keeps changing in a random way.

You can guard against your concern by changing any 0x00 to 0x01 etc :) Anyway you can test it here
https://encode-decode.com/aes256-encrypt-online/
and there are others.

Another thing I did in years past was to take the product S/N (which was factory set and always unique) and do CRC16 or CRC32 on it.

Also see this
https://www.eevblog.com/forum/microcontrollers/32f417-cpu-id-and-creating-a-product-sn-out-of-it/
« Last Edit: January 31, 2025, 04:24:29 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline ksjh

  • Regular Contributor
  • *
  • Posts: 63
  • Country: de
While all these solutions are quite interesting, I think that for this use case, the individual numbers generated by different systems need not necessarily be truely random. As I see it, it is enough that the probability of different systems generating the same numbers is low. So, I think that the approach proposed by Nominal Animal and westf is more than sufficient: Just take the ID as a seed for a PRNG and use this PRNG to generate the different numbers you need. I am not even sure all properties of the PRNG are relevant here. For example, depending on the number of invocations of the PRNG (number of needed pseudo-random numbers), the period of the PRNG for a given seed does not necessarily be large. Perhaps, it does not even matter when the numbers generated by a single system repeat after some time. Even a very primitive LCG could be enough here.
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 21679
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
If you are to avoid this famous example https://xkcd.com/221/ , you need a source of entropy. And obviously that cannot be an algorithm such as PRBS or LFSR, whether implemented in hardware or software.

If you do have a source of entropy, do you then need a PRBS/LFSR?

Without knowing the available hardware, an entropy source cannot be defined.
« Last Edit: January 31, 2025, 05:26:46 pm by tggzzz »
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline ksjh

  • Regular Contributor
  • *
  • Posts: 63
  • Country: de
... you need a source of entropy.

Why? I see no reason at all. The starting point was that different systems should have different startup times. If you need just one number that is different on each and every system, indeed, something similar to the xkcd will work, the unique ID is unique. If you need multiple different numbers, you can algorithmically derive a sequence of pseudo-random numbers from this ID, i.e., use it as a seed for a PRNG. They are not truely random, since there is no entropy, you are right. But why would you need entropy? If you have a discrete true random process (e.g., rolling dice), the probability that two systems start with the same number is non-zero, whereas the unique ID is supposed to be unique. We are not gambling or doing crypto here.
Even in simulation, you might want to repeat simulation runs. You can simply do this by using the same seeds for the PRNGs (for each stream of pseudo-random numbers). For independent replications, you would need different seeds in each run, you would use a source of entropy. But in the given use-case, I see no need for that.
 

Offline fourfathom

  • Super Contributor
  • ***
  • Posts: 2019
  • Country: us
Why not just use a hashing function?  I use one for product ID numbers (seeded from the ucontroller internal serial number).  I borrowed liberally from this: http://isthe.com/chongo/tech/comp/fnv/

[edit:]

Sorry, I didn't notice the part about the unique sequence of numbers.
Never mind.
« Last Edit: February 01, 2025, 12:45:52 am by fourfathom »
We'll search out every place a sick, twisted, solitary misfit might run to! -- I'll start with Radio Shack.
 

Offline bson

  • Supporter
  • ****
  • Posts: 2517
  • Country: us
MD5 or SHA-1 (both are very easy to implement and efficient) and use the resulting digest as the initialization vector for ARC4.  Any symmetric stream cipher initialized with a random seed will make an excellent and fast rng.  Encrypt a sequence of 0's and out comes a random sequence.  (Although not of great cryptographic strength these days, but perfectly fine for a random sequence.)  Easily found as portable C/C++ code.

In addition to die numbers, you can also use a dangling ADC input and just read a few k, then digest it down to an IV.  With cryptographic digests a single bit change on the input, on average, will change half the output bits so it takes very little random noise to completely change the IV.

CRC isn't that great; it's generally not large enough, not even CRC-32.
« Last Edit: January 31, 2025, 09:35:25 pm by bson »
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 21679
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
... you need a source of entropy.

Why? I see no reason at all. The starting point was that different systems should have different startup times. If you need just one number that is different on each and every system, indeed, something similar to the xkcd will work, the unique ID is unique.

A unique identifier can be a sufficient source of entropy provided different ids are uncorrelated.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 16098
  • Country: fr
From the latests posts, yes, many MCUs these days embed a hardware RNG of some kind and these are usually pretty good, being based on noise.

Since this all started with the unique ID, I think we got into this idea of seeding a PRNG, but while you'll have a very low probability of generating the same sequence with two different devices, it will generate the same sequence for each of them every time the PRNG is seeded.

Using a hardware RNG will avoid this if that matters at all (but depending on the application, having a predictable sequence for each device may be an issue even if they are all different, or it may not matter).

I haven't run into a MCU that didn't embed a hardware RNG lately, among STM32, WCH, NXP and a few others. There sure are, but a hardware RNG is certainly not a rarity. That can be a bit "slow" to generate values though, so if you need a very fast sequence of "random" numbers, you can use a decent PRNG and seed it with a random value generated with the hardware RNG. You can even reseed it on a regular basis to improve the "randomness".
« Last Edit: January 31, 2025, 10:51:09 pm by SiliconWizard »
 

Online incfTopic starter

  • Regular Contributor
  • *
  • Posts: 154
  • Country: us
  • ASCII > UTF8
Smaller STM32 devices don't have RNG. (Mine doesn't!)

If AES isn't too slow, I think that is likely to be the least error prone to implement.

I can just copy my 96 bit ID into a zero padded input buffer at startup. Then once every 1 to 10 milliseconds run AES, and then copy the output back into the input.

Not sure how many cycles it would take to run AES-128/256/etc. on a cortex-M0 with single-cylce hardware multiply at 8MHz, but I'm sure it can't be that bad. I'm guessing 4000 instruction cycles at most for a 16-byte block.

I only expect to need a random number once every few seconds at most.
« Last Edit: January 31, 2025, 11:35:23 pm by incf »
 

Online westfw

  • Super Contributor
  • ***
  • Posts: 4376
  • Country: us
Quote
Why not just use a hashing function?
Um, OP wants a SEQUENCE.
Nearly ALL of the things mention (CRC, LFSR, assorted Crypto algorithms) can be considered to be "hashing functions."  So we're sort of down to "which hashing functions are most applicable to converting my 96bit ID to a sequence of much smaller numbers.  (ie are liable to produce the most unique sequence.)"

The FNV hash you refer to involves a bunch of 64bit multiplies; not necessarily friendly to a microcontroller environment.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 16098
  • Country: fr
I think we suggested a few methods that should be more than good enough while being relatively cheap, I'm not sure why incf ended up selecting AES.

In any case, I just suggest evaluating different approaches first on a computer rather than directly on the final target. That'll save a lot of time and will lend itself way better to checking the fitness of the approach. Just use your favorite scripting language or C, generate long sequences that you can analyze directly or save to a file and analyze with other tools.
 

Online westfw

  • Super Contributor
  • ***
  • Posts: 4376
  • Country: us
Quote
suggest evaluating different approaches first on a computer
The computer won’t have the 96bit IDs…
Some manufacturers document what goes into forming each byte of the id (wafer#, x,y,etc), but others just say “all 96bits will be a unique id”…

 

Online peter-h

  • Super Contributor
  • ***
  • Posts: 4502
  • Country: gb
  • Doing electronics since the 1960s...
From my project notes (32F417 168MHz)

The "tiny AES256" https://github.com/kokke/tiny-AES-c does 100kbytes/sec. It is 900 bytes in size. The AES256 which comes with MbedTLS does 800kbytes/sec.  It is 4026 bytes in size.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online incfTopic starter

  • Regular Contributor
  • *
  • Posts: 154
  • Country: us
  • ASCII > UTF8
... I'm not sure why incf ended up selecting AES.

I started leaning towards it for three (edit: to be honest) four reasons (1) familiarity (2) ease of use (3) size of the state variable (edit:) (4) the vague feeling that it is so inscrutably complex that it 'must be better'.

I played around a bit and it "seemed" occasionally that certain generators (which I am most certainly using wrong due to a lack of familiarly) can put out spans of numbers that look a bit repetitive or predictable (LFSRs*). I'm not confident that a random sequence generator that uses a 32 bit or 64 bit number internally is 'as good at' representing a 96 bit ID as one that uses 128 bits. For certain algorithms certain ranges of bits in the output looked very predicable and repetitive at times. I think I could easily get it wrong and never know despite rigorous testing until long after it gets out into the field.

*(I know certain bit ranges of certain LFSRs are widely aclaimed as being industry standard level of 'good')

For better or worse I've used AES before and there are a lot of ready to use implementations to 'copy 'n paste' into the firmware. The fact that it operates on 128 bit numbers make it possible to be able to copy in the full unique ID and be confident that no bits of state are being lost due to truncation.

side note: I suppose I absolutely need to be 'mixing in' the unique ID every so often with many of these CRC-like functions, otherwise, they would eventually collide and get stuck in a loop [ AES(AES(AES(ID))) vs AES(ID+AES(ID+AES(ID))) ]
edit: in the case of TinyAES I guess I'd be using AES128 in counter mode with the key set to MCU_ID and initial value set to zero. And just operating on the same buffer repeatedly (output=input).

I feel like I can't simulate this system as well as I'd like (and I certainly can't say that it meets mathematical constraints that I'd like it to meet), so I'm compensating by going overkill on the underlying number generator.

I definitely agree that there are significantly cheaper, simpler, and more efficient methods - I just lack the knowledge, familiarity, etc. to be very confident that they are worth the additional effort.
« Last Edit: February 01, 2025, 01:09:01 pm by incf »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf