Author Topic: Super-long-term Retention of small data files  (Read 8450 times)

0 Members and 1 Guest are viewing this topic.

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Super-long-term Retention of small data files
« on: May 12, 2017, 07:49:44 am »
I'm basically just mucking around here, but also being half-serious.

I have a some data files which are fairly small (<1 megabyte) but contain a raft of critical data such as an encrypted container full of passwords, copies of birth certificates and other vital documents. I currently back these up to various different locations/media (including physically printing the data out (in encrypted form) using a program called PAPERBACK.

But as we know, optical media can degrade and be easily damaged, normal NAND flash (like what's in USB sticks or SD Cards) isn't overly reliable in the long-term and hard disks can die unexpectedly.

I was thinking of backing vital data up to EEPROM (like the 1MB Microchip 24AA1026-I/P) which can then be stored off-site and is small and durable.

I'm just thinking, if the worst was to happen and house burnt down, all I'd need is a ROM reader to recover vital data. Thoughts?
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 2085
  • Country: sk
Re: Super-long-term Retention of small data files
« Reply #1 on: May 12, 2017, 07:56:03 am »
standard storage media + regular refresh of the media is the way to go.  copy to NEW media in regular intervals (few years) and shred the old media.
 

Online tszaboo

  • Super Contributor
  • ***
  • Posts: 7304
  • Country: nl
  • Current job: ATEX product design
Re: Super-long-term Retention of small data files
« Reply #2 on: May 12, 2017, 09:22:35 am »
Maxim iButton? If the capacity is enough.
Although, nowadays I would just place the info into into a password protected file and send myself in an email.
 

Offline Brumby

  • Supporter
  • ****
  • Posts: 12288
  • Country: au
Re: Super-long-term Retention of small data files
« Reply #3 on: May 12, 2017, 09:24:12 am »
Yes ... I'm not a fan of any solution where a sense of "set and forget" can creep in.

Regular refreshing has a number of benefits - from ensuring the archives are readable to resetting the aging clock.

What the best solution is - I don't know.
 

Offline hayatepilot

  • Frequent Contributor
  • **
  • Posts: 263
  • Country: ch
Re: Super-long-term Retention of small data files
« Reply #4 on: May 12, 2017, 09:46:23 am »
There are DVDs and Blurays that are rated to last 1000 years:
http://www.mdisc.com/corporate/

They're not even that expensive (3$ per DVD and 5$ per Bluray)

Greetings
 

Online tszaboo

  • Super Contributor
  • ***
  • Posts: 7304
  • Country: nl
  • Current job: ATEX product design
Re: Super-long-term Retention of small data files
« Reply #5 on: May 12, 2017, 10:25:59 am »
There are DVDs and Blurays that are rated to last 1000 years:
Nothing guarantees, that you could buy a Bluray reader in 10 years. Imagine buying a VHS, Laserdisk, HD-DVD or a Zip drive today. Also, you cannot just make a EEPROM USB drive, because maybe USB will not be around, and the Flash in the controller chip might die, and the programming toolchain might be gone.
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Super-long-term Retention of small data files
« Reply #6 on: May 12, 2017, 10:34:14 am »
There are DVDs and Blurays that are rated to last 1000 years:

Yeah... no. We use good quality DVD and Bluray discs at work for archival purposes and even now we are worried about longevity. We face media which is 10 years old and worry about it. The first thing we do before anything else is image it because we just don't know how reliable it is.

Nothing guarantees, that you could buy a Bluray reader in 10 years. Imagine buying a VHS, Laserdisk, HD-DVD or a Zip drive today. Also, you cannot just make a EEPROM USB drive, because maybe USB will not be around, and the Flash in the controller chip might die, and the programming toolchain might be gone.

Correct! I have a hard disk sitting on my desk right now where the data is just totally lost. The USB controller on-board has failed and there is nothing I can do (with the tools I have at home) to recover anything.

Yes ... I'm not a fan of any solution where a sense of "set and forget" can creep in.

Regular refreshing has a number of benefits - from ensuring the archives are readable to resetting the aging clock.

Sure, but when stored off-site, regular refreshing is a pain in the ass. For something like a birth certificate or copy of a driving licence, that image won't change in 10 years (or ever). For things that are updated more often (like a password DB), that sort of system would work I suppose.

Although, nowadays I would just place the info into into a password protected file and send myself in an email.

That doesn't work for me. I don't actually "know" my e-mail password. It's a very long string on purpose. Without my password DB, I couldn't get into it.
« Last Edit: May 12, 2017, 10:37:45 am by Halcyon »
 

Offline Rerouter

  • Super Contributor
  • ***
  • Posts: 4694
  • Country: au
  • Question Everything... Except This Statement
Re: Super-long-term Retention of small data files
« Reply #7 on: May 12, 2017, 11:06:06 am »
If most of the size is photos, and you only have a few pages of text i might say go the analog approach, get them printed to archive grade negatives or microfilm, then store them in a small sealable container,

In 50 years when bluray is only something grandpa talks about, you can still get out a magnifying glass.
 

Offline macboy

  • Super Contributor
  • ***
  • Posts: 2252
  • Country: ca
Re: Super-long-term Retention of small data files
« Reply #8 on: May 12, 2017, 01:22:30 pm »
EEPROM, FLASH, etc. are not any more viable than magnetic media, and maybe worse. They will lose the cell charge over time and lose the contents.

OTP ROM may be a possibility. I am talking true OTP, not just UV erasable EPROM packaged in a plastic non-windowed package which is what most "OTP" ROMs are. True OTP would use a series of fusable links that are blown out or kept during programming. This seems like an immutable, physical change type of programming that can't occur or revert spontaneously. It should last.  You may need to build something to read out the contents of your ROM chip in 25+ years, but that will always be possible.

I have a pile of TDK archival DVD+R discs for backup of small data. They are rated 100 years, and they are coated with the same ultra-hard coating that blu-ray discs use to prevent scratches. Obviously it is impossible to truly test the data retention, but so far so good. I don't need them to last forever, but longer than the 'cheapest I can find' CD-Rs I bought back in the early days of optical disc burning, would be nice.

Maybe your best policy is to not depend upon longevity of your chosen medium media, but to refresh the media (whatever they are) every N years, possibly updating the data while you are at it.
 

Offline eugenenine

  • Frequent Contributor
  • **
  • Posts: 865
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #9 on: May 12, 2017, 01:27:11 pm »
Maxim iButton? If the capacity is enough.
Although, nowadays I would just place the info into into a password protected file and send myself in an email.

What happens when your hdd dies and you loose all your e-mail?
 

Online tszaboo

  • Super Contributor
  • ***
  • Posts: 7304
  • Country: nl
  • Current job: ATEX product design
Re: Super-long-term Retention of small data files
« Reply #10 on: May 12, 2017, 02:07:11 pm »
Maxim iButton? If the capacity is enough.
Although, nowadays I would just place the info into into a password protected file and send myself in an email.

What happens when your hdd dies and you loose all your e-mail?
You mean when google looses all my emails? I just send a request to NSA, surely they have a copy of it.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 4982
  • Country: ro
  • .
Re: Super-long-term Retention of small data files
« Reply #11 on: May 12, 2017, 02:32:01 pm »
Use base64/OCR friendly font or downright binary / QR schemes and A LOT of redundancy (error correction) to print the data on archival paper then store the paper. You can OCR and use the error recovery information if needed to recover the information.

You could also look at MRAM chips, apparently they have a data retention of 20 years at 125c  (for some versions) and they're very easy to work with:

https://www.digikey.com/product-detail/en/everspin-technologies-inc/MR25H40CDF/819-1040-ND/3471117

You could store a few of these at random locations and maybe once a year read back the data, check them for errors, write the chip again (if needed to "refresh" the memory cells) and put it back in the archive.


 

Offline Tom45

  • Frequent Contributor
  • **
  • Posts: 555
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #12 on: May 12, 2017, 03:39:30 pm »
standard storage media + regular refresh of the media is the way to go.  copy to NEW media in regular intervals (few years) and shred the old media.

+1

To this I'd add that you add error correcting using something like MultiPar.
 

Offline daqq

  • Super Contributor
  • ***
  • Posts: 2301
  • Country: sk
    • My site
Re: Super-long-term Retention of small data files
« Reply #13 on: May 12, 2017, 05:32:39 pm »
If you truly trust your encryption, you can store them in multiple public-ish places (various cloud drives (google, microsoft, take your pick)), attach them as forum posts, email them, facebook, paint the Internet with them. If the Internet goes away permanently, you have bigger problems.

The most durable non-electronic medium would probably be granite+PaperBack+engraver.

The most durable electronic medium would be as was said antifuse PROM memories, or mask programmed memories.

Or, if you are looking for obscure stuff: https://www.theverge.com/2016/2/16/11018018/5d-data-storage-glass

:)
Believe it or not, pointy haired people do exist!
+++Divide By Cucumber Error. Please Reinstall Universe And Reboot +++
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26751
  • Country: nl
    • NCT Developments
Re: Super-long-term Retention of small data files
« Reply #14 on: May 12, 2017, 05:38:32 pm »
Yes ... I'm not a fan of any solution where a sense of "set and forget" can creep in.
That was my first thought as well. My current strategy: If you want to keep data, then make it part of your active data set which gets backed up and transferred to new media during an upgrade.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Gyro

  • Super Contributor
  • ***
  • Posts: 9410
  • Country: gb
Re: Super-long-term Retention of small data files
« Reply #15 on: May 12, 2017, 07:25:18 pm »
For small volumes of data, paper tape was always a very durable medium. It had the advantage of being human repairable and readable (in small chunks anyway). Maybe we need an updated version with higher data density due to modern technology (smaller optical sensors, laser punching?). Maybe a more durable tape material too, although tight wound paper is extremely resistant to most threats (including water and fire), plastics might actually suffer more long term degradation.

There's a big attraction to a media format that is just so simple that it simply cannot go obsolete - if you discovered it and wanted to read it in the far distant future, then it would be a simple matter to constuct a suitable reader from scratch. It's the sort of media you might want to put in a deep space probe!
« Last Edit: May 12, 2017, 07:28:11 pm by Gyro »
Best Regards, Chris
 

Offline john_c

  • Contributor
  • Posts: 43
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #16 on: May 12, 2017, 09:20:40 pm »
OTP ROM may be a possibility. I am talking true OTP, not just UV erasable EPROM packaged in a plastic non-windowed package which is what most "OTP" ROMs are. True OTP would use a series of fusable links that are blown out or kept during programming. This seems like an immutable, physical change type of programming that can't occur or revert spontaneously. It should last.  You may need to build something to read out the contents of your ROM chip in 25+ years, but that will always be possible.

I like this option. It's totally weird, and sounds fun!

Looks like the good kind, with the titanium-tungsten fuses, is rare but available. Here is 63S141: http://www.ebay.com/itm/222365236136

edited to add: alright, so only 8 kB is available, hahaha. Text files only!
« Last Edit: May 12, 2017, 09:25:41 pm by john_c »
 

Offline macboy

  • Super Contributor
  • ***
  • Posts: 2252
  • Country: ca
Re: Super-long-term Retention of small data files
« Reply #17 on: May 12, 2017, 10:24:31 pm »
OTP ROM may be a possibility. I am talking true OTP, not just UV erasable EPROM packaged in a plastic non-windowed package which is what most "OTP" ROMs are. True OTP would use a series of fusable links that are blown out or kept during programming. This seems like an immutable, physical change type of programming that can't occur or revert spontaneously. It should last.  You may need to build something to read out the contents of your ROM chip in 25+ years, but that will always be possible.

I like this option. It's totally weird, and sounds fun!

Looks like the good kind, with the titanium-tungsten fuses, is rare but available. Here is 63S141: http://www.ebay.com/itm/222365236136

edited to add: alright, so only 8 kB is available, hahaha. Text files only!
8 kB ? What are you looking at. The one you linked is 256x4, that is 4 bits wide, 256 locations. In other words... 128 bytes. Enough for one password maybe.   :-DD
 

Offline john_c

  • Contributor
  • Posts: 43
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #18 on: May 12, 2017, 10:34:10 pm »
Oops, I meant kilobit, because there are 8 available!

Here, this is much better: only 1.5c / byte! http://www.ebay.com/itm/322224601177  :-DD
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26751
  • Country: nl
    • NCT Developments
Re: Super-long-term Retention of small data files
« Reply #19 on: May 12, 2017, 10:39:10 pm »
OTP ROM may be a possibility. I am talking true OTP, not just UV erasable EPROM packaged in a plastic non-windowed package which is what most "OTP" ROMs are. True OTP would use a series of fusable links that are blown out or kept during programming. This seems like an immutable, physical change type of programming that can't occur or revert spontaneously. It should last.  You may need to build something to read out the contents of your ROM chip in 25+ years, but that will always be possible.
I like this option. It's totally weird, and sounds fun!

Looks like the good kind, with the titanium-tungsten fuses, is rare but available. Here is 63S141: http://www.ebay.com/itm/222365236136
For utter coolness you could put a huge amount of SMT fuses on a board and some simple decoding logic to program and read them.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 4982
  • Country: ro
  • .
Re: Super-long-term Retention of small data files
« Reply #20 on: May 12, 2017, 10:42:26 pm »
At what point would it be easier to just get some thin steel sheet, paint the sheet with a layer of black paint (if it helps) and then use a laser engraver to make dots in it ?

or maybe make some steel tape, a few bits per line .. some holes on the side to advance the tape and you should have a pretty reliable medium

Or maybe we could revert back to wire memory, since apparently you can still hear music and voice recorded decades ago like it was recorded today ?

(jump to ~ 8:20)

https://youtu.be/K_haRMbKZ1M
 

Offline rdl

  • Super Contributor
  • ***
  • Posts: 3665
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #21 on: May 12, 2017, 11:15:36 pm »
standard storage media + regular refresh of the media is the way to go.  copy to NEW media in regular intervals (few years) and shred the old media.

This is basically what I do. I don't have too much stuff, right now it fits on less than 2 TB. I have it all stored on a microserver running FreeNAS (2 drives mirrored). I have a backup copy on another small computer running WHS2011. I update that backup about once a month and the rest of the time that computer is turned off. I keep a third copy, for just in case, on a external drive connected to yet another computer.
 

Offline CatalinaWOW

  • Super Contributor
  • ***
  • Posts: 5170
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #22 on: May 13, 2017, 12:30:54 am »
There are really only a few types of storage media with demonstrated long term reliability.

Cut letters in stone.  Demonstrated thousands of years.  Low density.  Expensive.
Pressed letters in baked clay.  Demonstrated thousands of years.  Low density.  Relatively cheap.
Ink on paper.  Demonstrated reliability of a couple hundred years.  Modest density.  Cheap.

At the beginning of the PC era there was a technology based on printing pixels on paper.  Assuming 300 dpi both directions a normal sheet of paper would be good for something like 5 megabits allowing some overhead for alignment and error correction.  Decoding the result shouldn't be overly hard either since some form of camera/scanner is likely to be around for a long time.
 

Offline eugenenine

  • Frequent Contributor
  • **
  • Posts: 865
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #23 on: May 13, 2017, 12:42:55 am »
Maxim iButton? If the capacity is enough.
Although, nowadays I would just place the info into into a password protected file and send myself in an email.

What happens when your hdd dies and you loose all your e-mail?
You mean when google looses all my emails? I just send a request to NSA, surely they have a copy of it.

Many years ago I had a hotmail account.  Then Microsoft bought Hotmail and one day I signed in and found only one message that was a 'sorry we lost your e-mail' message.  So it can happen.
 

Offline CatalinaWOW

  • Super Contributor
  • ***
  • Posts: 5170
  • Country: us
Re: Super-long-term Retention of small data files
« Reply #24 on: May 13, 2017, 03:00:42 am »
Even if the emails are not lost, the software to recover them may become difficult or impossible to obtain.  I have many megabytes of archived email from one system that I was using a couple of decades ago.  I don't own the software currently, it is expensive and is likely to become impossible to get in the near future as it lost the war many years ago and has been hanging on by fingertips since.  Another demonstration of why a rotating refresh is necessary if you really want to save the data.  I obviously didn't want any of this data that badly.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf