[solved] catastrophic event, file recovery on ext3

[legacy]

a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.

Thousands precious files lost, with Inodes overwritten by houndred of useless files 

When the Kernel primive "file_delete" is invoked, the ext3 filesystem doesn't leave the inode pointing to a file or to a folder, but it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.

This means, common recovery tools (like DiskTest) simply do not work with ext3, you have to cross your finger, and try special tools, like described here by the first person who tried to find a solution.

Code: [Select]

2019-10-21--14-33-18---2019-10-21--14-35-13 - [ sys-fs/extundelete ] - success - @2.29.1/7.3.0
------------------------------------------------------------------------------------------------------
2019-10-21--15-21-12---2019-10-21--15-21-58 - [ sys-fs/ext3grep ] - failure - @2.29.1/7.3.0
      configure: error: ext3grep doesn't work on Big Endian systems
------------------------------------------------------------------------------------------------------


• ext3grep
• extundelete

Unfortunately ext3grep does not even compile on BIG Endian systems ( I am on a HPPA workstation ), while extundelete is less advanced, hene I moved the harddrive to a x86 machine.

But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.

That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.

I am saying, the last four weekends of work is gone 

So ... I am trying searching withing blocks with the idea of grabbing some block containing the header of tarball so the block might be reversed somehow into the inode and ext3grep might try to resume the file.

Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz

[SiliconWizard]

You never back up?

[I wanted a rude username]

The first thing to do is remount read-only.

If the underlying storage is SSD, your data have probably been trimmed already.

If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.

[legacy]

You never back up?

The hard drive is mirrored, but this means the content is equal to both disks. We usually save a snapshot dail, it's automatically done before midnight, and then we burn one-week-of-snapshots into a DVDRAM every Sunday afternoon; but in this case we didn't because we are still moving our storage stuff to a new place, therefore everything was manual, and the backup was planned to be monthly.

We are speaking about it from the mid of September to the mid of October 

[legacy]

The first thing to do is remount read-only.

Done, but it was too late. The script silently did its job before someone noticed what it was doing.

If the underlying storage is SSD

The couple of disks in RAID1 are electromechanical.

If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.

Precisely what I am trying to do 

[I wanted a rude username]

Good luck ... you'll need it!

I was successful using this method a few times. grep -C, obviously.

There are also tools that scan a block device for magic such as gzip headers. This may be more efficient, if you can get it to work.

[legacy]

The disk is currently attached to a mac-mini/x86, running Linux.
It's not mounted, only accessed as raw device

Code: [Select]

ext3grep /dev/sdb3 --search "DT11-v12-r2.tgz"
Running ext3grep version 0.10.2
Minimum / maximum journal block: 1545 / 35886
Loading journal descriptors... sorting... done
The oldest inode block that is still in the journal, appears to be from 1571661094 = Mon Oct 21 14:31:34 2019
Journal transaction 3453486 wraps around, some data blocks might have been lost of this transaction.
Number of descriptors in journal: 31738; min / max sequence numbers: 3452751 / 3453770
Blocks containing "DT11-v12-r2.tgz": 3310552 4215792


Code: [Select]

  --search str           Find blocks that contain the fixed string 'str'.
  --search-inode blk     Find inodes that refer to block 'blk'.

Unfortunaly the above blocks had no good luck to find inodes that refer to them

-

What did you use?

[I wanted a rude username]

I treated the entire disk as a file* and searched not for filenames, but for plaintext file contents. (Also passed it through strings(1) first, but that's an oldskool performance optimisation, and raw grep may actually be quicker these days.) Something like:

# dd if=/dev/sdb3 bs=128k | grep -C100 "some string I remember from the text file" > /tmp/sdb3_matches.txt

If you add the -b parameter to grep, it will also tell you where on the disk the matching text is.

This is all super low level and also will take a long time to run (about two hours per terabyte). Ideally try other non-destructive options first.

*Mantra: "everything is a stream of bytes".  Repeat until enlightened. -Linus Torvalds

[techman-001]

a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.

Thousands precious files lost, with Inodes overwritten by houndred of useless files 

Damn, sorry to hear that, my sympathies.

I've lost data in the past from unforeseen circumstances so I feel your pain.

I do hope none of the Forth stuff you were writing has been lost.

[RoGeorge]

To avoid such events for the future, ZFS filesystem can roll back changes, self repair RAID arrays and much more.

[Whales]

Eep.  I know the feelings you are going through right now.  Be cautious, you can easily make things worse, time spent sitting down with a cup of tea is relatively cheap in comparison.

So you are saying that the metadata for your files might still exist, just spread out somewhere on the disk?  Try asking for help on a relevant IRC channel, see if anyone can help.  This forum isn't the best place to find kernel/filesystem experts.

Otherwise: your only hope is the horrid process of piecing together blocks of raw data to get some parts of files back. Files are often not contiguous, so as file sizes increase you (exp?) have lower chance of finding a file as a single piece.  Tools like photorec are fun to play with and give a graphical description of this problem (where jpegs are recovered with a reasonable chance of appearing corrupt at some point in).

[drussell]

Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz

Photorec knows how to find tar headers, including the gzipped variety.

If you point it at your entire disk you may be able to recover some of the ones that have not actually been overwritten (if there are any.)

[legacy]

I do hope none of the Forth stuff you were writing has been lost.

yup, fortunately uForth and uLisp were on a different machine, with a copy on my PDA. But the the latest firmware is gone.

Fortunately, there is a girl in my team, and you know ... women are used to taking a lot of notes, therefore we have a copy on paper about what she had modified and fixed.

[legacy]

# dd if=/dev/sdb3 bs=128k | grep -C100 "some string I remember from the text file" > /tmp/sdb3_matches.txt

yes, I did something similar to grab part of C files and assembly files: all partially resumed.
But the real problem is how to get the whole tgz archive containing patches, documentation, etc.

All the mentioned tools require you to know the inode, and I have no idea about how to get it.
With the inode, and with luck, you can try to get the direct, indirect nodes pointing to data blocks.

I am talking about 20Mbyte of tarball | gzipped, on a filesystem whose datablock is 4K byte of size: it means a lot (~20M/4k) of blocks sparse around the whole block device.

So 

[legacy]

Photorec knows how to find tar headers, including the gzipped variety.

Yup, I was thinking about something to recover files based on their headers, footers, and internal data structures. Sort of "file" combined with "grab". This process is commonly referred to as data carving.

PhotoRec seems based on Foremost, which doens't come with a defined pattern analysis for tgz, but the tool Scalpel has it!

Anyway, I tried them: no luck

Code: [Select]

mi2q resumed_tgz # ls
09506760.tgz  12115656.tgz  12737448.tgz  13134776.tgz  14371096.tgz  14969360.tgz  25218536.tgz  30216704.tgz
10152855.tgz  12115768.tgz  12781008.tgz  13140864.tgz  14379160.tgz  14969920.tgz  25424944.tgz  30219264.tgz
10152865.tgz  12115952.tgz  12793184.tgz  13145016.tgz  14404592.tgz  14975712.tgz  25501688.tgz  30517472.tgz
10953552.tgz  12116064.tgz  12793296.tgz  13145144.tgz  14422032.tgz  14975960.tgz  25514160.tgz  30585584.tgz
10953960.tgz  12117600.tgz  12793936.tgz  13168200.tgz  14422064.tgz  14983272.tgz  25911848.tgz  30585624.tgz
10954024.tgz  12127760.tgz  12793960.tgz  13173048.tgz  14429496.tgz  14983304.tgz  26810464.tgz  30585680.tgz
10959928.tgz  12127872.tgz  12822856.tgz  13176160.tgz  14443064.tgz  14985280.tgz  26810816.tgz  30917376.tgz
11003920.tgz  12128136.tgz  12823816.tgz  13191216.tgz  14445176.tgz  14985776.tgz  26811400.tgz  31672944.tgz
11003984.tgz  12128376.tgz  12824008.tgz  13193008.tgz  14473360.tgz  15050808.tgz  26828256.tgz  31973480.tgz
11004472.tgz  12178408.tgz  12842872.tgz  13202656.tgz  14474856.tgz  15050840.tgz  26859136.tgz  31973776.tgz
11004720.tgz  12214216.tgz  12844416.tgz  13226392.tgz  14480456.tgz  15053824.tgz  26860096.tgz  31973936.tgz
11005048.tgz  12214240.tgz  12878272.tgz  13253704.tgz  14497008.tgz  15065088.tgz  26861120.tgz  31974104.tgz
11008800.tgz  12215200.tgz  12892440.tgz  13257312.tgz  14502912.tgz  15074776.tgz  26907552.tgz  31975272.tgz
11014696.tgz  12237608.tgz  12892552.tgz  13278584.tgz  14503304.tgz  15074808.tgz  26925080.tgz  31977392.tgz
11017720.tgz  12238704.tgz  12893136.tgz  13309800.tgz  14521384.tgz  15082456.tgz  26925400.tgz  32156760.tgz
11019304.tgz  12264064.tgz  12893200.tgz  13346128.tgz  14528528.tgz  15088328.tgz  26925656.tgz  32157520.tgz
11019664.tgz  12264176.tgz  12893312.tgz  13387248.tgz  14528704.tgz  15088352.tgz  26925704.tgz  32169808.tgz
11019856.tgz  12265304.tgz  12899288.tgz  13402472.tgz  14528736.tgz  15089696.tgz  26925720.tgz  32875048.tgz
11036032.tgz  12269328.tgz  12901656.tgz  13500424.tgz  14531616.tgz  15089992.tgz  26925976.tgz  32875608.tgz
11036912.tgz  12279504.tgz  12901752.tgz  13533984.tgz  14531624.tgz  15091032.tgz  26926064.tgz  32875720.tgz
11087600.tgz  12307440.tgz  12905376.tgz  13697072.tgz  14589952.tgz  15091512.tgz  26926192.tgz  32922264.tgz
11087736.tgz  12324880.tgz  12913696.tgz  13753096.tgz  14591400.tgz  15094136.tgz  26926320.tgz  34556280.tgz
11107296.tgz  12324904.tgz  12914336.tgz  13753128.tgz  14596968.tgz  15140232.tgz  27087368.tgz  35274120.tgz
11107408.tgz  12364560.tgz  12914912.tgz  13793424.tgz  14610016.tgz  15140672.tgz  27590720.tgz  36124224.tgz
11107856.tgz  12368992.tgz  12930200.tgz  13800064.tgz  14653816.tgz  15142496.tgz  27591280.tgz  36132736.tgz
11144448.tgz  12369040.tgz  12962144.tgz  13802328.tgz  14678856.tgz  15145936.tgz  27593784.tgz  36135656.tgz
11192576.tgz  12369400.tgz  12966176.tgz  13803056.tgz  14679688.tgz  15147656.tgz  27597448.tgz  37041488.tgz
11203656.tgz  12405472.tgz  12966288.tgz  13964280.tgz  14687224.tgz  15148512.tgz  27597576.tgz  37444912.tgz
11276304.tgz  12405784.tgz  12966312.tgz  13964312.tgz  14716616.tgz  15149960.tgz  27772584.tgz  37595856.tgz
11336408.tgz  12406232.tgz  12967064.tgz  13964512.tgz  14716648.tgz  15151904.tgz  27772616.tgz  37599360.tgz
11436040.tgz  12406680.tgz  12969016.tgz  13966152.tgz  14726344.tgz  15152096.tgz  27774048.tgz  37617360.tgz
11549512.tgz  12410992.tgz  12973000.tgz  13967672.tgz  14729736.tgz  15152432.tgz  28345912.tgz  37618920.tgz
11549624.tgz  12491720.tgz  12999128.tgz  13968256.tgz  14733000.tgz  15162216.tgz  28461032.tgz  37629456.tgz
11659040.tgz  12495576.tgz  12999384.tgz  13969088.tgz  14778624.tgz  15166152.tgz  28461536.tgz  37646632.tgz
11661024.tgz  12495600.tgz  12999472.tgz  13970272.tgz  14778656.tgz  15169424.tgz  28465496.tgz  37669640.tgz
11661048.tgz  12496096.tgz  12999568.tgz  13972912.tgz  14779816.tgz  15173192.tgz  28466640.tgz  37674344.tgz
11662832.tgz  12506040.tgz  13000248.tgz  13975120.tgz  14780352.tgz  15173400.tgz  28467144.tgz  38024024.tgz
11822680.tgz  12535776.tgz  13001304.tgz  13975464.tgz  14783648.tgz  15173648.tgz  28532472.tgz  38917432.tgz
11844048.tgz  12535824.tgz  13015920.tgz  14069072.tgz  14786024.tgz  15174736.tgz  28533653.tgz  38917616.tgz
11844088.tgz  12599704.tgz  13016720.tgz  14069104.tgz  14786648.tgz  15175928.tgz  28535973.tgz  39415904.tgz
11844344.tgz  12599816.tgz  13020136.tgz  14071424.tgz  14796440.tgz  15195520.tgz  28542256.tgz  39570672.tgz
11868384.tgz  12600496.tgz  13020168.tgz  14094680.tgz  14797480.tgz  15197296.tgz  28542624.tgz  39626720.tgz
11885624.tgz  12601920.tgz  13020680.tgz  14095392.tgz  14798088.tgz  15197592.tgz  28543712.tgz  40094032.tgz
11895688.tgz  12621656.tgz  13022216.tgz  14095752.tgz  14799832.tgz  15199968.tgz  28790640.tgz  40094384.tgz
11895800.tgz  12629320.tgz  13025496.tgz  14098560.tgz  14800680.tgz  15201384.tgz  29022120.tgz  40631362.tgz
11895824.tgz  12631368.tgz  13030496.tgz  14116024.tgz  14802280.tgz  15208648.tgz  29026112.tgz  42173504.tgz
11905904.tgz  12632328.tgz  13038760.tgz  14116744.tgz  14837008.tgz  15211904.tgz  29026120.tgz  42813432.tgz
11915272.tgz  12632352.tgz  13044680.tgz  14140384.tgz  14837040.tgz  15212400.tgz  29321488.tgz  43116816.tgz
11918904.tgz  12632688.tgz  13044944.tgz  14256096.tgz  14838632.tgz  15213616.tgz  29321728.tgz  43848776.tgz
11933072.tgz  12632800.tgz  13045016.tgz  14256152.tgz  14839488.tgz  15233464.tgz  29321976.tgz  43922464.tgz
11933208.tgz  12635264.tgz  13045344.tgz  14256272.tgz  14840616.tgz  15237800.tgz  29322224.tgz  43924256.tgz
11952016.tgz  12662808.tgz  13058936.tgz  14257200.tgz  14840648.tgz  15240848.tgz  29322344.tgz  44101832.tgz
11993192.tgz  12662832.tgz  13090912.tgz  14259200.tgz  14841840.tgz  15284136.tgz  29322616.tgz  44276472.tgz
12003536.tgz  12667416.tgz  13092656.tgz  14259360.tgz  14880976.tgz  15301216.tgz  29323120.tgz  44863504.tgz
12045704.tgz  12667992.tgz  13092704.tgz  14259544.tgz  14909200.tgz  16203928.tgz  29323536.tgz  44863536.tgz
12045816.tgz  12668432.tgz  13093080.tgz  14259800.tgz  14909872.tgz  22935264.tgz  29323944.tgz  44863952.tgz
12046144.tgz  12669936.tgz  13095320.tgz  14275152.tgz  14911160.tgz  23929496.tgz  29327792.tgz
12046168.tgz  12669960.tgz  13095456.tgz  14275368.tgz  14914024.tgz  24261726.tgz  29717482.tgz
12049392.tgz  12670048.tgz  13120712.tgz  14295008.tgz  14939112.tgz  24274846.tgz  30200904.tgz
12083312.tgz  12703408.tgz  13122936.tgz  14297088.tgz  14939568.tgz  24519358.tgz  30204416.tgz
12083400.tgz  12733936.tgz  13123000.tgz  14365832.tgz  14947088.tgz  24521778.tgz  30206536.tgz
12083424.tgz  12735496.tgz  13123408.tgz  14370048.tgz  14963608.tgz  24522970.tgz  30209848.tgz
12101320.tgz  12736320.tgz  13128016.tgz  14370080.tgz  14968488.tgz  25213160.tgz  30214208.tgz


The 90% of these files is a corrupted tgz, and the 10% of them don't contain what I was looking for, but it contains a very old backup, that was deleted time ago.

And this is what I do not understand: the data carving stuff is able to find and resume tgz files made 2 months ago, and deleted 1 month ago, but it's not able to find and resume a tgz that was created 2 days ago and deleted yesterday.

It should have a high probability to have been overwritten, as well as recent deleted files should have a low probability to have been overwritten 

Something tells me the LAST algorithm is not used for ext3, which, if it's true, will make me abandon this filesystem because it's worrying.

Combined with these annoying things:

Code: [Select]

EXT4-fs (sda3): couldn't mount as ext3 due to feature incompatibilities


which I still have to understand ...

[BravoV]

Damn .. I'm in the phase of preparing to move my whole rig to full Linux as host from Windows, as Linux noob, you guys are scaring me. 

[legacy]

Damn, sorry to hear that, my sympathies.

Thanks for this. Fresh news, my partner is telling me just now that the problem was caused not by a bug in a script but rather by a bug in the bash shell.

Which explains what really happened: the env variable about the project's root was not evaluated at all, pointing rm -r to the /home/project rather than to /home/project/${var}

This symptom silently and randomly happens on our new stage4 after having recompiled the core system, which is planned weekly at the beginning of the week.

Yesterday, it was Monday 

(I seriously hates Gentoo for this), it seems the last stage4 for MIPS and HPPA is affected, so updating bash, python and perl (and probably other stuff) was not a good idea.

I have the backup of the previous stage4. I am going to buy a new hard drive, and to restore it blocking any update until the bash scripting engine is 100% sure working, keeping this harddrive here in read-only for data resuming purpose.

[I wanted a rude username]

Gentoo in production? You're a braver man than I.

Newer files being more likely to be corrupted may be due to fragmentation. Those recovery tools usually can only recover contiguous files. If the file spans more than one contiguous extent, they won't even have a good idea of where it ends.

Have you tried running the more promising looking files through gzrt to see if at least the first part of them is recoverable?

[techman-001]

To avoid such events for the future, ZFS filesystem can roll back changes, self repair RAID arrays and much more.

Agreed and probably why I haven't lost any data in the last three years.

About a year ago I made the incredibly stupid mistake of setting *all* my Workstation /home file attributes to read/write only, about 4GB of data with hundreds of directories. I had a brief WTF moment then remembered I had snapshots of /home  running and historical snapshots dating back the previous 2 years (zfstools). No gap in snapshot data exceeding 15 minutes meant I I would only lose data to a maximum of 15 minutes age.

About 5 minutes later everything was fixed.

[TK]

If you rewrite the code starting from the 30-day old backup, you can finish it in 2 weeks with a code that will be much better than the original one that was lost.  I suggest you take this route and not waste more time trying to recover the files from the maze that is going to be tracking the blocks and hoping for a partial recovery

[AndyC_772]

That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.

Do they not have copies of the changes they made?

[sokoloff]

You never back up?

The hard drive is mirrored, but this means the content is equal to both disks.

This is why “RAID is not a backup!” is a sysadmin aphorism. Some levels of RAID help data availability; others help access speed; none are backups.

If others have git repo copies, you should be able to recover most of the content. If they otherwise have working copies, you may be able to recover partially. If they have backups (TimeMachine or other?), you may have partial data around somewhere.

Otherwise, I like the “rewriting is often faster and better anyway” outlook.

[magic]

But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.

I believe ext3 journals writes to directory data blocks so full information about older revisions of this directory should be found in the journal.
So start with reviewing older revisions of /home inode. See what directory data blocks it points to (may be the same blocks as the current revision of /home). Find older revisions of those blocks in the journal. They will point you to your file inodes.

Of course no guarantee that file data are not overwritten.

[sleemanj]

Ouch.  I can't offer any remedial help that has not already been, but for future, borg backup is good, chunk de-duped compressed incremental backups very efficient on space and pretty fast, mounts backups as userfs for easy restore, and is easy to use.

Good luck with the recovery, worst case, at least now you know all the mistakes you made in the last month so maybe it only takes you 3 weeks re-coding :-/

[legacy]

ZFS filesystem does not compile on HPPA, MIPS, HPPA, and it consumes too much ram.
It's no go.

BTRFS, may be, but for now we have reason to say with ext3