-
a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.
Thousands precious files lost, with Inodes overwritten by houndred of useless files
When the Kernel primive "file_delete" is invoked, the ext3 filesystem doesn't leave the inode pointing to a file or to a folder, but it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.
This means, common recovery tools (like DiskTest) simply do not work with ext3, you have to cross your finger, and try special tools, like described here by the first person who tried to find a solution.Code: [Select]2019-10-21--14-33-18---2019-10-21--14-35-13 - [ sys-fs/extundelete ] - success - @2.29.1/7.3.0
------------------------------------------------------------------------------------------------------
2019-10-21--15-21-12---2019-10-21--15-21-58 - [ sys-fs/ext3grep ] - failure - @2.29.1/7.3.0
configure: error: ext3grep doesn't work on Big Endian systems
------------------------------------------------------------------------------------------------------- ext3grep
- extundelete
Unfortunately ext3grep does not even compile on BIG Endian systems ( I am on a HPPA workstation ), while extundelete is less advanced, hene I moved the harddrive to a x86 machine.
But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.
That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.
I am saying, the last four weekends of work is gone
So ... I am trying searching withing blocks with the idea of grabbing some block containing the header of tarball so the block might be reversed somehow into the inode and ext3grep might try to resume the file.
Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz -
You never back up?
-
The first thing to do is remount read-only.
If the underlying storage is SSD, your data have probably been trimmed already.
If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually. -
You never back up?
The hard drive is mirrored, but this means the content is equal to both disks. We usually save a snapshot dail, it's automatically done before midnight, and then we burn one-week-of-snapshots into a DVDRAM every Sunday afternoon; but in this case we didn't because we are still moving our storage stuff to a new place, therefore everything was manual, and the backup was planned to be monthly.
We are speaking about it from the mid of September to the mid of October
-
The first thing to do is remount read-only.
Done, but it was too late. The script silently did its job before someone noticed what it was doing.If the underlying storage is SSD
The couple of disks in RAID1 are electromechanical.If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.
Precisely what I am trying to do
-
Good luck ... you'll need it!
I was successful using this method a few times. grep -C, obviously.
There are also tools that scan a block device for magic such as gzip headers. This may be more efficient, if you can get it to work. -
The disk is currently attached to a mac-mini/x86, running Linux.
It's not mounted, only accessed as raw deviceCode: [Select]ext3grep /dev/sdb3 --search "DT11-v12-r2.tgz"
Running ext3grep version 0.10.2
Minimum / maximum journal block: 1545 / 35886
Loading journal descriptors... sorting... done
The oldest inode block that is still in the journal, appears to be from 1571661094 = Mon Oct 21 14:31:34 2019
Journal transaction 3453486 wraps around, some data blocks might have been lost of this transaction.
Number of descriptors in journal: 31738; min / max sequence numbers: 3452751 / 3453770
Blocks containing "DT11-v12-r2.tgz": 3310552 4215792Code: [Select]--search str Find blocks that contain the fixed string 'str'.
--search-inode blk Find inodes that refer to block 'blk'.
Unfortunaly the above blocks had no good luck to find inodes that refer to them
-
What did you use?
-
I treated the entire disk as a file* and searched not for filenames, but for plaintext file contents. (Also passed it through strings(1) first, but that's an oldskool performance optimisation, and raw grep may actually be quicker these days.) Something like:
# dd if=/dev/sdb3 bs=128k | grep -C100 "some string I remember from the text file" > /tmp/sdb3_matches.txt
If you add the -b parameter to grep, it will also tell you where on the disk the matching text is.
This is all super low level and also will take a long time to run (about two hours per terabyte). Ideally try other non-destructive options first.
*Mantra: "everything is a stream of bytes". Repeat until enlightened. -Linus Torvalds -
a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.
Thousands precious files lost, with Inodes overwritten by houndred of useless files
Damn, sorry to hear that, my sympathies.
I've lost data in the past from unforeseen circumstances so I feel your pain.
I do hope none of the Forth stuff you were writing has been lost. -
To avoid such events for the future, ZFS filesystem can roll back changes, self repair RAID arrays and much more.
-
Eep. I know the feelings you are going through right now. Be cautious, you can easily make things worse, time spent sitting down with a cup of tea is relatively cheap in comparison.
So you are saying that the metadata for your files might still exist, just spread out somewhere on the disk? Try asking for help on a relevant IRC channel, see if anyone can help. This forum isn't the best place to find kernel/filesystem experts.
Otherwise: your only hope is the horrid process of piecing together blocks of raw data to get some parts of files back. Files are often not contiguous, so as file sizes increase you (exp?) have lower chance of finding a file as a single piece. Tools like photorec are fun to play with and give a graphical description of this problem (where jpegs are recovered with a reasonable chance of appearing corrupt at some point in). -
Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz
Photorec knows how to find tar headers, including the gzipped variety.
If you point it at your entire disk you may be able to recover some of the ones that have not actually been overwritten (if there are any.) -
I do hope none of the Forth stuff you were writing has been lost.
yup, fortunately uForth and uLisp were on a different machine, with a copy on my PDA. But the the latest firmware is gone.
Fortunately, there is a girl in my team, and you know ... women are used to taking a lot of notes, therefore we have a copy on paper about what she had modified and fixed.
-
# dd if=/dev/sdb3 bs=128k | grep -C100 "some string I remember from the text file" > /tmp/sdb3_matches.txt
yes, I did something similar to grab part of C files and assembly files: all partially resumed.
But the real problem is how to get the whole tgz archive containing patches, documentation, etc.
All the mentioned tools require you to know the inode, and I have no idea about how to get it.
With the inode, and with luck, you can try to get the direct, indirect nodes pointing to data blocks.
I am talking about 20Mbyte of tarball | gzipped, on a filesystem whose datablock is 4K byte of size: it means a lot (~20M/4k) of blocks sparse around the whole block device.
So
-
Photorec knows how to find tar headers, including the gzipped variety.
Yup, I was thinking about something to recover files based on their headers, footers, and internal data structures. Sort of "file" combined with "grab". This process is commonly referred to as data carving.
PhotoRec seems based on Foremost, which doens't come with a defined pattern analysis for tgz, but the tool Scalpel has it!
Anyway, I tried them: no luckCode: [Select]mi2q resumed_tgz # ls
09506760.tgz 12115656.tgz 12737448.tgz 13134776.tgz 14371096.tgz 14969360.tgz 25218536.tgz 30216704.tgz
10152855.tgz 12115768.tgz 12781008.tgz 13140864.tgz 14379160.tgz 14969920.tgz 25424944.tgz 30219264.tgz
10152865.tgz 12115952.tgz 12793184.tgz 13145016.tgz 14404592.tgz 14975712.tgz 25501688.tgz 30517472.tgz
10953552.tgz 12116064.tgz 12793296.tgz 13145144.tgz 14422032.tgz 14975960.tgz 25514160.tgz 30585584.tgz
10953960.tgz 12117600.tgz 12793936.tgz 13168200.tgz 14422064.tgz 14983272.tgz 25911848.tgz 30585624.tgz
10954024.tgz 12127760.tgz 12793960.tgz 13173048.tgz 14429496.tgz 14983304.tgz 26810464.tgz 30585680.tgz
10959928.tgz 12127872.tgz 12822856.tgz 13176160.tgz 14443064.tgz 14985280.tgz 26810816.tgz 30917376.tgz
11003920.tgz 12128136.tgz 12823816.tgz 13191216.tgz 14445176.tgz 14985776.tgz 26811400.tgz 31672944.tgz
11003984.tgz 12128376.tgz 12824008.tgz 13193008.tgz 14473360.tgz 15050808.tgz 26828256.tgz 31973480.tgz
11004472.tgz 12178408.tgz 12842872.tgz 13202656.tgz 14474856.tgz 15050840.tgz 26859136.tgz 31973776.tgz
11004720.tgz 12214216.tgz 12844416.tgz 13226392.tgz 14480456.tgz 15053824.tgz 26860096.tgz 31973936.tgz
11005048.tgz 12214240.tgz 12878272.tgz 13253704.tgz 14497008.tgz 15065088.tgz 26861120.tgz 31974104.tgz
11008800.tgz 12215200.tgz 12892440.tgz 13257312.tgz 14502912.tgz 15074776.tgz 26907552.tgz 31975272.tgz
11014696.tgz 12237608.tgz 12892552.tgz 13278584.tgz 14503304.tgz 15074808.tgz 26925080.tgz 31977392.tgz
11017720.tgz 12238704.tgz 12893136.tgz 13309800.tgz 14521384.tgz 15082456.tgz 26925400.tgz 32156760.tgz
11019304.tgz 12264064.tgz 12893200.tgz 13346128.tgz 14528528.tgz 15088328.tgz 26925656.tgz 32157520.tgz
11019664.tgz 12264176.tgz 12893312.tgz 13387248.tgz 14528704.tgz 15088352.tgz 26925704.tgz 32169808.tgz
11019856.tgz 12265304.tgz 12899288.tgz 13402472.tgz 14528736.tgz 15089696.tgz 26925720.tgz 32875048.tgz
11036032.tgz 12269328.tgz 12901656.tgz 13500424.tgz 14531616.tgz 15089992.tgz 26925976.tgz 32875608.tgz
11036912.tgz 12279504.tgz 12901752.tgz 13533984.tgz 14531624.tgz 15091032.tgz 26926064.tgz 32875720.tgz
11087600.tgz 12307440.tgz 12905376.tgz 13697072.tgz 14589952.tgz 15091512.tgz 26926192.tgz 32922264.tgz
11087736.tgz 12324880.tgz 12913696.tgz 13753096.tgz 14591400.tgz 15094136.tgz 26926320.tgz 34556280.tgz
11107296.tgz 12324904.tgz 12914336.tgz 13753128.tgz 14596968.tgz 15140232.tgz 27087368.tgz 35274120.tgz
11107408.tgz 12364560.tgz 12914912.tgz 13793424.tgz 14610016.tgz 15140672.tgz 27590720.tgz 36124224.tgz
11107856.tgz 12368992.tgz 12930200.tgz 13800064.tgz 14653816.tgz 15142496.tgz 27591280.tgz 36132736.tgz
11144448.tgz 12369040.tgz 12962144.tgz 13802328.tgz 14678856.tgz 15145936.tgz 27593784.tgz 36135656.tgz
11192576.tgz 12369400.tgz 12966176.tgz 13803056.tgz 14679688.tgz 15147656.tgz 27597448.tgz 37041488.tgz
11203656.tgz 12405472.tgz 12966288.tgz 13964280.tgz 14687224.tgz 15148512.tgz 27597576.tgz 37444912.tgz
11276304.tgz 12405784.tgz 12966312.tgz 13964312.tgz 14716616.tgz 15149960.tgz 27772584.tgz 37595856.tgz
11336408.tgz 12406232.tgz 12967064.tgz 13964512.tgz 14716648.tgz 15151904.tgz 27772616.tgz 37599360.tgz
11436040.tgz 12406680.tgz 12969016.tgz 13966152.tgz 14726344.tgz 15152096.tgz 27774048.tgz 37617360.tgz
11549512.tgz 12410992.tgz 12973000.tgz 13967672.tgz 14729736.tgz 15152432.tgz 28345912.tgz 37618920.tgz
11549624.tgz 12491720.tgz 12999128.tgz 13968256.tgz 14733000.tgz 15162216.tgz 28461032.tgz 37629456.tgz
11659040.tgz 12495576.tgz 12999384.tgz 13969088.tgz 14778624.tgz 15166152.tgz 28461536.tgz 37646632.tgz
11661024.tgz 12495600.tgz 12999472.tgz 13970272.tgz 14778656.tgz 15169424.tgz 28465496.tgz 37669640.tgz
11661048.tgz 12496096.tgz 12999568.tgz 13972912.tgz 14779816.tgz 15173192.tgz 28466640.tgz 37674344.tgz
11662832.tgz 12506040.tgz 13000248.tgz 13975120.tgz 14780352.tgz 15173400.tgz 28467144.tgz 38024024.tgz
11822680.tgz 12535776.tgz 13001304.tgz 13975464.tgz 14783648.tgz 15173648.tgz 28532472.tgz 38917432.tgz
11844048.tgz 12535824.tgz 13015920.tgz 14069072.tgz 14786024.tgz 15174736.tgz 28533653.tgz 38917616.tgz
11844088.tgz 12599704.tgz 13016720.tgz 14069104.tgz 14786648.tgz 15175928.tgz 28535973.tgz 39415904.tgz
11844344.tgz 12599816.tgz 13020136.tgz 14071424.tgz 14796440.tgz 15195520.tgz 28542256.tgz 39570672.tgz
11868384.tgz 12600496.tgz 13020168.tgz 14094680.tgz 14797480.tgz 15197296.tgz 28542624.tgz 39626720.tgz
11885624.tgz 12601920.tgz 13020680.tgz 14095392.tgz 14798088.tgz 15197592.tgz 28543712.tgz 40094032.tgz
11895688.tgz 12621656.tgz 13022216.tgz 14095752.tgz 14799832.tgz 15199968.tgz 28790640.tgz 40094384.tgz
11895800.tgz 12629320.tgz 13025496.tgz 14098560.tgz 14800680.tgz 15201384.tgz 29022120.tgz 40631362.tgz
11895824.tgz 12631368.tgz 13030496.tgz 14116024.tgz 14802280.tgz 15208648.tgz 29026112.tgz 42173504.tgz
11905904.tgz 12632328.tgz 13038760.tgz 14116744.tgz 14837008.tgz 15211904.tgz 29026120.tgz 42813432.tgz
11915272.tgz 12632352.tgz 13044680.tgz 14140384.tgz 14837040.tgz 15212400.tgz 29321488.tgz 43116816.tgz
11918904.tgz 12632688.tgz 13044944.tgz 14256096.tgz 14838632.tgz 15213616.tgz 29321728.tgz 43848776.tgz
11933072.tgz 12632800.tgz 13045016.tgz 14256152.tgz 14839488.tgz 15233464.tgz 29321976.tgz 43922464.tgz
11933208.tgz 12635264.tgz 13045344.tgz 14256272.tgz 14840616.tgz 15237800.tgz 29322224.tgz 43924256.tgz
11952016.tgz 12662808.tgz 13058936.tgz 14257200.tgz 14840648.tgz 15240848.tgz 29322344.tgz 44101832.tgz
11993192.tgz 12662832.tgz 13090912.tgz 14259200.tgz 14841840.tgz 15284136.tgz 29322616.tgz 44276472.tgz
12003536.tgz 12667416.tgz 13092656.tgz 14259360.tgz 14880976.tgz 15301216.tgz 29323120.tgz 44863504.tgz
12045704.tgz 12667992.tgz 13092704.tgz 14259544.tgz 14909200.tgz 16203928.tgz 29323536.tgz 44863536.tgz
12045816.tgz 12668432.tgz 13093080.tgz 14259800.tgz 14909872.tgz 22935264.tgz 29323944.tgz 44863952.tgz
12046144.tgz 12669936.tgz 13095320.tgz 14275152.tgz 14911160.tgz 23929496.tgz 29327792.tgz
12046168.tgz 12669960.tgz 13095456.tgz 14275368.tgz 14914024.tgz 24261726.tgz 29717482.tgz
12049392.tgz 12670048.tgz 13120712.tgz 14295008.tgz 14939112.tgz 24274846.tgz 30200904.tgz
12083312.tgz 12703408.tgz 13122936.tgz 14297088.tgz 14939568.tgz 24519358.tgz 30204416.tgz
12083400.tgz 12733936.tgz 13123000.tgz 14365832.tgz 14947088.tgz 24521778.tgz 30206536.tgz
12083424.tgz 12735496.tgz 13123408.tgz 14370048.tgz 14963608.tgz 24522970.tgz 30209848.tgz
12101320.tgz 12736320.tgz 13128016.tgz 14370080.tgz 14968488.tgz 25213160.tgz 30214208.tgz
The 90% of these files is a corrupted tgz, and the 10% of them don't contain what I was looking for, but it contains a very old backup, that was deleted time ago.
And this is what I do not understand: the data carving stuff is able to find and resume tgz files made 2 months ago, and deleted 1 month ago, but it's not able to find and resume a tgz that was created 2 days ago and deleted yesterday.
It should have a high probability to have been overwritten, as well as recent deleted files should have a low probability to have been overwritten
Something tells me the LAST algorithm is not used for ext3, which, if it's true, will make me abandon this filesystem because it's worrying.
Combined with these annoying things:Code: [Select]EXT4-fs (sda3): couldn't mount as ext3 due to feature incompatibilities
which I still have to understand ... -
Damn .. I'm in the phase of preparing to move my whole rig to full Linux as host from Windows, as Linux noob, you guys are scaring me.
-
Damn, sorry to hear that, my sympathies.
Thanks for this. Fresh news, my partner is telling me just now that the problem was caused not by a bug in a script but rather by a bug in the bash shell.
Which explains what really happened: the env variable about the project's root was not evaluated at all, pointing rm -r to the /home/project rather than to /home/project/${var}
This symptom silently and randomly happens on our new stage4 after having recompiled the core system, which is planned weekly at the beginning of the week.
Yesterday, it was Monday
(I seriously hates Gentoo for this), it seems the last stage4 for MIPS and HPPA is affected, so updating bash, python and perl (and probably other stuff) was not a good idea.
I have the backup of the previous stage4. I am going to buy a new hard drive, and to restore it blocking any update until the bash scripting engine is 100% sure working, keeping this harddrive here in read-only for data resuming purpose.
-
Gentoo in production? You're a braver man than I.
Newer files being more likely to be corrupted may be due to fragmentation. Those recovery tools usually can only recover contiguous files. If the file spans more than one contiguous extent, they won't even have a good idea of where it ends.
Have you tried running the more promising looking files through gzrt to see if at least the first part of them is recoverable? -
To avoid such events for the future, ZFS filesystem can roll back changes, self repair RAID arrays and much more.
Agreed and probably why I haven't lost any data in the last three years.
About a year ago I made the incredibly stupid mistake of setting *all* my Workstation /home file attributes to read/write only, about 4GB of data with hundreds of directories. I had a brief WTF moment then remembered I had snapshots of /home running and historical snapshots dating back the previous 2 years (zfstools). No gap in snapshot data exceeding 15 minutes meant I I would only lose data to a maximum of 15 minutes age.
About 5 minutes later everything was fixed.
-
If you rewrite the code starting from the 30-day old backup, you can finish it in 2 weeks with a code that will be much better than the original one that was lost. I suggest you take this route and not waste more time trying to recover the files from the maze that is going to be tracking the blocks and hoping for a partial recovery
-
That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.
Do they not have copies of the changes they made? -
This is why “RAID is not a backup!” is a sysadmin aphorism. Some levels of RAID help data availability; others help access speed; none are backups.You never back up?
The hard drive is mirrored, but this means the content is equal to both disks.
If others have git repo copies, you should be able to recover most of the content. If they otherwise have working copies, you may be able to recover partially. If they have backups (TimeMachine or other?), you may have partial data around somewhere.
Otherwise, I like the “rewriting is often faster and better anyway” outlook. -
But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.
I believe ext3 journals writes to directory data blocks so full information about older revisions of this directory should be found in the journal.
So start with reviewing older revisions of /home inode. See what directory data blocks it points to (may be the same blocks as the current revision of /home). Find older revisions of those blocks in the journal. They will point you to your file inodes.
Of course no guarantee that file data are not overwritten. -
Ouch. I can't offer any remedial help that has not already been, but for future, borg backup is good, chunk de-duped compressed incremental backups very efficient on space and pretty fast, mounts backups as userfs for easy restore, and is easy to use.
Good luck with the recovery, worst case, at least now you know all the mistakes you made in the last month so maybe it only takes you 3 weeks re-coding :-/ -
ZFS filesystem does not compile on HPPA, MIPS, HPPA, and it consumes too much ram.
It's no go.
BTRFS, may be, but for now we have reason to say with ext3