Products > Computers

[solved] catastrophic event, file recovery on ext3

(1/10) > >>

legacy:
a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.

Thousands precious files lost, with Inodes overwritten by houndred of useless files  :palm: :palm:

When the Kernel primive "file_delete" is invoked, the ext3 filesystem doesn't leave the inode pointing to a file or to a folder, but it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.

This means, common recovery tools (like DiskTest) simply do not work with ext3, you have to cross your finger, and try special tools, like described here by the first person who tried to find a solution.


--- Code: ---2019-10-21--14-33-18---2019-10-21--14-35-13 - [ sys-fs/extundelete ] - success - @2.29.1/7.3.0
------------------------------------------------------------------------------------------------------
2019-10-21--15-21-12---2019-10-21--15-21-58 - [ sys-fs/ext3grep ] - failure - @2.29.1/7.3.0
      configure: error: ext3grep doesn't work on Big Endian systems
------------------------------------------------------------------------------------------------------

--- End code ---


* ext3grep
* extundelete
Unfortunately ext3grep does not even compile on BIG Endian systems ( I am on a HPPA workstation ), while extundelete is less advanced, hene I moved the harddrive to a x86 machine.

But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.

That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.

I am saying, the last four weekends of work is gone  :palm:

So ... I am trying searching withing blocks with the idea of grabbing some block containing the header of tarball so the block might be reversed somehow into the inode and ext3grep might try to resume the file.

Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz

SiliconWizard:
You never back up?

I wanted a rude username:
The first thing to do is remount read-only.

If the underlying storage is SSD, your data have probably been trimmed already.

If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.

legacy:

--- Quote from: SiliconWizard on October 21, 2019, 10:55:54 pm ---You never back up?

--- End quote ---

The hard drive is mirrored, but this means the content is equal to both disks. We usually save a snapshot dail, it's automatically done before midnight, and then we burn one-week-of-snapshots into a DVDRAM every Sunday afternoon; but in this case we didn't because we are still moving our storage stuff to a new place, therefore everything was manual, and the backup was planned to be monthly.

We are speaking about it from the mid of September to the mid of October  :-//

legacy:

--- Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pm ---The first thing to do is remount read-only.

--- End quote ---

Done, but it was too late. The script silently did its job before someone noticed what it was doing.


--- Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pm ---If the underlying storage is SSD

--- End quote ---

The couple of disks in RAID1 are electromechanical.


--- Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pm ---If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.

--- End quote ---

Precisely what I am trying to do  :D

Navigation

[0] Message Index

[#] Next page

There was an error while thanking
Thanking...
Go to full version