Fixing fsck errors on headless system

[nali]

I have an Western Digital NAS which has been repurposed and now runs a few things under Debian 11 (ARM). It now has developed some filesystem errors, presumably after some multiple power outages we've had recently.

Any ideas how to repair? I can't use fsck as the only interface I have is through the network, so I can't unmount the active partition. I tried /forcefsck, which does force a fsck on reboot but that still reports errors.

The ext4 filesystem is black magic to me, so I've no idea how serious or fixable the errors are.

Output from e2fsck after forcefsck:

Code: [Select]

e2fsck 1.46.2 (28-Feb-2021)
Warning!  /dev/sda19 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +1728534 -1789536
Fix? no

Free blocks count wrong for group #54 (18214, counted=18213).
Fix? no

Free blocks count wrong (4033656, counted=4031438).
Fix? no

Free inodes count wrong (1568096, counted=1568074).
Fix? no

Padding at end of inode bitmap is not set. Fix? no


System: ********** WARNING: Filesystem still has errors **********


       72064 inodes used (4.39%, out of 1640160)
         269 non-contiguous files (0.4%)
          70 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 66582/106
     2526344 blocks used (38.51%, out of 6560000)
           0 bad blocks
           1 large file

       56599 regular files
        9869 directories
          10 character device files
           1 block device file
           7 fifos
          13 links
        5371 symbolic links (5152 fast symbolic links)
         220 sockets
------------
       72090 files


[magic]

Checking a mounted filesystem can give spurious errors but dunno if that's the case here.
If in doubt, you can pull the disk out of the machine and test the unmounted filesystem elsewhere.
Or see if you can boot a different OS (from a pendrive or something) on this box.

[Psi]

If you have a spare drive you could make a binary copy with DD and then fsck the copy you are not currently booted from.

If you do some digging you may be able to find a linux live CD/DVD/USB that allows SSH access with default user/pass once booted, So you could ssh into the live cd and mount/fsck the drive from there.

[nali]

If you have a spare drive you could make a binary copy with DD and then fsck the copy you are not currently booted from.

If you do some digging you may be able to find a linux live CD/DVD/USB that allows SSH access with default user/pass once booted, So you could ssh into the live cd and mount/fsck the drive from there.

That's interesting, I didn't know you could fsck an image. I haven't spent too much time looking at this stage as it's still working OK, but I did look at making a Debian live boot - unfortunately they only supply images for amd64.

[DiTBho]

With kernel support, a loopback mounted image works as a block-device.

[audiotubes]

https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/

[edavid]

That's interesting, I didn't know you could fsck an image. I haven't spent too much time looking at this stage as it's still working OK, but I did look at making a Debian live boot - unfortunately they only supply images for amd64.

Don't you have any PC?  You could pull the drive out of the NAS and connect it to the PC (running the live boot distribution) to do the fsck.

[nali]

https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/

Are you referring to using /forcefsck (as already tried per my OP)?

[nali]

That's interesting, I didn't know you could fsck an image. I haven't spent too much time looking at this stage as it's still working OK, but I did look at making a Debian live boot - unfortunately they only supply images for amd64.

Don't you have any PC?  You could have pulled the drive out of the NAS and connected it to the PC to do the fsck.

Yes, I have 2 workstations, but they are working tools and I'm disinclined to mess around with them unless I have to. The NAS is a dinky WD MyCloud Home so doesn't have pluggable drives, it'll be easier just making a boot stick with network support and going from there. The point of my post was to see if there was a trick I'm missing - seems not by the look of it.

[edavid]

The NAS is a dinky WD MyCloud Home so doesn't have pluggable drives, it'll be easier just making a boot stick with network support and going from there.

Sure it does, you just have to get the case open (which I admit is not much fun, but also not difficult).

[audiotubes]

https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/

Are you referring to using /forcefsck (as already tried per my OP)?

No, I meant that you can pass options to fsck so it will fix the errors instead of taking the default of no for everything, which seems to be happening in the output you posted.

[magic]

I don't think that it will attempt repairing a mounted filesystem, even if it's mounted read only.

The only real options without moving the disk to a different machine are:
1. booting another OS on the same machine
2. creating an initramfs which contains fsck and booting into it with init=/bin/sh kernel parameter to run a shell instead of the usual init script

edit
But if it's headless then pretty much only 1 is viable, unless you also put ssh, network scripts and other such into the initramfs - quite a hassle.

[audiotubes]

I can't remember but I think the root system may be mounted R/O until some point in the boot process and then remounted R/W. And I expected that after setting the correct options to answer yes to everything, the box would be rebooted.

[magic]

With kernel support, a loopback mounted image works as a block-device.

If you have an image file then you can run fsck on the file, no need for loop.

But that's futile too, because an image of a filesystem mounted RW may easily be inconsistent even if the original filesystem was OK.

edit
If you unmount cleanly, then mount RO, then fsck the mounted filesystem or make an image and fsck the image, there should be no errors. If there are errors, something is wrong. But as soon as you mount RW, all bets are off until it's unmounted.

[grumpydoc]

The NAS is a dinky WD MyCloud Home so doesn't have pluggable drives, it'll be easier just making a boot stick with network support and going from there. The point of my post was to see if there was a trick I'm missing - seems not by the look of it.

If the PCB looks like anything this:



Then the arrow points to the serial port for the console ("TTL", not RS-232 levels), even if your board looks nothing like the above chances are there's a serial port.

Though if you've got it that far apart plugging the drive into another machine using a SATA<->USB adapter becomes an option.

I'm assuming the board uses GRUB to boot Linux?

If you can get a console you should be able to add (might be systemd specific but AFAIK Debian 11 uses systemd) "fsck.mode=force fsck.repair=yes" to the boot command line.

If you don't want to dismantle your NAS go to /boot/grub2/grub.cfg and find the command line for the kernel you are booting and add the above, then reboot.

Good luck

[nali]

Ah well I was bored last night so decided sod it and take the thing apart. They do not want people to get into those things!
Screws under stuck-on rubber bumpers. OK, that's pretty standard.
Really flimsy but tightly latched internal clips holding the two halves of the case together. Even with multiple spudgers the only way to get it apart is to break clips on one side.
The chassis is a snug push fit inside one half of the case. There are sprung clips hidden inside

After all that, they kindly provide a nice convenient UART on a 0.1" header!

I finally managed to fsck it. @magic guessed right earlier, the errors were false indications, file system was clean. Ah well.

[grumpydoc]

I finally managed to fsck it. @magic guessed right earlier, the errors were false indications, file system was clean. Ah well.

Which brings us to the question I suppose we should have asked initially - what lead you to think there were filesystem errors in the first place?

Looking back I see that fsck you ran did not replay the journal which might have explained some of the errors it reported.

[nali]

I finally managed to fsck it. @magic guessed right earlier, the errors were false indications, file system was clean. Ah well.

Which brings us to the question I suppose we should have asked initially - what lead you to think there were filesystem errors in the first place?

Looking back I see that fsck you ran did not replay the journal which might have explained some of the errors it reported.

We had some power issues during storm Darragh (which is not untypical round here) which resulted in intermittent power over the course of a few days which apparently was either caused or exacerbated by a substation fire. Often after a power cut we get some brief off/on events as they reroute power or whatever, so I normally turn things off until power has been back for a few hours but in this case I got caught a couple or three times.

Anyway to answer your question, Samba wasn't running. Looking at the logs it was reporting insufficient inodes which I assumed was file corruption so ran fsck (which reported 5% inodes used), then went down the rabbithole. There's a good chance there was some corruption and using /forcefsck earlier acrually fixed it, I don't know.

Yea, I know I should get a UPS...