Author Topic: [solved] weird behavior with an ext4 filesystem (Read 2835 times)

legacy · « **on:** March 16, 2019, 09:53:48 pm »

so I have a 4GB microdrive, and I am trying to populate it with files

mkfs.ext4 /dev/sda3

mke2fs 1.43.9 (8-Feb-2018)
Creating filesystem with 968562 4k blocks and 242400 inodes
Filesystem UUID: ba8d7da1-7ad5-451a-97a3-ee7de5228d98
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information .. done

Code: [Select]

tar -xzf pack-01.tgz -C /mnt/disk2/
No space left on device

It says the disk is full, but it's not the truth since the disk still has free space

Code: [Select]

df | grep sda3
/dev/sda3              3747808   2694608    843104  77% /mnt/disk2

33% of space still available, let's check it

Code: [Select]

dd if=/dev/zero of=raw bs=1024

dd: writing `raw': No space left on device
1036813+0 records in
1036812+0 records out
1061695488 bytes (1.1 GB) copied, 1184.42 s, 896 kB/s
[code]

[code]
df | grep sda3
/dev/sda3              3747808   3731424         0 100% /mnt/disk2

the disk is not really full!

so, it sounds it's possible to allocate the space for a singular file, but not for a multitude of files.
This should have something to deal with block-size?

Any idea?

legacy · « **Reply #1 on:** March 16, 2019, 09:59:38 pm »

Code: [Select]

tune2fs -l /dev/sda3 | grep -i 'block size'
Block size:               4096

magic · « **Reply #2 on:** March 16, 2019, 10:01:23 pm »

Probably some bullshit like reserved space (only root can allocate it) or the FS run out of inodes (only 242400 files can exist on yours). Both can be checked with tune2fs I think and overridden with mkfs flags.

legacy · « **Reply #3 on:** March 16, 2019, 10:06:28 pm »

Quote from: magic on March 16, 2019, 10:01:23 pm

only root can allocate it

UID=0, root

and there is no quota

Quote

I think and overridden with mkfs flags.

I can only override the block-size. Ext3 and Ext4 use block sizes of { 1024, 2048, 4096 } bytes. The required size is chosen when setting up the file system with the mke2fs formatting program of Ext3/4. In theory, 4 KB is the maximum, and this block size corresponds to that of the kernel's memory pages in RAM, which makes paging easier for the operating system.

wilfred · « **Reply #4 on:** March 16, 2019, 11:41:17 pm »

You can use "sudo df -i /" to check you haven't run out of inodes.

How much space does your archive expand out to on a disk with ample space?

legacy · « **Reply #5 on:** March 17, 2019, 12:39:16 am »

Quote from: wilfred on March 16, 2019, 11:41:17 pm

How much space does your archive expand out to on a disk with ample space?

2.8Gbyte of 3.7GB available

legacy · « **Reply #6 on:** March 17, 2019, 12:41:17 am »

Code: [Select]

mkfs.ext4 -b 1024 /dev/sda3
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
242176 inodes, 3874248 blocks
193712 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=71041024
473 block groups
8192 blocks per group, 8192 fragments per group
512 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409, 663553,
        1024001, 1990657, 2809857

so, now the block-size is 1Kbyte and there are 242176 inodes ... let's check if it's ok

Nominal Animal · « **Reply #7 on:** March 17, 2019, 02:39:40 am »

How many files and directories does pack-01.tgz have? Say,

Code: [Select]

tar -tzf pack-01.tgz | wc -lIf that is over 200,000 you are definitely running out of inodes. For filesystems with lots of small files, I recommend

Code: [Select]

mkfs.ext4 -b 4096 -i 4096 -m 0 /dev/sda3 The -i parameter controls the number of inodes (bytes-per-inode), and having it the same size as the block size allows the maximum number of "files". You can also specify the exact number of inodes using the -N parameter. -m 0disables root-reserved blocks.

The number of inodes really matters for small filesystems, and filesystems used at near full capacity. For example, on mail servers with Maildir mailboxes (useful with IMAP clients!), you definitely want to allow the maximum number of small files, as each message will be a separate file. A web server tends to have slightly larger files (so fewer compared to the filesystem size). A file/media server tends to have large files. It is up to the system administrator to choose that setting. The default is pretty good, but -b SIZE -i SIZE is technically the safest choice, because it allows the maximum number of "files" on said filesystem. (The reason that is not the default, is that each potential inode reserves 128 or 256 bytes on disk; so, 200,000 inodes takes up 25.6 or 51.2 megabytes of disk space, permanently. Halving that, via -b SIZE -i SIZE/2 , halves both the maximum number of inodes/"files" and the permanently used space.)

If you do not intend to use extended attributes (xattr, or ACLs beyond the standard Unix ones), you can also use -I 128 to use specifically smaller inodes. In this case, the mount options ext_attr, acl, and user_xattr may not be available.

So, in your particular case, I suggest you try

Code: [Select]

mkfs.ext4 -b 4096 -i 4096 -I 128 -m 0 /dev/sda3

ejeffrey · « **Reply #8 on:** March 17, 2019, 02:50:30 am »

Yes, running out of inodes is a classic problem with small files on traditional Unix filesystems.

You can allocate more inodes at mkfs time with the -i flag or you can use -t to select a "use case" option that sets defaults.

Or you can use xfs or btrfs that don't have statically assigned inodes. One problem with ext4 is that inodes are now 256 bytes (compared to a default of 128 in ext3) so if you use -i 4096 you will actually waste a lot of space for the common case.

legacy · « **Reply #9 on:** March 17, 2019, 09:37:51 am »

Code: [Select]

df -i 
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda3             242176  242176       0  100% /mnt/disk2

yup, Ifree is equal to zero

legacy · « **Reply #10 on:** March 17, 2019, 09:47:08 am »

Quote from: Nominal Animal on March 17, 2019, 02:39:40 am

So, in your particular case, I suggest you try
Code: [Select]
mkfs.ext4 -b 4096 -i 4096 -I 128 -m 0 /dev/sda3

Code: [Select]

mkfs.ext4dev -b 1024 /dev/sda3    --> 242.176 inodes
mkfs.ext4dev -b 2048 /dev/sda3    --> 242.760 inodes
mkfs.ext4dev -b 4096 /dev/sda3    --> 242.400 inodes
mkfs.ext2 -b 1024 /dev/sda3    --> 242.176 inodes
mkfs.ext4 -b 4096 -i 4096 -I 128 -m 0 /dev/sda3    --> 968.640 inodes

more inodes

I don't need { xattr, ACLs, etc }.

legacy · « **Reply #11 on:** March 17, 2019, 09:53:21 am »

Quote from: ejeffrey on March 17, 2019, 02:50:30 am

you can use xfs or btrfs that don't have statically assigned inodes

xfs: maybe
btrfs: no, because it uses too many resources (on a router) and it's not supported with old kernels (2.6.37).

now we are on kernel v4.9.6, because it solves a lot of quirks with the miniPCI (and managing quirks is easier on kernel v4.* than on kernel 2.6.*), but we ought to be supporting old kernels, so .. xfs is known to be working, while btrfs is a "new entry".

anyway, xfs is on the way (the builder machine is compiling it), while I am going to try ext4 with the above setup

magic · « **Reply #12 on:** March 17, 2019, 10:07:12 am »

Isn't btrfs that filesystem which still gets data corruption bugs every other month?

ReiserFS could be more space efficient with tons of small files thanks to tail packing.

legacy · « **Reply #13 on:** March 17, 2019, 10:46:31 am »

To summarize what we are doing: we have created a Qemu/MIPSEL machine to native compile stuff for mips1-le (superstrict subset patched to be compatible with everything, Ci20, Chinese MIPS, Atheros, rb532, etc...).

it has been prepared with an old eabi2 portage, and it comes with a customized overlay with things that we are making specifically for our Sonoko project.

At the end of the process we obtained a rootfs to be loaded on a physical machine. and of course, the little rb532 is the chosen one because it's the router we are using.

Qemu can have and use a very large virtual hard drive with up to 20Gbyte of space, while the rb532 can only use microdrive (CF/Flash are not really compatible, due to a problem with the firmware) up to 6Gbyte, this due the compatibility with ATA-CF.

Seagate did a couple of 8Gbyte, and even a 12Gbyte Microdrive units, which are also compatible with the ATA-CF spec, and released with a CF2 connector, but these things are rare and expensive.

We currently have a stack of six 4Gbyte microdrive units, so this is the size of the disk layout we are aiming for

Code: [Select]

Disk: /dev/sda
                                     Size: 3.8 GiB, 4095737856 bytes, 7999488 sectors
                                            Label: dos, identifier: 0x00000000

    Device            Boot              Start           End       Sectors         Size       Id Type
>>  /dev/sda1                              63         16127         16065         7.9M       27 raw
    /dev/sda2                           16128        250991        234864       114.7M       82 swap
    /dev/sda3                          250992       7999487       7748496         3.7G       83 ext3

Anyway, I have already changed the building rule of our-builder, so now I can part the big package into small packages, instead of obtaining a supermassive-all-packaged-together big file.

Code: [Select]

cyt restore --profile=small --list mips1-le-stage4-rb532.list --device=/dev/sda3
formatting /dev/sda3:ext4 ... done 
restoring mips1-le-stage4-rb532-bin ... done
restoring mips1-le-stage4-rb532-boot ... done
restoring mips1-le-stage4-rb532-dev ... done
restoring mips1-le-stage4-rb532-etc.my ... done
restoring mips1-le-stage4-rb532-etc ... done
restoring mips1-le-stage4-rb532-export ... done
restoring mips1-le-stage4-rb532-home ... done
restoring mips1-le-stage4-rb532-kernel ... done
restoring mips1-le-stage4-rb532-lib ... done
restoring mips1-le-stage4-rb532-mnt ... done
restoring mips1-le-stage4-rb532-opt ... done
restoring mips1-le-stage4-rb532-proc ... done
restoring mips1-le-stage4-rb532-proof ... done
restoring mips1-le-stage4-rb532-root ... done
restoring mips1-le-stage4-rb532-sbin ... done
restoring mips1-le-stage4-rb532-src ... done
restoring mips1-le-stage4-rb532-sys ... done
restoring mips1-le-stage4-rb532-tftproot ... done
restoring mips1-le-stage4-rb532-tmp ... done
restoring mips1-le-stage4-rb532-usr-bin ... done
restoring mips1-le-stage4-rb532-usr-include ... done
restoring mips1-le-stage4-rb532-usr-libexec ... done
restoring mips1-le-stage4-rb532-usr-lib ... done
restoring mips1-le-stage4-rb532-usr-man ... done
restoring mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... done
restoring mips1-le-stage4-rb532-usr-overlay ... done
restoring mips1-le-stage4-rb532-usr-portage--eabi-2 ... done
restoring mips1-le-stage4-rb532-usr-portage ... done
restoring mips1-le-stage4-rb532-usr-sbin ... done
restoring mips1-le-stage4-rb532-usr-share ... done
restoring mips1-le-stage4-rb532-usr ... done
restoring mips1-le-stage4-rb532-usr-tmp ... done
restoring mips1-le-stage4-rb532-var ... done
restoring mips1-le-stage4-rb532-www ... done

These files were the content of the pack-01.tgz, and are easier to be handled

Code: [Select]

cyt countfile --list mips1-le-stage4-rb532.list
counting mips1-le-stage4-rb532-bin ... 118 files
counting mips1-le-stage4-rb532-boot ... 8 files
counting mips1-le-stage4-rb532-dev ... 1557 files
counting mips1-le-stage4-rb532-etc ... 965 files
counting mips1-le-stage4-rb532-etc.my ... 225 files
counting mips1-le-stage4-rb532-export ... 5 files
counting mips1-le-stage4-rb532-home ... 2 files
counting mips1-le-stage4-rb532-kernel ... 3 files
counting mips1-le-stage4-rb532-lib ... 352 files
counting mips1-le-stage4-rb532-mnt ... 4 files
counting mips1-le-stage4-rb532-opt ... 2 files
counting mips1-le-stage4-rb532-proc ... 1 files
counting mips1-le-stage4-rb532-proof ... 3 files
counting mips1-le-stage4-rb532-root ... 818 files
counting mips1-le-stage4-rb532-sbin ... 160 files
counting mips1-le-stage4-rb532-src ... 41997 files
counting mips1-le-stage4-rb532-sys ... 2 files
counting mips1-le-stage4-rb532-tftproot ... 2 files
counting mips1-le-stage4-rb532-tmp ... 0 files
counting mips1-le-stage4-rb532-var ... 50745 files
counting mips1-le-stage4-rb532-www ... 2 files
counting mips1-le-stage4-rb532-usr-bin ... 925 files
counting mips1-le-stage4-rb532-usr-include ... 2351 files
counting mips1-le-stage4-rb532-usr-lib ... 12577 files
counting mips1-le-stage4-rb532-usr-libexec ... 155 files
counting mips1-le-stage4-rb532-usr-man ... 4 files
counting mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... 61 files
counting mips1-le-stage4-rb532-usr-overlay ... 9542 files
counting mips1-le-stage4-rb532-usr-portage ... 103543 files
counting mips1-le-stage4-rb532-usr-portage--eabi-2 ... 1 files
counting mips1-le-stage4-rb532-usr-sbin ... 198 files
counting mips1-le-stage4-rb532-usr-share ... 23858 files
counting mips1-le-stage4-rb532-usr-tmp ... 0 files

counted 250186 files

Quote from: Nominal Animal on March 17, 2019, 02:39:40 am

How many files and directories does pack-01.tgz have? Say,
Code: [Select]
tar -tzf pack-01.tgz | wc -l

The new implemented feature can answer this question

Code: [Select]

counted 250186 files

legacy · « **Reply #14 on:** March 17, 2019, 10:59:14 am »

Quote from: magic on March 17, 2019, 10:07:12 am

ReiserFS could be more space efficient with tons of small files thanks to tail packing.

Good idea! I have never used it, and it would be an interesting alternative. There is another little problem with the rb532: the kernel size must be inferior to 7.9Mbyte because the bootstrapping firmware is not able to manage a bigger raw-file form the first partition of the microdrive, neither it's able to manage a bigger file from the tftpboot.

This was not a problem with kernel 2.6.*, but it's a big problem with kernel 4.* because they are really bloated.

So I have to check how big the kernel 4.9.6 will go with
- RiserFS as built-in only, by disabling { ext2, ext3, xfs } as built-in
- xfs as built-in only, by disabling { ext2, ext3, RiserFS } as built-in

NFS is compiled as a module, but the main filesystem needs to be compiled built-in.

legacy · « **Reply #15 on:** March 17, 2019, 11:28:51 am »

Code: [Select]

kanojo rb532-stage4-2019-03-15 # cyt restore --profile=small --list mips1-le-stage4-rb532.list --device=/dev/sda3 --verbose
formatting /dev/sda3:ext4 {-b 4096 -i 4096 -I 128 -m 0} ... done 
mounting /dev/sda3 as /mnt/disk2 ... done
restoring mips1-le-stage4-rb532-bin ... done
restoring mips1-le-stage4-rb532-boot ... done
restoring mips1-le-stage4-rb532-dev ... done
restoring mips1-le-stage4-rb532-etc.my ... done
restoring mips1-le-stage4-rb532-etc ... done
restoring mips1-le-stage4-rb532-export ... done
restoring mips1-le-stage4-rb532-home ... done
restoring mips1-le-stage4-rb532-kernel ... done
restoring mips1-le-stage4-rb532-lib ... done
restoring mips1-le-stage4-rb532-mnt ... done
restoring mips1-le-stage4-rb532-opt ... done
restoring mips1-le-stage4-rb532-proc ... done
restoring mips1-le-stage4-rb532-proof ... done
restoring mips1-le-stage4-rb532-root ... done
restoring mips1-le-stage4-rb532-sbin ... done
restoring mips1-le-stage4-rb532-src ... done
restoring mips1-le-stage4-rb532-sys ... done
restoring mips1-le-stage4-rb532-tftproot ... done
restoring mips1-le-stage4-rb532-tmp ... done
restoring mips1-le-stage4-rb532-usr-bin ... done
restoring mips1-le-stage4-rb532-usr-include ... done
restoring mips1-le-stage4-rb532-usr-libexec ... done
restoring mips1-le-stage4-rb532-usr-lib ... done
restoring mips1-le-stage4-rb532-usr-man ... done
restoring mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... done
restoring mips1-le-stage4-rb532-usr-overlay ... done
restoring mips1-le-stage4-rb532-usr-portage--eabi-2 ... done
restoring mips1-le-stage4-rb532-usr-portage ... done
restoring mips1-le-stage4-rb532-usr-sbin ... done
restoring mips1-le-stage4-rb532-usr-share ... done
restoring mips1-le-stage4-rb532-usr ... done
restoring mips1-le-stage4-rb532-usr-tmp ... done
restoring mips1-le-stage4-rb532-var ... done
restoring mips1-le-stage4-rb532-www ... done
unmounting /dev/sda3 ... done

Perfect! It worked!

Code: [Select]

# mount /dev/sda3 /mnt/disk2
# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda3             968640  247711  720929   26% /mnt/disk2

LapTop006 · « **Reply #16 on:** March 17, 2019, 01:28:50 pm »

Quote from: legacy on March 17, 2019, 09:53:21 am

now we are on kernel v4.9.6, because it solves a lot of quirks with the miniPCI (and managing quirks is easier on kernel v4.* than on kernel 2.6.*), but we ought to be supporting old kernels, so .. xfs is known to be working, while btrfs is a "new entry".

The 2.6 series is more than old at this point, 3.0 came out in mid-2011.

legacy · « **Reply #17 on:** March 17, 2019, 01:50:56 pm »

Quote from: LapTop006 on March 17, 2019, 01:28:50 pm

The 2.6 series is more than old at this point, 3.0 came out in mid-2011.

kernel 3.* don't solve PCI quirks and introduce A LOT of work for our custom drivers that need to be rewritten specifically for a new kernel, and since we do not have the time for doing, we avoid switching to a new kernel until it's really necessary or until it really gives an enhancement.

ejeffrey · « **Reply #18 on:** March 18, 2019, 04:35:30 pm »

Quote from: magic on March 17, 2019, 10:07:12 am

Isn't btrfs that filesystem which still gets data corruption bugs every other month?

ReiserFS could be more space efficient with tons of small files thanks to tail packing.

I think btrfs is pretty stable these days. Probably more so than ReiserFS ever was. But definitely not an option if you are stuck supporting older kernels.

But honestly I wouldn't actually recommend any of them for most applications. ext4 is the most stable and well tested linux FS, and while it isn't the most performant it doesn't have any glaring performance holes. Maybe the better way to say it is that it has similar performance characteristics to most "traditional" linux filesystems so the things that are slow are somewhat expected. Storage is to cheap to worry about storage efficiency. Just use ext4 and format it with the right options.

I would only consider xfs or btrfs if ext4 just won't cut it. For instance if you need the COW or variable stripe raid of btrfs or if you need the 64 bit inodes of xfs (>2B files). If you are just trying to use storage 20% more efficiently don't bother. Again, with btrfs you need to be on recent kernels.

I wouldn't recommend ResierFS under any circumstances.

legacy · « **Reply #19 on:** March 18, 2019, 04:44:42 pm »

Quote from: ejeffrey on March 18, 2019, 04:35:30 pm

recent kernels.

can you define recent kernels?

hppa (workstation): we are on 4.16.2
rs/p (home router): we are on 2.6.39
rb532 (Sonoko project): we are on 4.9.6 <--- this is the machine the topic is indirectly talking about
Akita (personal PDA): we are on 2.6.23

magic · « **Reply #20 on:** March 18, 2019, 07:58:15 pm »

Well, with two minutes of searching, from last summer:
https://patchwork.kernel.org/patch/10568533/

I don't know if dedupe is on by default, maybe it's a nothingburger. But it fits into the general pattern I perceive that btrfs issues just "keep happening".

Can't comment on ReiserFS nowadays, maybe you are right. I certainly tried that barely maintained Reiser4 port once and managed to break the FS within half an hour, but that's a different animal.

legacy · « **Reply #21 on:** March 18, 2019, 08:21:52 pm »

so btrfs will be considered for any real projects starting from kernel >= v5.1

legacy · « **Reply #22 on:** March 24, 2019, 09:53:52 am »

for comparing reasons, we compiled the same kernel with different filesystem support

filesystem   kernel size
ext4,   7.746.092
ext2 + ext4,   7.791.414
xfs,   7.992.833
raiser,   7.551.430

kernel 4.9.1


EEVblog Main Site	EEVblog on Youtube	EEVblog on Twitter	EEVblog on Facebook	EEVblog on Odysee

Author Topic: [solved] weird behavior with an ext4 filesystem (Read 2835 times)

Share me