File systems!

[SiliconWizard]

On Linux, what alternative(s) to ext4 are you using, if anything? Why? And finally, How much experience you have with it and have you run into any particular issue?

My interest would not be (for the time being at least) for the root partition, but just for "data", including very small to very large files.

[RoGeorge]

Big fan of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

[tunk]

The default in RHEL (+clones) is xfs. Never had any problems.

[DiTBho]

On Linux, what alternative(s) to ext4 are you using, if anything?

• for both my routers and servers: xfs-v4
• on my graphical terminal: ext2

Why?

Compared to ext2 and ext3 (=ext2+journalling), I prefer the design of xfs, it's more linear, and it better suits 4GB microdrives (I am MD addicted). For me, xfs is more reliable and offers some practical possibilities to resume your files from a catastrophic filesystem disaster, while with ext3 you have rather zero possibilities.

ext2 is simpler, consumes less ram, and has less cache pressure, so it's perfect as ram-rootfs.
Also I don't even need any "journaling" for things that are "volatile" (only stay in ram).

How much experience you have with it and have you run into any particular issue?

worst encountered catastrophic filesystem disasters:

• A terrible bug in Bash made a script go nuts to the point it deleted a whole subtree. Files went lost because ext3 doesn't offer "undelete", so when you delete something ... you should immediately freeze and unmount your filesystem to later "raw" analysis, otherwise the things just deleted things are still on the harddisk as "data" content, but since their inodes are no more registered your data will be overwritten as soon as the kernel issues a data allocation, even for temporary files.
  
  I got my lost files back with a C program I wrote myself to search raw in /dev/sdc2 for patterns and grab the whole block, then rebuild every lost file piece by piece. Thank god, the lost files were only assembly and C files, so "text" files rather than "binary" files.
• the PCI-X-SATA controller on my HPPA big iron went nuts due to a bug with the CPU bus caching causing the CPU to randomly stuff holes of { 0x00 0x00 0x00 ... } here and there into files. Again here you have little survival chances with ext3 because if an inode is corrupted and seen as "unregistered" you quickly have things overwritten

I have ZERO experiences with ext4, anyway

[DiTBho]

The default in RHEL (+clones) is xfs. Never had any problems.

just, v3? v4? v5?
they behave differently, and they are not 100% down-compatible

[tunk]

Just checked, looks like on EL7+8+9 it's V5.

[SiliconWizard]

Big fun of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

I am considering ZFS, but I've heard again and again about licensing issues and performance issues with Linux. Not sure what the current state really is. And yes, it's also known to be memory hungry.
I remember threads about ZFS and it wasn't all rosy.

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

And then there is btrfs, but I'm not sure it's "stable" enough for any serious use yet. And it's kinda slow.

Never had any problem with ext4, but I'm just looking at alternatives.

[Ed.Kloonk]

Big fun of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

I am considering ZFS, but I've heard again and again about licensing issues and performance issues with Linux. Not sure what the current state really is. And yes, it's also known to be memory hungry.
I remember threads about ZFS and it wasn't all rosy.

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

And then there is btrfs, but I'm not sure it's "stable" enough for any serious use yet. And it's kinda slow.

Never had any problem with ext4, but I'm just looking at alternatives.

My view is that the big distros like Ubuntu don't want to have to clean it up the code to be industrial. If they have to put in that much energy, it would be better to rebuild a new brand from scratch and it would take too long to catch up.

I don't know why Ubuntu don't grab the bull by the horns with a clean snapshotting FS. But on a lot of distros, everyone did consider EXT4 'experimental' for what seemed like an eternity.

[RoGeorge]

AFAIK, none of the enumerated alternatives there are as good and as stable as ZFS.  Some are copycat implementations, but with incomplete or less features when compared to ZFS.

Put aside any fanboy-ism, being it pro or con, and try ZFS for yourself.  It's not trivial to learn its concepts, but once you figured that out you'll never want to go back.  ZFS can do much more than a typical file system like ext4.

ZFS is the most robust file system in existence, while also being the most feature rich, and the easiest to administrate.  Everything can be done on the fly, it's incredibly flexible and rock solid.  Linus might have had his licensing concerns to not include ZFS in the kernel, but I doubt any of those arguments are valid for you.

[DiTBho]

FreeBSD OS (where ZFS is builtin)

With Linux, ZFS is on FUSE.
You can dedicate a computer running FreeBSD to ZFS and export the filesystem via NFS 

[DiTBho]

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

You have to tune it, but XFS is based on B*tree, so it's a very flexible solution for small, medium, and large files.
Better than single, double, and triple indirection (ext2, ext3, ext4), at least.
It consumes less ram, and it's simpler than BtrFS and ZFS.

[RoGeorge]

Keeping two machines might be a burden, and with NFS most of the ZFS goodies will be lost, for example sending snapshots between machines.  ZFS knows to do that natively, with compression and incremental, with deduplication too if requested.

There's nothing wrong with ZFS on Linux.  Install it on Linux, and if you need another ZFS machine with BSD, just let them both do their ZFS things, without NFS as middle man.

[DiTBho]

sending snapshots between machines

not a problem, you can ssh to the NAS to manage snapshots.

There's nothing wrong with ZFS on Linux

zfsonlinux dot org suggested that one can get around this "license" issue by using zfs-fuse, but concerning *technical issues* incorporating zfs into fuse (F-ilesystem in USE-rspace) in userspace means less performance and more overhead.

Not a problem, just annoying if you move a lot of data and you don't have super powerful hw.

So, if you dare, there are out-of-tree Linux kernel modules available from zfsonlinux, and you can built-in it directly (just, it still has a regression test suite, so be careful)

[magic]

I tend to use XFS because the fixed number of inodes in ext4 bothers me, even if I have never had problems because of it.
Works so far, fingers crossed.

That being said, I'm not 100% sure how well those "clever" filesystems would tolerate data loss, for example the root of the inode tree or something like that. With ext4 at least the structure is dead simple (inodes are held in arrays in well known positions).

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

What does "optimized" mean and which filesystems are "optimized"?

Regarding small files, I know of two aids:
1. storing file data inside the inode structure itself, which may work for files no larger than tens of bytes; I think ext4 can do it
2. "tail packing", i.e. storing the final incomplete block of the file together with other such "tails" of other files - unique to ReiserFS

Somewhat related:
3. with ext4 you are forced to decide the maximum number of files at mkfs time.

[Monkeh]

(just, it still has a regression test suite, so be careful)

So at the point authors stop bothering checking for regressions, they become impossible?

ZFS works fine. No need for FUSE.

[RoGeorge]

there are out-of-tree Linux kernel modules available from zfsonlinux, and you can built-in it directly (just, it still has a regression test suite, so be careful)

The one on Ubuntu is OpenZFS, I think (not sure).  There's no need to compile it, can be installed with sudo apt install zfsutils-linux.  No idea where it runs, but it's certainly not slow.  Seems faster and less demanding (for example at high speed torrents over 1GBps FO), but I might be biased, didn't benchmark it.

On what hardware do you use ZFS?  Does ZFS runs slower for your machine, or you were considering the theoretical  aspects?

[DiTBho]

@RoGeorge
Talking about GNU/Linux in general, there are two possibilities:

• ZFS-FUSE, ZFS in userspace, was the first attempt to workaround the license issue
• ZFS-kmod, ZFS integrated into kernel space, recent solution, still with regression test-suites and several debug flags, which smells of "still experimental"

Both of those solutions were/are too slow on my HPPA C3750, and even slower on my PowerMac-G4, which, to make things worse, can only address 2GByte of ram.

ZFS performs largely better on my boss's Tyan POWER9 machine with 32 PPC64 cores @ 3Ghz and 128GByte of ram, but to make life easier, I set up a FreeBSD-based ZFS-NAS for her on cheaper x86 hardware, and ZFS is just the "RAID second layer"; so from her end-user point of view is just an NFS black box unit attached to a 10Gps connection that weekly issues automatic backups.

Internally the big PPC64 iron uses XFS-v4.

[Monkeh]

Both of those solutions were/are too slow on my HPPA C3750, and even slower on my PowerMac-G4, which, to make things worse, can only address 2GByte of ram.

20 year old systems are slow. Who knew.

[Whales]

Every few years I look at alternatives, but I always seem to come back to ext4 again.

One of the most important features of a filesystem is the "love" of its developers.  If a filesystem is developed as a job: they couldn't GAF if you lose data (unless you're paying customer).  If a filesystem is a labour of love: the devs actually care if something happens to you and your data and might even put warnings/errata into the formatting tools themselves (eg "don't use this feature it's unstable") rather than hiding them elsewhere in the docs.


ext4:

 - I found default fixed inode count to not be enough on small size USB drives for a specific workload (gentoo install?).  Reformat required to change
 - My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there and surprisingly easy (lucky?) to fully recover after I found its start and finish address.

XFS:
 - every time I look at benchmarks it seems slower than ext4 for my use cases

btrfs:

 - Cool fun feature: transparent file compression (was useful on my old 16GB fixed disk chromebook)
 - Lost lots of data once, probably 10 years back.  Not sure if things are better since then?
 - I don't get a feeling that the dev community for this one cares too much.  Vague impression that they're doing it as a paid job to tick boxes, not for the love of their data and users.  Perhaps the business case for this one is to make an alternative to ZFS so that licensing for ZFS is cheaper/easier?  Actual reliability doesn't matter.

ext2:

 - always shutdown cleanly!  Otherwise much drama.  Only ever seen on a historical environment I once virtualised + then killed the VM rather than waiting for shutdown.

exfat:

 - default block size is very large for TB disks.  Made a large collection of small wav files I needed for an experiment take up 400GB (on-disk) instead of 80GB (real)

Bcachefs will be interesting to see.  It promises some of the features of btrfs, but possibly made with more love?  I'll still tread cautiously, backups are great but still don't avoid (1) annoying time and effort spent recovering, (2) data lost between backups intervals and (3) unnoticed data loss/rot (not recoverable unless you keep all backups forever).

[DiTBho]

ext4: My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there nd surprisingly easy (lucky?) to fully recover after I found its start and finish address.

makes no bonus specifically for ext4, if you know where a partition starts/ends, you can easily recover the partition table

p.s.
daily case, ext4 on my MIPS32@400Mhz router is 2x slower than xfs-v4

[DiTBho]

ext2: always shutdown cleanly!  Otherwise much drama

shutdown -> /sbin/init -> (/etc/init.d/disk,stop): { close all apps with open files; sync; remount,ro; }

'sync' is enough

[paulca]

Right now my bug bear is InfluxDB.  I'm hitting it with over 100 metrics a second at the moment and while it's a memory mapped DB the hard-disk is constantly pinging and "io-top" shows about 10% CPU from the ext4 journald, sometimes higher.

This is onto an SSD.  I have to look into it.  It might not all be "write".  If the DB does not all fit in memory it will churn around if reads and writes don't exist in memory.

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources. 

I wasn't ready to go full server supervisor OS like unRaid or what not, mostly because unraid is payware and the alternatives I wasn't sure on.

A HUGE amount of guides and how-tos for ZFS and other multi-disk aggregation filesystems seem to suggest you are starting with a pile of brand new disks.  It's so lame and a complete cop out that nobody wants to cover the 80% of the work .... migration.  They make it sound easy, but only cover the 20% of default, empty, brand new hardware installation.  If you are starting with 2Tb spread across 6 disks and you want to migrate it....  that's not so easy.

[DiTBho]

memory mapped DB the hard-disk is constantly pinging

physical ramdisks like Hyperdrive, iRAM, and volatile storage on physical iSCSI connections have the same problem, and ext2 compiled with --minimal is largely better for them.

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources. 

Yup, just consider that zfs-kmod v2.* can be configured with

+ minimal
+ nls
no pam, no libressl ---> do you really need to crypt? save from loading encryption stuff for home datasets
no rootfs ---> do you really need to keep the whole system on a "ZFS" partition(2)?
no split-usr ---> are you really so paranoid that you want this(1)?
no custom-cflags
no debug ---> save memory and CPU cycles, unless you think ZFS is buggy, and in this case ... do you trust it?
no python ---> do you really need to add optional support/bindings for Python? save memory and CPU cycles!
no static-libs ---> save memory
no test-suite ---> do you trust ZFS? how much? enough? so you don't need this

it helps a bit 

(1) when enabled, it spends precious CPU cycles maintaining { /bin, /lib*, /sbin, /usr/sbin } separately from { /usr/bin, /usr/lib* }

(2)
partition1: /boot, on "ext2,ro" { /boot/kernel, /boot/kernel_failsafe }
partition2: system on "ext2,ro"  { /bin, /lib*, /sbin, /usr/sbin, /usr/bin, /usr/lib* }
partition3: /user/$home on ZFS { /user/root, /user/toor, /user/admin, /user/you, ... }
partition4: /work (shared among users) on ZFS

sometimes you need to remount "partition2,rw" to update/install new packages.

[Monkeh]

btrfs:

 - Cool fun feature: transparent file compression (was useful on my old 16GB fixed disk chromebook)
 - Lost lots of data once, probably 10 years back.  Not sure if things are better since then?
 - I don't get a feeling that the dev community for this one cares too much.  Vague impression that they're doing it as a paid job to tick boxes, not for the love of their data and users.  Perhaps the business case for this one is to make an alternative to ZFS so that licensing for ZFS is cheaper/easier?  Actual reliability doesn't matter.

btrfs is pretty slow, still known to have major issues in striped setups, and is incredibly fragile in anything except a mirror. Any corruption results in a completely broken filesystem for which no functional recovery tools exist. Dead end so far.

A HUGE amount of guides and how-tos for ZFS and other multi-disk aggregation filesystems seem to suggest you are starting with a pile of brand new disks.  It's so lame and a complete cop out that nobody wants to cover the 80% of the work .... migration.  They make it sound easy, but only cover the 20% of default, empty, brand new hardware installation.  If you are starting with 2Tb spread across 6 disks and you want to migrate it....  that's not so easy.

Why would anyone be overly concerned with providing a migration path between two totally different filesystems on the same medium in an inherently destructive process?

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources.

If you're not doing anything, it's not using anything. If you are, it's still a tiny fraction of the mechanical power of the drives. Yes, it uses relatively more resources than other filesystems, because it's doing more - not much else out there is doing continuous checksumming to make sure your precious data actually exists. If you don't care about the data, you may as well just use ext2 on a single drive!

For a general purpose filesystem, use ext4. If you have plans on keeping data, use ZFS in a suitably redundant configuration.

[Marco]

Keeping two machines might be a burden, and with NFS most of the ZFS goodies will be lost, for example sending snapshots between machines.

It's a pity no major distribution ever threw some support at Wyng, which does the same thing for LVM2 (not a filesystem, just a block layer). ZFS and Linux will always bite each other, but LVM2+XFS can provide pretty much the same thing ... it just needs to be packaged up in a more user friendly way.

Perhaps if Chromebooks start leaning more heavily into running Linux applications, Google will do it. They never needed it for ChromeOS in the past, because the user isn't supposed to have much more local than a download directory, with everything else in the cloud.