File systems!

[SiliconWizard]

On Linux, what alternative(s) to ext4 are you using, if anything? Why? And finally, How much experience you have with it and have you run into any particular issue?

My interest would not be (for the time being at least) for the root partition, but just for "data", including very small to very large files.

[RoGeorge]

Big fan of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

[tunk]

The default in RHEL (+clones) is xfs. Never had any problems.

[DiTBho]

On Linux, what alternative(s) to ext4 are you using, if anything?

• for both my routers and servers: xfs-v4
• on my graphical terminal: ext2

Why?

Compared to ext2 and ext3 (=ext2+journalling), I prefer the design of xfs, it's more linear, and it better suits 4GB microdrives (I am MD addicted). For me, xfs is more reliable and offers some practical possibilities to resume your files from a catastrophic filesystem disaster, while with ext3 you have rather zero possibilities.

ext2 is simpler, consumes less ram, and has less cache pressure, so it's perfect as ram-rootfs.
Also I don't even need any "journaling" for things that are "volatile" (only stay in ram).

How much experience you have with it and have you run into any particular issue?

worst encountered catastrophic filesystem disasters:

• A terrible bug in Bash made a script go nuts to the point it deleted a whole subtree. Files went lost because ext3 doesn't offer "undelete", so when you delete something ... you should immediately freeze and unmount your filesystem to later "raw" analysis, otherwise the things just deleted things are still on the harddisk as "data" content, but since their inodes are no more registered your data will be overwritten as soon as the kernel issues a data allocation, even for temporary files.
  
  I got my lost files back with a C program I wrote myself to search raw in /dev/sdc2 for patterns and grab the whole block, then rebuild every lost file piece by piece. Thank god, the lost files were only assembly and C files, so "text" files rather than "binary" files.
• the PCI-X-SATA controller on my HPPA big iron went nuts due to a bug with the CPU bus caching causing the CPU to randomly stuff holes of { 0x00 0x00 0x00 ... } here and there into files. Again here you have little survival chances with ext3 because if an inode is corrupted and seen as "unregistered" you quickly have things overwritten

I have ZERO experiences with ext4, anyway

[DiTBho]

The default in RHEL (+clones) is xfs. Never had any problems.

just, v3? v4? v5?
they behave differently, and they are not 100% down-compatible

[tunk]

Just checked, looks like on EL7+8+9 it's V5.

[SiliconWizard]

Big fun of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

I am considering ZFS, but I've heard again and again about licensing issues and performance issues with Linux. Not sure what the current state really is. And yes, it's also known to be memory hungry.
I remember threads about ZFS and it wasn't all rosy.

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

And then there is btrfs, but I'm not sure it's "stable" enough for any serious use yet. And it's kinda slow.

Never had any problem with ext4, but I'm just looking at alternatives.

[Ed.Kloonk]

Big fun of ZFS here.  Works flawless, and has features most other filesystems don't even dream about.  Read what it can do, and you'll probably want/need at least a handful of those features.

Have ZFS on the daily desktop (Kubuntu with ZFS on root) and on another FreeBSD OS (where ZFS is builtin).  Ubuntu with ZFS on root can do rollbacks at boot (back and forth), if needed.  That's an Ubuntu experimental feature, but similar functionality can be achieved with other tools.  Both the SSD and the HDD seems to be fine with it.  ZFS is SSD friendly by its nature.

It is said ZFS is memory hungry (particularly when deduplication is enabled), though I don't need/use sector deduplication, and never noticed any memory hogging (the desktop has 32GB RAM). 

I am considering ZFS, but I've heard again and again about licensing issues and performance issues with Linux. Not sure what the current state really is. And yes, it's also known to be memory hungry.
I remember threads about ZFS and it wasn't all rosy.

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

And then there is btrfs, but I'm not sure it's "stable" enough for any serious use yet. And it's kinda slow.

Never had any problem with ext4, but I'm just looking at alternatives.

My view is that the big distros like Ubuntu don't want to have to clean it up the code to be industrial. If they have to put in that much energy, it would be better to rebuild a new brand from scratch and it would take too long to catch up.

I don't know why Ubuntu don't grab the bull by the horns with a clean snapshotting FS. But on a lot of distros, everyone did consider EXT4 'experimental' for what seemed like an eternity.

[RoGeorge]

AFAIK, none of the enumerated alternatives there are as good and as stable as ZFS.  Some are copycat implementations, but with incomplete or less features when compared to ZFS.

Put aside any fanboy-ism, being it pro or con, and try ZFS for yourself.  It's not trivial to learn its concepts, but once you figured that out you'll never want to go back.  ZFS can do much more than a typical file system like ext4.

ZFS is the most robust file system in existence, while also being the most feature rich, and the easiest to administrate.  Everything can be done on the fly, it's incredibly flexible and rock solid.  Linus might have had his licensing concerns to not include ZFS in the kernel, but I doubt any of those arguments are valid for you.

[DiTBho]

FreeBSD OS (where ZFS is builtin)

With Linux, ZFS is on FUSE.
You can dedicate a computer running FreeBSD to ZFS and export the filesystem via NFS 

[DiTBho]

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

You have to tune it, but XFS is based on B*tree, so it's a very flexible solution for small, medium, and large files.
Better than single, double, and triple indirection (ext2, ext3, ext4), at least.
It consumes less ram, and it's simpler than BtrFS and ZFS.

[RoGeorge]

Keeping two machines might be a burden, and with NFS most of the ZFS goodies will be lost, for example sending snapshots between machines.  ZFS knows to do that natively, with compression and incremental, with deduplication too if requested.

There's nothing wrong with ZFS on Linux.  Install it on Linux, and if you need another ZFS machine with BSD, just let them both do their ZFS things, without NFS as middle man.

[DiTBho]

sending snapshots between machines

not a problem, you can ssh to the NAS to manage snapshots.

There's nothing wrong with ZFS on Linux

zfsonlinux dot org suggested that one can get around this "license" issue by using zfs-fuse, but concerning *technical issues* incorporating zfs into fuse (F-ilesystem in USE-rspace) in userspace means less performance and more overhead.

Not a problem, just annoying if you move a lot of data and you don't have super powerful hw.

So, if you dare, there are out-of-tree Linux kernel modules available from zfsonlinux, and you can built-in it directly (just, it still has a regression test suite, so be careful)

[magic]

I tend to use XFS because the fixed number of inodes in ext4 bothers me, even if I have never had problems because of it.
Works so far, fingers crossed.

That being said, I'm not 100% sure how well those "clever" filesystems would tolerate data loss, for example the root of the inode tree or something like that. With ext4 at least the structure is dead simple (inodes are held in arrays in well known positions).

XFS sounds also nice, but it is apparently not optimized at all when you have lots of small files.

What does "optimized" mean and which filesystems are "optimized"?

Regarding small files, I know of two aids:
1. storing file data inside the inode structure itself, which may work for files no larger than tens of bytes; I think ext4 can do it
2. "tail packing", i.e. storing the final incomplete block of the file together with other such "tails" of other files - unique to ReiserFS

Somewhat related:
3. with ext4 you are forced to decide the maximum number of files at mkfs time.

[Monkeh]

(just, it still has a regression test suite, so be careful)

So at the point authors stop bothering checking for regressions, they become impossible?

ZFS works fine. No need for FUSE.

[RoGeorge]

there are out-of-tree Linux kernel modules available from zfsonlinux, and you can built-in it directly (just, it still has a regression test suite, so be careful)

The one on Ubuntu is OpenZFS, I think (not sure).  There's no need to compile it, can be installed with sudo apt install zfsutils-linux.  No idea where it runs, but it's certainly not slow.  Seems faster and less demanding (for example at high speed torrents over 1GBps FO), but I might be biased, didn't benchmark it.

On what hardware do you use ZFS?  Does ZFS runs slower for your machine, or you were considering the theoretical  aspects?

[DiTBho]

@RoGeorge
Talking about GNU/Linux in general, there are two possibilities:

• ZFS-FUSE, ZFS in userspace, was the first attempt to workaround the license issue
• ZFS-kmod, ZFS integrated into kernel space, recent solution, still with regression test-suites and several debug flags, which smells of "still experimental"

Both of those solutions were/are too slow on my HPPA C3750, and even slower on my PowerMac-G4, which, to make things worse, can only address 2GByte of ram.

ZFS performs largely better on my boss's Tyan POWER9 machine with 32 PPC64 cores @ 3Ghz and 128GByte of ram, but to make life easier, I set up a FreeBSD-based ZFS-NAS for her on cheaper x86 hardware, and ZFS is just the "RAID second layer"; so from her end-user point of view is just an NFS black box unit attached to a 10Gps connection that weekly issues automatic backups.

Internally the big PPC64 iron uses XFS-v4.

[Monkeh]

Both of those solutions were/are too slow on my HPPA C3750, and even slower on my PowerMac-G4, which, to make things worse, can only address 2GByte of ram.

20 year old systems are slow. Who knew.

[Whales]

Every few years I look at alternatives, but I always seem to come back to ext4 again.

One of the most important features of a filesystem is the "love" of its developers.  If a filesystem is developed as a job: they couldn't GAF if you lose data (unless you're paying customer).  If a filesystem is a labour of love: the devs actually care if something happens to you and your data and might even put warnings/errata into the formatting tools themselves (eg "don't use this feature it's unstable") rather than hiding them elsewhere in the docs.


ext4:

 - I found default fixed inode count to not be enough on small size USB drives for a specific workload (gentoo install?).  Reformat required to change
 - My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there and surprisingly easy (lucky?) to fully recover after I found its start and finish address.

XFS:
 - every time I look at benchmarks it seems slower than ext4 for my use cases

btrfs:

 - Cool fun feature: transparent file compression (was useful on my old 16GB fixed disk chromebook)
 - Lost lots of data once, probably 10 years back.  Not sure if things are better since then?
 - I don't get a feeling that the dev community for this one cares too much.  Vague impression that they're doing it as a paid job to tick boxes, not for the love of their data and users.  Perhaps the business case for this one is to make an alternative to ZFS so that licensing for ZFS is cheaper/easier?  Actual reliability doesn't matter.

ext2:

 - always shutdown cleanly!  Otherwise much drama.  Only ever seen on a historical environment I once virtualised + then killed the VM rather than waiting for shutdown.

exfat:

 - default block size is very large for TB disks.  Made a large collection of small wav files I needed for an experiment take up 400GB (on-disk) instead of 80GB (real)

Bcachefs will be interesting to see.  It promises some of the features of btrfs, but possibly made with more love?  I'll still tread cautiously, backups are great but still don't avoid (1) annoying time and effort spent recovering, (2) data lost between backups intervals and (3) unnoticed data loss/rot (not recoverable unless you keep all backups forever).

[DiTBho]

ext4: My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there nd surprisingly easy (lucky?) to fully recover after I found its start and finish address.

makes no bonus specifically for ext4, if you know where a partition starts/ends, you can easily recover the partition table

p.s.
daily case, ext4 on my MIPS32@400Mhz router is 2x slower than xfs-v4

[DiTBho]

ext2: always shutdown cleanly!  Otherwise much drama

shutdown -> /sbin/init -> (/etc/init.d/disk,stop): { close all apps with open files; sync; remount,ro; }

'sync' is enough

[paulca]

Right now my bug bear is InfluxDB.  I'm hitting it with over 100 metrics a second at the moment and while it's a memory mapped DB the hard-disk is constantly pinging and "io-top" shows about 10% CPU from the ext4 journald, sometimes higher.

This is onto an SSD.  I have to look into it.  It might not all be "write".  If the DB does not all fit in memory it will churn around if reads and writes don't exist in memory.

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources. 

I wasn't ready to go full server supervisor OS like unRaid or what not, mostly because unraid is payware and the alternatives I wasn't sure on.

A HUGE amount of guides and how-tos for ZFS and other multi-disk aggregation filesystems seem to suggest you are starting with a pile of brand new disks.  It's so lame and a complete cop out that nobody wants to cover the 80% of the work .... migration.  They make it sound easy, but only cover the 20% of default, empty, brand new hardware installation.  If you are starting with 2Tb spread across 6 disks and you want to migrate it....  that's not so easy.

[DiTBho]

memory mapped DB the hard-disk is constantly pinging

physical ramdisks like Hyperdrive, iRAM, and volatile storage on physical iSCSI connections have the same problem, and ext2 compiled with --minimal is largely better for them.

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources. 

Yup, just consider that zfs-kmod v2.* can be configured with

+ minimal
+ nls
no pam, no libressl ---> do you really need to crypt? save from loading encryption stuff for home datasets
no rootfs ---> do you really need to keep the whole system on a "ZFS" partition(2)?
no split-usr ---> are you really so paranoid that you want this(1)?
no custom-cflags
no debug ---> save memory and CPU cycles, unless you think ZFS is buggy, and in this case ... do you trust it?
no python ---> do you really need to add optional support/bindings for Python? save memory and CPU cycles!
no static-libs ---> save memory
no test-suite ---> do you trust ZFS? how much? enough? so you don't need this

it helps a bit 

(1) when enabled, it spends precious CPU cycles maintaining { /bin, /lib*, /sbin, /usr/sbin } separately from { /usr/bin, /usr/lib* }

(2)
partition1: /boot, on "ext2,ro" { /boot/kernel, /boot/kernel_failsafe }
partition2: system on "ext2,ro"  { /bin, /lib*, /sbin, /usr/sbin, /usr/bin, /usr/lib* }
partition3: /user/$home on ZFS { /user/root, /user/toor, /user/admin, /user/you, ... }
partition4: /work (shared among users) on ZFS

sometimes you need to remount "partition2,rw" to update/install new packages.

[Monkeh]

btrfs:

 - Cool fun feature: transparent file compression (was useful on my old 16GB fixed disk chromebook)
 - Lost lots of data once, probably 10 years back.  Not sure if things are better since then?
 - I don't get a feeling that the dev community for this one cares too much.  Vague impression that they're doing it as a paid job to tick boxes, not for the love of their data and users.  Perhaps the business case for this one is to make an alternative to ZFS so that licensing for ZFS is cheaper/easier?  Actual reliability doesn't matter.

btrfs is pretty slow, still known to have major issues in striped setups, and is incredibly fragile in anything except a mirror. Any corruption results in a completely broken filesystem for which no functional recovery tools exist. Dead end so far.

A HUGE amount of guides and how-tos for ZFS and other multi-disk aggregation filesystems seem to suggest you are starting with a pile of brand new disks.  It's so lame and a complete cop out that nobody wants to cover the 80% of the work .... migration.  They make it sound easy, but only cover the 20% of default, empty, brand new hardware installation.  If you are starting with 2Tb spread across 6 disks and you want to migrate it....  that's not so easy.

Why would anyone be overly concerned with providing a migration path between two totally different filesystems on the same medium in an inherently destructive process?

I considered ZFS, but, I want to run the server on the absolute minimum of hardware I can and consume the minimum of watts doing it.  Most of the reviews which mentioned negatives mentioned it was heavy on resources.

If you're not doing anything, it's not using anything. If you are, it's still a tiny fraction of the mechanical power of the drives. Yes, it uses relatively more resources than other filesystems, because it's doing more - not much else out there is doing continuous checksumming to make sure your precious data actually exists. If you don't care about the data, you may as well just use ext2 on a single drive!

For a general purpose filesystem, use ext4. If you have plans on keeping data, use ZFS in a suitably redundant configuration.

[Marco]

Keeping two machines might be a burden, and with NFS most of the ZFS goodies will be lost, for example sending snapshots between machines.

It's a pity no major distribution ever threw some support at Wyng, which does the same thing for LVM2 (not a filesystem, just a block layer). ZFS and Linux will always bite each other, but LVM2+XFS can provide pretty much the same thing ... it just needs to be packaged up in a more user friendly way.

Perhaps if Chromebooks start leaning more heavily into running Linux applications, Google will do it. They never needed it for ChromeOS in the past, because the user isn't supposed to have much more local than a download directory, with everything else in the cloud.

[SiliconWizard]

ZFS is good but yes, it's resource-intensive. But the fact it's not officially supported by Linux is a major issue for me. Whether one cares about the licensing or not, it's just NOT supported by the Linux kernel, which means that there is no guarantee of any working state between a given version of (Open)ZFS and the kernel. That's a no-no for me. I want to be able to freely update my kernels, and not have this restriction, and have to use specific kernels just for this.

btrfs looks good on paper, but it's a half-baked implementation this far as I got it.

I'm probably going to test XFS on a partition to see how it goes.

As to ZFS, the plan is rather to use it with FreeBSD for my next NAS. My current one was built years ago and runs a RAID array on CentOS, it does the job just fine, never lost any data.

[John B]

BTRFS is the default filesystem for fedora. There must have been some reason?

[DiTBho]

be able to freely update my kernels

can be compiled as kernel module, META gives you min/max supported kernel versions

Code: [Select]

Meta:          1
Name:          zfs
Branch:        1.0
Version:       2.1.11
Release:      1
Release-Tags:  relext
License:       CDDL
Author:        OpenZFS
Linux-Maximum: 6.2
Linux-Minimum: 3.10
(zfs-2.1.11.tar.gz)

when you update the kernel, you get headers and kernel sources in /usr/src/linux, and can recompile openZFS.

[Nominal Animal]

I do tend to use ext4 as my filesystem, but on non-virtual instances, I use it on top of LVM2 for ease of snapshotting and miscellaneous shenanigans.

Similar to how massively widespread use of JS in web stuff has made JS interpreters extremely efficient, widespread use of ext4 in Linux has made it technologically robust and reliable.  My logic: if ext4 + LVM2 suffices, use it; otherwise look into the alternatives.

[Monkeh]

BTRFS is the default filesystem for fedora. There must have been some reason?

Yes, free testing.

[SiliconWizard]

be able to freely update my kernels

can be compiled as kernel module, META gives you min/max supported kernel versions

Code: [Select]

Meta:          1
Name:          zfs
Branch:        1.0
Version:       2.1.11
Release:      1
Release-Tags:  relext
License:       CDDL
Author:        OpenZFS
Linux-Maximum: 6.2
Linux-Minimum: 3.10
(zfs-2.1.11.tar.gz)

when you update the kernel, you get headers and kernel sources in /usr/src/linux, and can recompile openZFS.

Sure, but I don't really want to go through this hassle in this instance. Unless there was an absolutely gigantic benefit.

As I pointed out, one key issue is that the kernel may break ZFS at any point in time, Linus was clear about it, and that makes sense as they can't embed it anyway, so there's no reason to go extra miles to support it as a module. For my use cases, that's not going to be an option. But certainly in cases where you would settle for, say, a given LTS line of kernel with which OpenZFS is fine and tested, why not.

[ejeffrey]

For a boot drive I don't see enough advantage for anything to overcome the stability, support, and universality of ext4.  It's not the best by a long shot but it is good enough l.

For a multi drive array you definitely can do md+lvm+ext4 but I would seriously consider zfs even if there is a bit more hassle because it's not in the mainline kernel.

I have used xfs in the past, especially in the ext3 days.  it's fine, just not really enough of an advantage over ext4 to be worth it to me. 

[magic]

I'm probably going to test XFS on a partition to see how it goes.

XFS is a solid filesystem that just works. It's widely used on large storage servers because it's said to perform faster than ext4, particularly with multiple concurrent accesses. I personally knew people who ditched ext4 for XFS on their servers a few years ago and never looked back. It allocates inode blocks dynamically as new inodes are created, so no stupid problems here. I don't think it does anything fancy with small files, they just get a 4KB block allocated for data and that's it.

The latest XFS v5 (not readable by some old kernels) supports "reflinks", i.e files sharing their data blocks with "copy on write". There is no support (and AFAIK will never be in v5) for true snapshotting, with preservation of inode numbers and everything.

There is probably not much difference on a desktop system. If you start another thread asking which text editor to use on Linux, I will officially declare you a troll

[Marco]

There is no real need for filesystem snapshotting on Linux, thin provisioned LVM does it ... not as conveniently as ZFS though.

[DiTBho]

I have even designed my own toy filesystem (standalone or on-FUSE), which has zero repairing tools and it's far from being fast when it also needs to be robust and reliable. It just works.

This brings us to the question: with which criteria do you entrust your precious data to such a delicate component?

[magic]

zero repairing tools
it works

Pick one.
It will only work until you hit a filesystem or block layer bug, or hardware data loss.

I seem to recall that btrfs also started out with no fsck or repair tools. That's how I knew to give it a pass for now.

[mapleLC]

I'm using ZFS with Proxmox, it looks very stable. 

I find it hard to use for 2 reasons:

The Linux file system is rather confusing.  Etc, bin, dev, sys, srv - it all makes for a lot of necessary domain knowledge to be effective knowing where file locations are

This is kinda Proxmox specific but ZFS is confusing, shares are confusing, and needing to use FTP regularly is annoying

[Monkeh]

I have never used FTP with Proxmox. Why would you need to?

[lyxmoo]

gridefs in mongodb is fast, especially for huge mount of small files (such as fsize < 10kB)

[SiliconWizard]

GridFS is not really a filesystem per se as far as I understand, but is on top of a conventional filesystem: https://www.mongodb.com/docs/manual/core/gridfs/
Unless I missed something.

[SiliconWizard]

One "downside" to XFS is that, while it has a 'discard' option, it apparently hinders performance quite a bit, so if on a SSD, they recommend disabling this option and issuing fstrim separately.
ext4 handles this much better.

[SiliconWizard]

ext4:
 - I found default fixed inode count to not be enough on small size USB drives for a specific workload (gentoo install?).  Reformat required to change
 - My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there and surprisingly easy (lucky?) to fully recover after I found its start and finish address.

The fixed total number of inodes with ext4 looks problematic, but it really is only in pretty particular situations, like indeed with very small partitions and a large number of small files.
Otherwise, I've never had a problem with that.

(Probably obvious for many but just in case: you can have a quick look at your current inode utilization with 'df -i' )

I have 3 different machines running Linux, one is a headless box (for running Linux tools), one is a NAS and the 3rd is a latptop. All have ext4 partitions only, the NAS with LVM (and its boot partition is ext2 for historical reasons, but that's beyond the point.)

inode utilization on each of them never exceeds 16%, even on the NAS where there's a ton of files, and all have been in use for several years.

The default at creation is 1 inode per 16KB of space.

And yes, ext4 is overall pretty resilient and easy to recover.

[DiTBho]

ext4 is [..] easy to recover.

it depends on what you need to recover from.

[DiTBho]

• recover deleted by accident files, with their original file names
• recover files from a damaged filesystem: how logical damages impact the filesystem, how much it degrades, and to what extent
• recover files from a damaged hard disk: how physical damages impact the filesystem, how much it degrades, and to what extent

[DiTBho]

zero repairing tools
it works

Pick one.
It will only work until you hit a filesystem or block layer bug, or hardware data loss.

I seem to recall that btrfs also started out with no fsck or repair tools. That's how I knew to give it a pass for now.

I meant making a testbench to test how a filesystem reacts.

something like ummm, you continuously write files (e.g. scp from a server) of which you know the hash (md5? sha-something), and randomly power down the SBC, then you check for logical damages, how they propagated, if files can be restored (if they), and compare hashes to be sure the content is exact.

[RoGeorge]

ZFS is good but yes, it's resource-intensive. But the fact it's not officially supported by Linux is a major issue for me. Whether one cares about the licensing or not, it's just NOT supported by the Linux kernel, which means that there is no guarantee of any working state between a given version of (Open)ZFS and the kernel. That's a no-no for me. I want to be able to freely update my kernels, and not have this restriction, and have to use specific kernels just for this.

How do you know it's resource intensive if you didn't use it? 

There is no such things as "supported" by Linux.  "Supported by" is about proprietary and closed source OSs.  The Linux kernel has a set of well defined mechanisms to interface with.  Kernel interfacing mechanisms do not change over night, and do not get deprecated/discontinued without warning the users and devs with many years in advance.

There is nothing else you need to do about ZFS when the kernel updates.  You type once "sudo apt install zfsutils-linux" (or whatever install would be for the Linux flavor you use), and that's it.  After that, you can upgrade the kernel or the rest of the OS any time you want, as if you never installed the OpenZFS.  ZFS doesn't need any special attention at a Linux update.

Not trying to convince you to install ZFS, only pointing to misbeliefs.

Aside from the FS question, it's curious to me how nitpicking arguments returned by a search engine can weight more than first-hand testimonies in this very topic, from people saying they used ZFS for years.

Whatever FS you'll choose, keep backups for important/personal data.  A good FS/OS is not enough when it comes to theft, fire, malware, etc.

Good luck testing FSs, and pick the one(s) you like most! 

[DiTBho]

p.s.
if you go for ext*, this is useful for the first-class of the mentioned test-bench

Code: [Select]

sys-fs/extundelete
      Description:   A utility to undelete files from an ext3 or ext4 partition
      License:       GPL-2


[mapleLC]

What about using Ciph as an alternative?

Edit, typo, Ceph

[DiTBho]

just added it to my local overlay, for xfs try this

Code: [Select]

overlay::sys-fs/xfsundelete
      Description:   A utility to undelete files from an xfs partition
      License:       not yet checked


[DiTBho]

What about using Ciph as an alternative?

what is Ciph? URL? any doc?

[MK14]

What about using Ciph as an alternative?

what is Ciph? URL? any doc?

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

[DiTBho]

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

thanks! This is very interesting 

[ejeffrey]

ext4:
 - I found default fixed inode count to not be enough on small size USB drives for a specific workload (gentoo install?).  Reformat required to change
 - My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there and surprisingly easy (lucky?) to fully recover after I found its start and finish address.

The fixed total number of inodes with ext4 looks problematic, but it really is only in pretty particular situations, like indeed with very small partitions and a large number of small files.
Otherwise, I've never had a problem with that.

Yeah, realistically storage is cheap enough now, even on SSDs that on most systems it's usually fine to just create way more inodes than you will ever use.  It's a bit of a awkward limitation and it's annoying to have to even think about it, but it's not a problem I have seen in the real world in ages.  It might come up on very small drives (especially those using 1k blocks instead of the default 4k), but for something like the OS drive on a typical desktop or something it's not an issue. 

[mapleLC]

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

thanks! This is very interesting 

That was a typo, yes Ceph.

I've had it on my to do list but have not gotten around to installing it.  It requires 3 nodes, so many create 2 virtual nodes and a real one to get a cluster.  Defeats the purpose but you grow into it being the idea and then create a real node at a time as you expand.

[SiliconWizard]

I have even designed my own toy filesystem (standalone or on-FUSE), which has zero repairing tools and it's far from being fast when it also needs to be robust and reliable. It just works.

Can you describe its architecture?

[paulca]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

[mapleLC]

I have even designed my own toy filesystem (standalone or on-FUSE), which has zero repairing tools and it's far from being fast when it also needs to be robust and reliable. It just works.

Can you describe its architecture?

unfortunately with big kernel locks so it doesn't scale very well, but for simple tasks like being the filesystem of an MP3 player or the filesystem of a PDF reader with the possibility to take notes, it works well.

It has no journaling and no defense mechanisms. Due to the mechanism that keeps in ram only the last and most used blocks, a single crash can seriously damage the structure of the disk, so it cannot be trusted for serious storage, even because there are no repair tools!

Sounds like you were the dev behind the file system of Windows 8.

[DiTBho]

I don't have any interest in Microsoft stuff, I just Googled to understand why someone quoted Microsoft referring to my toy-fs and found there was a filesystem *somehow using a binary tree* I have never heard about.
 - - -
inferring b*tree, I found that there is a new filesystem, namely "ReFS", which *somehow using a binary tree* uses B+trees for all on-disk structures, including all metadata and file data(1), so I learned something from your comment: NTFS++=ReFS.


edit:
(1)the Wikipedia needs a fix

[SiliconWizard]

inferring b*tree, I found that there is a new filesystem namely "ReFS", which uses B+trees for all on-disk structures, including all metadata and file data, so I learned something from your comment: NTFS++=ReFS.

ReFS is MS's new filesystem. It's not that bad actually. Heck, even NTFS is not that bad, but ReFS certainly takes more modern approaches.

[SiliconWizard]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

[mapleLC]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

[Zucca]

The more I research, the more I read, the more it looks like ZFS is for servers and for the clients just pick something you like.

It is a difficult task to suggest a file system because a lot of little things needs to be considered.
That said, I will use soon arch linux with ext4. I decided to skip LMV for now... if I will ever missing something I will give LMV a try.

Interesting reading as always, thanks everybody.

[DiTBho]

so, just bought a third lot of qty=4 sATA disks and a hotplug disk case. It looks perfect for experimenting with bcachefs, a new filesystem that somehow looks similar(1) to like zfs and btrfs.

bcachefs/ Stability: October 2022 talk for Redhat, bcachefs is stable, and will not eat your data.
The following tests and testimonials should help in proving this.
Testimonials: None yet

well, trust is required 


(1) Copy On Write filesystem

[paulca]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

All I'm looking for is more than a 24 hour window to recover mistakes.  Like when you only realise the mistake after the backups have run, or you don't even realise the thing you changed/deleted wasn't the thing you meant to delete, but it's a week later.

I did start to implement a grandfathering incremental backup system using rsync incremental.  The ideal was that in addition to the normal bulk rsync to RAID backups I would additional add the development folder as an incremental "store change only" backup.

I didn't finish it as I ran into a complication when I proposed the question, "How long does it "increment" for and should it not have fixed full copy replay points to stop it growing forever, just like SQL Server's transaction log files.   Then it got more complicated with feature creep to where I wanted more than one layer.  One that was incremental "hourly", but took a complete backup nightly, erasing the incrementals one day behind it.  Then another layer doing the same but weekly and monthly.

I figure I should just axe those later requirements and see if I can't get it to work.  Then I can create "Checkout for date" shell scripts to pull and replay to any day/hour in time.

Potentially the incremental rsyncs could be triggered in NRT from filesystem watchers looking for changes.  As soon as you save the file an incremental rsync runs (with appropriate rate throttling).

[magic]

inferring b*tree, I found that there is a new filesystem namely "ReFS", which uses B+trees for all on-disk structures, including all metadata and file data, so I learned something from your comment: NTFS++=ReFS.

Yeah, you learned something from Wikipedia

Their own reference says:

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums).

IOW, it's basically like every other filesystem, besides dinosaurs like FAT, EXT, FFS or NTFS.

AFAIK the only filesystem which actually stores file data in a tree structure is ReiserFS. IIRC it uses one massive tree for virtually all metadata and it also packs small files in there if they aren't worth allocating a whole block.

[Monkeh]

I did start to implement a grandfathering incremental backup system using rsync incremental.  The ideal was that in addition to the normal bulk rsync to RAID backups I would additional add the development folder as an incremental "store change only" backup.

There are already tools to do this, what's wrong with the wheel?

[SiliconWizard]

so, just bought a third lot of qty=4 sATA disks and a hotplug disk case. It looks perfect for experimenting with bcachefs, a new filesystem that somehow looks similar(1) to like zfs and btrfs.

bcachefs/ Stability: October 2022 talk for Redhat, bcachefs is stable, and will not eat your data.
The following tests and testimonials should help in proving this.
Testimonials: None yet

well, trust is required 


(1) Copy On Write filesystem

Yeah, also got curious about bcachefs, but the current state seems not too stable. If you're worried about btrfs not being stable, bcachefs is probably not your best bet.
Experimenting won't harm though.

[SiliconWizard]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

All I'm looking for is more than a 24 hour window to recover mistakes.  Like when you only realise the mistake after the backups have run, or you don't even realise the thing you changed/deleted wasn't the thing you meant to delete, but it's a week later.

As I said, isn't any FS with snapshots exactly what you need then? So that would be either btrfs or ZFS I think.

But otherwise you can absolutely do the same with simple tools on any filesystem, just automate incremental backups every 24h, and that's it.

[DiTBho]

bcachefs:
- it can be integrated into kernel 6.0.*, built as "built-in" as well as "kmod" it and actually works
- utilities { mkfs.bcachefs, fsck.bcachefs, .. } need dev-lang/rust to be compiled

doesn't seem too bad, but we will see!

[Ed.Kloonk]

bcachefs:
- it can be integrated into kernel 6.0.*, built as "built-in" as well as "kmod" it and actually works
- utilities { mkfs.bcachefs, fsck.bcachefs, .. } need dev-lang/rust to be compiled

doesn't seem too bad, but we will see!

Wouldn't you be better off with ZFS if you have to go to that much trouble?

[DiTBho]

Wouldn't you be better off with ZFS if you have to go to that much trouble?

just looking around, for production, I will likely still use xfs-v4 with kernel-softraid support, but it's not bad to experiment with new experimental things for contributing.

[Marco]

bcachefs

I dislike how alien it is to Linux. There's a certain Unix like elegance to DM+LVM+whateverFS stacks which get lost ... for what exactly?

OS level block checksumming. CoW snapshotting. A SSD cache for rotating storage. Drive combining with or without redundancy. DM can do all that and it fits fine  in a block layer. What's left? A better solution for the write hole for RAID 4/5/6? (DM-Raid does have a solution by the way, it can use a journal.) Who cares? Only RAID 1/10 matter, everything else was a giant mistake. Portability? Meh.

[magic]

It's flexibility. You can have two or three copies of each data block spread over five disks. If a disk fails or you run out of space you just plug another one or two or three, not even necessarily of the same capacity as the old, and life keeps going on and everything is still on the same filesystem. If there is enough free space you can free up one disk and retire it from the array.

[Marco]

Nice to have, but I don't see how that couldn't have been implemented in LVM/DM. Make one volume group to rule them all. Make some thin provisioned logical volumes with desired levels of striping/mirroring and let the DM layer take care of it, put filesystems on all logical volumes and just link files according to the desired level.

Then it could have benefitted all filesystems.

PS. because of the way storage could run out for striping/mirroring with variable sized drives, you could run into limitations long before the array is full. Having to fear the edge case "failure" would prevent me from enjoying the initial convenience.

[magic]

Another use case for btrfs/ZFS: USB disk enclosures.

I have this weird USB enclosure which seems OK by itself, and an HDD which is OK too, but putting it in this enclosure causes large amount of random bit flips.
Don't ask me why, and I found that it only happens on reading - I can write anything and it reads perfectly outside this cursed enclosure

This damn thing just corrupted my ext4 filesystem because it read free space bitmaps wrong and reused blocks which weren't actually free.

BTW, this is not the first time I'm having data corruption with USB/SATA bridges. IME these chips are shit until proven otherwise with thorough testing. (Frankly, even then they may still be shit, just a matter of testing more).

[magic]

Regarding layered alternatives, with conventional filesystems on top of DM/MD/LVM, here's a few thoughts:

I'm not a sysadmin and I'm not very familiar with those things, to begin with. I think I have only ever used RAID0/1 and dm-snapshot, and I stopped needing the latter when XFS v5 gained reflink capability (I used it for temporary modifications to disk images). So for me it's a question of learning a bunch of new things, or a single filesystem-specific tool or even standard Linux features (like said reflinks).

I find dm-snapshot less flexible than COW disk images on XFS, as it needs a fixed size block device provided to store those snapshots. I can give it a loop device backed by a sparse file on XFS, or a thin provisioned logical volume, but WTF. And this gets me to the second problem: you can put together any configuration you want out of those components, but you have to build it in advance, layer upon layer of virtual block devices. If you change your mind about something later, you are screwed.

It also is simply much more PITA to setup all those things, maintain the configuration, move it from machine to machine, deal with occasional initramfs breakage which may render the system unbootable and any other stupid problems.

If you use a filesystem with built-in storage management, you don't need to plan anything in advance. Start with one disk in USB enclosure. Add another disk and move both into a NAS. Change RAID level when desired.

[Marco]

That's a question of tooling. You can change the level of mirroring of a logical volume, wyng-backup does ZFS send equivalent for LVM, etc.

I'd rather see effort go to making that all convenient and transparent than have yet another filesystem.

[DiTBho]

XFS v5 gained reflink capability

yeah, that's the main reason to move from xfs-v4 to v5
cp --reflink: supported 

you can put together any configuration you want out of those components, but you have to build it in advance, layer upon layer of virtual block devices.

yup, exactly my criteria

p.s.
and it's also the point I agree with this article: with ZFS there is a "Hidden Cost" of using for Home NAS if you don't correctly plan in advance the size (therefore the purchase) of your disk devices thinking you can always upgrade them in the future without paying an extra cost.

With btrfs, it's not a problem.

[DiTBho]

(
Basically, x->y: (x) operates at (y) level

• symlinks -> filename-path
• hardlinks -> inode
• reflinks -> block(via copy-on-write)

)

[magic]

and it's also the point I agree with this article: with ZFS there is a "Hidden Cost" of using for Home NAS if you don't correctly plan in advance the size (therefore the purchase) of your disk devices thinking you can always upgrade them in the future without paying an extra cost.

With btrfs, it's not a problem.

That's rather sad.

With btrfs it's not a problem because AFAIK its RAID5 implementation is still broken and not recommended for regular use

[magic]

That's a question of tooling. You can change the level of mirroring of a logical volume, wyng-backup does ZFS send equivalent for LVM, etc.

I'd rather see effort go to making that all convenient and transparent than have yet another filesystem.

If you want to add a feature you haven't planned for in advance - snapshots, encryption, compression, whatever - then you need to insert a new component into the stack, which is usually not as quick and easy as setting some attribute on a file or directory.

[Marco]

If you want to add a feature you haven't planned for in advance - snapshots, encryption, compression, whatever - then you need to insert a new component into the stack, which is usually not as quick and easy as setting some attribute on a file or directory.

Easier than putting it in a filesystem where it wasn't planned in advance.

[Marco]

With btrfs it's not a problem because AFAIK its RAID5 implementation is still broken and not recommended for regular use

RAID5 is not recommended for any use.

[DiTBho]

RAID5 is "decent" with 4 disks
RAID6 is "better"

[SiliconWizard]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

[JeremyC]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

[SiliconWizard]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

I don't disagree with this - sure RAID5 is not ideal and takes a good while to rebuild when adding or changing drives. But it's not as bad as some say. At least for home or small business use.
Now certainly cost was an issue when deciding.
As I said earlier, my next NAS may very well be with FreeBSD and ZFS. We'll see!

[JeremyC]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

I don't disagree with this - sure RAID5 is not ideal and takes a good while to rebuild when adding or changing drives. But it's not as bad as some say. At least for home or small business use.
Now certainly cost was an issue when deciding.
As I said earlier, my next NAS may very well be with FreeBSD and ZFS. We'll see!

Good choice. If you didn’t already, check the TrueNAS (previously FreeNAS).
I’m using ZFS since ~2007 on Solaris 10. I’m glad that SM released to open source before Oracle got hands on. From my experience ZFS does not perform best on Linux, but does much better in BSD.
If you decide go with ZFS make sure that you give a lot of space for ARC on SSD, it’s the key.

[Ed.Kloonk]

From my experience ZFS does not perform best on Linux, but does much better in BSD.

I'll bite. Why?

[DiTBho]

I don't disagree with this - sure RAID5 is not ideal

one can also argue that double parity is better with big disks.
that's why I said RAID6 is "better"

[Marco]

At least for home or small business use.

For a hobby time is free, for a small business storage hasn't cost enough to do anything more than mirroring in decades.

[JeremyC]

From my experience ZFS does not perform best on Linux, but does much better in BSD.

I'll bite. Why?

Back in ~2015 I was resourcing alternative and redundant storage solution.
On the same hardware, ZFS on Linux couldn’t handle ~10M iops, when at the same hardware BSD (FreeNAS) didn’t have any problems. Solaris 11 performed best, but it requires license in production…
We used 45 SAS (WD Gold) in RAID1+0 (each mirror had 4 drives for redundancy) drives and 8 x Samsung SSD for ARC.
Solaris 11 was the kink, 2nd and acceptable was FreeNAS (BSD) and ZFS on Linux was intermittently choking...

[magic]

For a hobby time is free, for a small business storage hasn't cost enough to do anything more than mirroring in decades.

Maybe for OS boot disks and small servers.

In high capacity storage arrays the cost of disks is still the deciding factor and parity is very much used, although it is typically doubly redundant parity these days.

Hardware RAID5/6 controllers with battery backed cache and all that jazz are also still being made, sold and bought.

[paulca]

I just switched roles in work from a project with an HDFS filesystem consisting of several dozen petabytes over 5000 nodes.  Thousands of cores and terrabytes of online RAM.

Mostly the disks where "node local" SAS arrays or "rack local" SAS arrays.  Due to it being BigData/Compute cluster, the disks stay with the cores.

That wasn't even the scary part of the architecture.  It was the "near real time" SQL caching layer which got used by end business customers.

Somehow it managed to provide, almost, normal SQL query latency < 1 minute from that HDFS cluster.

I believe it runs on a sub cluster with 1000s of terrabytes of RAM.  Whatever it is, it replaced an IBM Netezza data warehose on IBM Power Servers so I expect it's a beast.  Direct "Hadoop" based queries to the same distributed on disk data would take orders of magnitude longer to query.

[DiTBho]

On the same hardware, ZFS on Linux couldn’t handle ~10M iop

which one? zfs-kmod or zfs-fuse? 

if it was the fuse one, well ... userspace <- from/to -> kernelspace adds some overhead

[DiTBho]

SAS arrays or "rack local" SAS arrays.

got it from the last dumpster dive:
      SAS, 2 channels(1) ------ fiber optic bridge ------ fiber optic HBA

is it like this?

(1)
ch0: { disk0,disk1,disk2,disk4 } on one cable
ch1: { disk0,disk1,disk2,disk4 } on one cable


... why not Infiniband? 

[ve7xen]

I am using btrfs for more or less all my personal stuff for 5ish years. I have had no major issues with it, though obviously it is a small sample size.

On desktops, doing frequent more or less 'free' snapshots both periodically and before system updates has been very valuable, and this is not something that is well supported with 'traditional' filesystems. ZFS support on Linux has always felt quite half-baked to me, and most distros have good support for btrfs built in these days, from bootloader through to snapshotting tools and hooks. I am not too interested in using something 'bolt-on', even if it works well.

On my personal servers, I find piping incremental snapshots around to be an elegant solution to the incremental backup issue. For my personal long-term archives I also insist on a filesystem with data (and metadata) checksumming and scrubbing, which again means choosing between two half-baked options, but btrfs feels by far the more Linux-native.

At $dayjob we are using ext4 for general workloads and XFS for dedicated storage workloads. I have in the long-distant past experienced an XFS data loss incident, so haven't used it on my personal stuff in a decade or more.

One "downside" to XFS is that, while it has a 'discard' option, it apparently hinders performance quite a bit, so if on a SSD, they recommend disabling this option and issuing fstrim separately.
ext4 handles this much better.

This is generally the best-practice advice these days anyway, so I wouldn't really count it against XFS.

[SiliconWizard]

This is generally the best-practice advice these days anyway, so I wouldn't really count it against XFS.

You're right. That's how I set up my workstation lately in the end, even with ext4.

[Marco]

In high capacity storage arrays

Why does a small business need a high capacity storage array for its Excel sheet? That's going to be the reality for the vast majority, they just need storage of email and some bookkeeping cobbled together or some crap off the shelf, not vast amounts of media or measurement data. Even the off the shelf crap can't use up that much storage.

Sure large companies can generate enough plain business data to justify some more effort on the storage, but a small business 99% of the time won't have enough data to mess around with something with finnicky failure modes.

[Nominal Animal]

In high capacity storage arrays

Why does a small business need a high capacity storage array for its Excel sheet? That's going to be the reality for the vast majority, they just need storage of email and some bookkeeping cobbled together or some crap off the shelf, not vast amounts of media or measurement data. Even the off the shelf crap can't use up that much storage.

Sure large companies can generate enough plain business data to justify some more effort on the storage, but a small business 99% of the time won't have enough data to mess around with something with finnicky failure modes.

Nightly backups for a couple of weeks, then weekly, and finally monthly, before stored on RO media.

The reason is often not so much because of hardware failure, but learning that humans do rather stupid things now and then, and recovering from those without angering your clients and losing business is definitely worth the few thousand euros/dollars it costs to set up.

And this is on top of "standard" off-site backups, assuming there is enough internet bandwidth to do so.

[magic]

Another use case for btrfs/ZFS: USB disk enclosures.

[...]
I'm having data corruption with USB/SATA bridges. IME these chips are shit until proven otherwise with thorough testing. (Frankly, even then they may still be shit, just a matter of testing more).

My first ever btrfs is up and running, on a slightly dubious USB disk enclosure.

Passes a simple corruption detection test: locate a file on disk with filefrag, overwrite it on the raw block device with dd.
Scrub finds the error, and no other errors so far.

Will give it a try to see if btrfs bugs manage to kill my data faster than USB bridge bugs normally would

As a bonus, my data are completely protected from Windows users without encryption.

[Marco]

Nightly backups for a couple of weeks, then weekly, and finally monthly, before stored on RO media.

Don't forget to have two sets of tapes with multiple storage sites, cause that's really what small businesses do

They'll more likely buy something like Acronis, with incremental backups to whatever NAS the salesman pushes and offsite backup to the cloud. The salesman should push raid 1/10, because storage is too cheap and rebuild SNAFUs too big a a headache to deal with for the tiny data amounts of ye average small business with incremental backup.

Until storage overhead starts being a significant part of your bottom line and you have the expertise to take RAID degradation into account, just use the simplest solution with the easiest to deal with failure modes.

[golden_labels]

Since ReiserFS has been mentioned: you may wish to not deploy it to new installations. ReiserFS is deprecated and planned for removal in 2025. LTS releases will retain it until at least 2026, but moving towards a deprecated technology is a poor choice.

[DiTBho]

(Youtube video, "The New File System in Windows: ReFS")

[paulca]

Slightly related to filesystems.

I was checking disk speeds on Windows and discovered that the basic SATA SSD runs at about 550MB/s.  However a single Gen4 NVMe drive runs at 5500MB/s.

So, as I had two of them, both on Gen4 4X bandwidth, I striped them into one volume.  11000MB/s read speed woot.

A small improvement.    Should make steam games load a bit faster.

In other news I built a new linux server and this time it's getting  RAID1 rootfs.  On AMD EFI you can have cloned boot contexts and all!

If it sounds excessive, do notice the sweat spot we are in right not for 500Gb and 1Tb Gen4 NVMes.  You can find them for £30 hunt.  And a dual riser card is also about £30.