File systems!

[magic]

Regarding layered alternatives, with conventional filesystems on top of DM/MD/LVM, here's a few thoughts:

I'm not a sysadmin and I'm not very familiar with those things, to begin with. I think I have only ever used RAID0/1 and dm-snapshot, and I stopped needing the latter when XFS v5 gained reflink capability (I used it for temporary modifications to disk images). So for me it's a question of learning a bunch of new things, or a single filesystem-specific tool or even standard Linux features (like said reflinks).

I find dm-snapshot less flexible than COW disk images on XFS, as it needs a fixed size block device provided to store those snapshots. I can give it a loop device backed by a sparse file on XFS, or a thin provisioned logical volume, but WTF. And this gets me to the second problem: you can put together any configuration you want out of those components, but you have to build it in advance, layer upon layer of virtual block devices. If you change your mind about something later, you are screwed.

It also is simply much more PITA to setup all those things, maintain the configuration, move it from machine to machine, deal with occasional initramfs breakage which may render the system unbootable and any other stupid problems.

If you use a filesystem with built-in storage management, you don't need to plan anything in advance. Start with one disk in USB enclosure. Add another disk and move both into a NAS. Change RAID level when desired.

[Marco]

That's a question of tooling. You can change the level of mirroring of a logical volume, wyng-backup does ZFS send equivalent for LVM, etc.

I'd rather see effort go to making that all convenient and transparent than have yet another filesystem.

[DiTBho]

XFS v5 gained reflink capability

yeah, that's the main reason to move from xfs-v4 to v5
cp --reflink: supported 

you can put together any configuration you want out of those components, but you have to build it in advance, layer upon layer of virtual block devices.

yup, exactly my criteria

p.s.
and it's also the point I agree with this article: with ZFS there is a "Hidden Cost" of using for Home NAS if you don't correctly plan in advance the size (therefore the purchase) of your disk devices thinking you can always upgrade them in the future without paying an extra cost.

With btrfs, it's not a problem.

[DiTBho]

(
Basically, x->y: (x) operates at (y) level

• symlinks -> filename-path
• hardlinks -> inode
• reflinks -> block(via copy-on-write)

)

[magic]

and it's also the point I agree with this article: with ZFS there is a "Hidden Cost" of using for Home NAS if you don't correctly plan in advance the size (therefore the purchase) of your disk devices thinking you can always upgrade them in the future without paying an extra cost.

With btrfs, it's not a problem.

That's rather sad.

With btrfs it's not a problem because AFAIK its RAID5 implementation is still broken and not recommended for regular use

[magic]

That's a question of tooling. You can change the level of mirroring of a logical volume, wyng-backup does ZFS send equivalent for LVM, etc.

I'd rather see effort go to making that all convenient and transparent than have yet another filesystem.

If you want to add a feature you haven't planned for in advance - snapshots, encryption, compression, whatever - then you need to insert a new component into the stack, which is usually not as quick and easy as setting some attribute on a file or directory.

[Marco]

If you want to add a feature you haven't planned for in advance - snapshots, encryption, compression, whatever - then you need to insert a new component into the stack, which is usually not as quick and easy as setting some attribute on a file or directory.

Easier than putting it in a filesystem where it wasn't planned in advance.

[Marco]

With btrfs it's not a problem because AFAIK its RAID5 implementation is still broken and not recommended for regular use

RAID5 is not recommended for any use.

[DiTBho]

RAID5 is "decent" with 4 disks
RAID6 is "better"

[SiliconWizard]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

[JeremyC]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

[SiliconWizard]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

I don't disagree with this - sure RAID5 is not ideal and takes a good while to rebuild when adding or changing drives. But it's not as bad as some say. At least for home or small business use.
Now certainly cost was an issue when deciding.
As I said earlier, my next NAS may very well be with FreeBSD and ZFS. We'll see!

[JeremyC]

My NAS uses RAID1 for the system partition and RAID5 for all user data. Never had any issue with that.
Yes RAID5 works decently with 4 disks.
It's cool to say that it's bad, though, so if you want to look cool.

Sure it does. But if you have 1 hot spare in RAID5 it may take a while before the spare it's usable... sometimes it require too long time. I would suggest RAID 6 (RAIDZ2 in the ZFS world).

I don't disagree with this - sure RAID5 is not ideal and takes a good while to rebuild when adding or changing drives. But it's not as bad as some say. At least for home or small business use.
Now certainly cost was an issue when deciding.
As I said earlier, my next NAS may very well be with FreeBSD and ZFS. We'll see!

Good choice. If you didn’t already, check the TrueNAS (previously FreeNAS).
I’m using ZFS since ~2007 on Solaris 10. I’m glad that SM released to open source before Oracle got hands on. From my experience ZFS does not perform best on Linux, but does much better in BSD.
If you decide go with ZFS make sure that you give a lot of space for ARC on SSD, it’s the key.

[Ed.Kloonk]

From my experience ZFS does not perform best on Linux, but does much better in BSD.

I'll bite. Why?

[DiTBho]

I don't disagree with this - sure RAID5 is not ideal

one can also argue that double parity is better with big disks.
that's why I said RAID6 is "better"

[Marco]

At least for home or small business use.

For a hobby time is free, for a small business storage hasn't cost enough to do anything more than mirroring in decades.

[JeremyC]

From my experience ZFS does not perform best on Linux, but does much better in BSD.

I'll bite. Why?

Back in ~2015 I was resourcing alternative and redundant storage solution.
On the same hardware, ZFS on Linux couldn’t handle ~10M iops, when at the same hardware BSD (FreeNAS) didn’t have any problems. Solaris 11 performed best, but it requires license in production…
We used 45 SAS (WD Gold) in RAID1+0 (each mirror had 4 drives for redundancy) drives and 8 x Samsung SSD for ARC.
Solaris 11 was the kink, 2nd and acceptable was FreeNAS (BSD) and ZFS on Linux was intermittently choking...

[magic]

For a hobby time is free, for a small business storage hasn't cost enough to do anything more than mirroring in decades.

Maybe for OS boot disks and small servers.

In high capacity storage arrays the cost of disks is still the deciding factor and parity is very much used, although it is typically doubly redundant parity these days.

Hardware RAID5/6 controllers with battery backed cache and all that jazz are also still being made, sold and bought.

[paulca]

I just switched roles in work from a project with an HDFS filesystem consisting of several dozen petabytes over 5000 nodes.  Thousands of cores and terrabytes of online RAM.

Mostly the disks where "node local" SAS arrays or "rack local" SAS arrays.  Due to it being BigData/Compute cluster, the disks stay with the cores.

That wasn't even the scary part of the architecture.  It was the "near real time" SQL caching layer which got used by end business customers.

Somehow it managed to provide, almost, normal SQL query latency < 1 minute from that HDFS cluster.

I believe it runs on a sub cluster with 1000s of terrabytes of RAM.  Whatever it is, it replaced an IBM Netezza data warehose on IBM Power Servers so I expect it's a beast.  Direct "Hadoop" based queries to the same distributed on disk data would take orders of magnitude longer to query.

[DiTBho]

On the same hardware, ZFS on Linux couldn’t handle ~10M iop

which one? zfs-kmod or zfs-fuse? 

if it was the fuse one, well ... userspace <- from/to -> kernelspace adds some overhead

[DiTBho]

SAS arrays or "rack local" SAS arrays.

got it from the last dumpster dive:
      SAS, 2 channels(1) ------ fiber optic bridge ------ fiber optic HBA

is it like this?

(1)
ch0: { disk0,disk1,disk2,disk4 } on one cable
ch1: { disk0,disk1,disk2,disk4 } on one cable


... why not Infiniband? 

[ve7xen]

I am using btrfs for more or less all my personal stuff for 5ish years. I have had no major issues with it, though obviously it is a small sample size.

On desktops, doing frequent more or less 'free' snapshots both periodically and before system updates has been very valuable, and this is not something that is well supported with 'traditional' filesystems. ZFS support on Linux has always felt quite half-baked to me, and most distros have good support for btrfs built in these days, from bootloader through to snapshotting tools and hooks. I am not too interested in using something 'bolt-on', even if it works well.

On my personal servers, I find piping incremental snapshots around to be an elegant solution to the incremental backup issue. For my personal long-term archives I also insist on a filesystem with data (and metadata) checksumming and scrubbing, which again means choosing between two half-baked options, but btrfs feels by far the more Linux-native.

At $dayjob we are using ext4 for general workloads and XFS for dedicated storage workloads. I have in the long-distant past experienced an XFS data loss incident, so haven't used it on my personal stuff in a decade or more.

One "downside" to XFS is that, while it has a 'discard' option, it apparently hinders performance quite a bit, so if on a SSD, they recommend disabling this option and issuing fstrim separately.
ext4 handles this much better.

This is generally the best-practice advice these days anyway, so I wouldn't really count it against XFS.

[SiliconWizard]

This is generally the best-practice advice these days anyway, so I wouldn't really count it against XFS.

You're right. That's how I set up my workstation lately in the end, even with ext4.

[Marco]

In high capacity storage arrays

Why does a small business need a high capacity storage array for its Excel sheet? That's going to be the reality for the vast majority, they just need storage of email and some bookkeeping cobbled together or some crap off the shelf, not vast amounts of media or measurement data. Even the off the shelf crap can't use up that much storage.

Sure large companies can generate enough plain business data to justify some more effort on the storage, but a small business 99% of the time won't have enough data to mess around with something with finnicky failure modes.

[Nominal Animal]

In high capacity storage arrays

Why does a small business need a high capacity storage array for its Excel sheet? That's going to be the reality for the vast majority, they just need storage of email and some bookkeeping cobbled together or some crap off the shelf, not vast amounts of media or measurement data. Even the off the shelf crap can't use up that much storage.

Sure large companies can generate enough plain business data to justify some more effort on the storage, but a small business 99% of the time won't have enough data to mess around with something with finnicky failure modes.

Nightly backups for a couple of weeks, then weekly, and finally monthly, before stored on RO media.

The reason is often not so much because of hardware failure, but learning that humans do rather stupid things now and then, and recovering from those without angering your clients and losing business is definitely worth the few thousand euros/dollars it costs to set up.

And this is on top of "standard" off-site backups, assuming there is enough internet bandwidth to do so.