Yet another ... which Linux file systems ?

[Wuerstchenhund]

Just a warning using BTRFS, it can destroy your data.   I had a BTRFS RAID which sounded great and seemed to work... until a month or two after I put my VM files there.  I think (read it somewhere) that the constant read/write of the VM didn't allow BTRFS enough time to do its magic. (this happened 3 times before I switched filesystems) The VM files were corrupt and could not be salvaged.  I am still using the same host hardware and software with EXT4 and all is fine years later. It may be fine for storing pictures, but DO NOT use it to store your VM or any high volume read/write.

You are right, but your problem wasn't BTRFS as such, your problem was BTRFS *RAID* which is still considered 'unstable' (actually, it's now only RAID5/6 which is unstable, the other RAID levels work fine, although have still performance issues)  and should really not be used.

BTRFS as file system however is absoutely fine. It's been the default file system for SUSE Linux Enterprise for a few years now, which means its a fully supported file system cleared for enterprise use. We have a lot of data on BTRFS, and have been for years with no problems whatsoever. However, SUSE doesn't use the BTRFS RAID modes, for good reason (I don't think it's even enabled in SLES and openSUSE). If you need software RAID then the current solution is to run BTRFS on top of an lvm2 RAID.

(I might have mentioned this before) also all my personal data rests on spinning rust in my homeserver (trusty old HP Microserver Gen8) running openSUSE 15.1 and with BTRFS as file system. Not a single problem. But then, I don't use BTRFS RAID, as the server has a proper SAS/SATA HW RAID controller (which also takes care of bit rot).

I have three other Microserver Gen8 and a Dell PowerEdge T320 for other stuff I work on, all on openSUSE and BTRFS. Same here, no problems.

When it comes to computing I tend to go with "if it's good enough for mission-critical business then it's good enough for me", staying away from consumer-grade hardware (or any Linux which is in any way related to something from Canonical) as far as possible, and (knock on wood) so far this has paid off very well

[Monkeh]

Just a warning using BTRFS, it can destroy your data.   I had a BTRFS RAID which sounded great and seemed to work... until a month or two after I put my VM files there.  I think (read it somewhere) that the constant read/write of the VM didn't allow BTRFS enough time to do its magic. (this happened 3 times before I switched filesystems) The VM files were corrupt and could not be salvaged.  I am still using the same host hardware and software with EXT4 and all is fine years later. It may be fine for storing pictures, but DO NOT use it to store your VM or any high volume read/write.

You are right, but your problem wasn't BTRFS as such, your problem was BTRFS *RAID* which is still considered 'unstable' (actually, it's now only RAID5/6 which is unstable, the other RAID levels work fine, although have still performance issues)  and should really not be used.

BTRFS as file system however is absoutely fine. It's been the default file system for SUSE Linux Enterprise for a few years now, which means its a fully supported file system cleared for enterprise use. We have a lot of data on BTRFS, and have been for years with no problems whatsoever. However, SUSE doesn't use the BTRFS RAID modes, for good reason (I don't think it's even enabled in SLES and openSUSE).

I might have mentioned this before: all my personal data rests on spinning rust in my homeserver (trusty old HP Microserver Gen8) running openSUSE 15.1 and with BTRFS as file system. Not a single problem. But again, I don't use BTRFS RAID, as the server has a proper SAS/SATA HW RAID controller (which also takes care of bit rot).

I have three other Microserver Gen8 and a Dell PowerEdge T320 for other stuff I work on, all on openSUSE and BTRFS. Same here, no problems.

When it comes to computing I tend to go with "if it's good enough for mission-critical business then it's good enough for me", staying away from consumer-grade hardware (or any Linux which is in any way related to something from Canonical) as far as possible, and (knock on wood) so far this has paid out.

Even non-RAID, I have had a nasty failure with btrfs, and their fsck just doesn't work. IMO it just isn't mature enough yet. I like it, I love the features, and I love that it's actually upstream unlike ZFS, but I'm not convinced it's there yet.

[Wuerstchenhund]

Even non-RAID, I have had a nasty failure with btrfs, and their fsck just doesn't work. IMO it just isn't mature enough yet. I like it, I love the features, and I love that it's actually upstream unlike ZFS, but I'm not convinced it's there yet.

I guess it's down to the distro. SUSE is one of the major contributor to BTRFS and they wouldn't set on it for their main cash cow (Enterprise Linux) and tie their reputation to it if the file system wasn't up to scratch. But (like Red Hat) they often put in tweaks which may not (yet) be part of the mainline kernel, so I have no idea how BTRFS works on other Linux variants. I certainly wouldn't use it with any of the rolling distros (but then, Rolling Release and important data doesn't really fit together anyways).

[PKTKS]

There are a lot of straw men being built and burned here now.

Bavarians like myself will  only smell royal mumies stinking arrogant words

Paul

[Wuerstchenhund]

There are a lot of straw men being built and burned here now.

Bavarians like myself will  only smell royal mumies stinking arrogant words

Well, at least this Bavarian here is certain to have smelled some manure 

[bd139]

There are a lot of straw men being built and burned here now.

Bavarians like myself will  only smell royal mumies stinking arrogant words

Besserwisser is the word you're looking for

[PKTKS]

There are a lot of straw men being built and burned here now.

Bavarians like myself will  only smell royal mumies stinking arrogant words

Besserwisser is the word you're looking for

versuchen Sie es stattdessen der Dummkopf Worte des Knochenkopf

[bd139]

I was insulting myself there. I'd expect some decorum but alas there is none...

[BravoV]

When it comes to computing I tend to go with "if it's good enough for mission-critical business then it's good enough for me", staying away from consumer-grade hardware...

I guess this is my problem challenge, isn't it ? 

[Wuerstchenhund]

When it comes to computing I tend to go with "if it's good enough for mission-critical business then it's good enough for me", staying away from consumer-grade hardware...

I guess this is my problem challenge, isn't it ? 

I don't know. Lots of people like building their own computers, or just want the latest and fastest desktop processors just to push them further through overclocking. I used to do the same back then when the Pentium 60 is still a thing.

Since then I'm just buying business stuff (often 2nd hand). I can plug it together, switch it on and it works. No fuss, no hassle, nothing. The only exception is graphics cards (mostly Nvidia, also because AMD cards mostly got excessively tall for some reason) but even here I try to stay with the conservative variants. Everything has ECC RAM, and if it has spinning rust then it's connected to a hardware RAID controller.

The bonus is that 2nd hand enterprise gear is often dirt cheap

I keep the same principle for my working OSes (openSUSE/CentOS Linux and Windows 10 LTSB), although my gaming PC has standard Windows 10 Pro and there are two Macs as well. I stick with openSUSE also because I'm naturally lazy and YAST saves me to fiddle with the command line and OBS to search for packages which aren't in the standard repos.

There are downsides, though. For example, no Ryzen or EPYC for me because right now there is no HPE/Dell tower server or workstation using these processors. CPUs are pretty much all intel (I stick with dual processor capable XEONs). No superfast gaming CPUs or extremely clocked memory. So there's that. 

It works for me because I value stability and reliability over performance and looks. Doesn't mean it's for everyone

[bill_c]

Wuerstchenhund, it may have been the 3 way combo of BTRFS + RAID + VM that did it in.  I never got around to trying BTRFS + VM without RAID so I can't say if that is safe or not. If I had more time I would have given it a try.

[radar_macgyver]

Btrfs gave me more grief today. Workstation with btrfs root fs, single SSD (no RAID, not even dup metadata). It's a small (128 GB) SSD, with scripts to watch "df" and email me if the utilization went above 90%. Everything seemed to be going great, until a "yum update" bombed out with ENOSPC on all writes. df still showed the disk was only about 45% full, but btrfs filesystem show indicated 100% used. A bit of searching pulled up gems like this and this  . While doing a btrfs balance 'recovered' the filesystem, the failed update hosed a bunch of stuff on /, so I was forced to nuke everything and rebuild (hooray for backups). I'm now running xfs on top of lvm (Centos 8 defaults). Let's see how well that goes. Not thrilled about the direction Centos 8 has taken, but that's another story.

[BravoV]

Boot drive at NVME will be on EXT4, still digesting on the 3 HDs. 

md and ext4. ZFS is a moving target, btfs is a moving target with bits falling off it.

*awaits flames*

Done deal, md ext4 for the 3 HDs it is, while keeping my eyes on the ZFS progress.

To be honest the main points that interested me in the ZFS (still keeps learning though) are the checksum, self curing and SLOG + L2ARC feature that probably I could coupled with mirrored 2 x cheap SSDs, time will tell what next.

Again, the desktop is protected by UPS (and works as gracefully shutdown when power outage) and also we "diligently" do routine full backup externally.

Thanks everyone.   

[Wuerstchenhund]

Btrfs gave me more grief today. Workstation with btrfs root fs, single SSD (no RAID, not even dup metadata). It's a small (128 GB) SSD, with scripts to watch "df" and email me if the utilization went above 90%. Everything seemed to be going great, until a "yum update" bombed out with ENOSPC on all writes. df still showed the disk was only about 45% full, but btrfs filesystem show indicated 100% used. A bit of searching pulled up gems like this and this  . While doing a btrfs balance 'recovered' the filesystem, the failed update hosed a bunch of stuff on /, so I was forced to nuke everything and rebuild (hooray for backups). I'm now running xfs on top of lvm (Centos 8 defaults). Let's see how well that goes. Not thrilled about the direction Centos 8 has taken, but that's another story.

Well, "yum" sounds suspiciously like something RH based (Fedora?).

I would never use BRTFS under a RH based distro, considering that RH has never moved BTRFS to a fully supported feature (up to RHEL 7 it was available also unsupported) and has completely removed all support from RHEL 8. It's mostly political, as RH wants to push its XFS extensions (Project Stratis) for its platform, with which BTRFS does compete.

With Fedora, there's the additional thing that it's pretty much a playground for Red Hat, and while it's not truly rolling release it's mostly bleeding edge and stability more often than not takes a backseat.

In my view, as of this day the only way to get reliable BTRFS is with SUSE. This may be different in a couple of years, but right now I personally wouldn't trust BTRFS on any other distro than SLES/openSUSE.

Wuerstchenhund, it may have been the 3 way combo of BTRFS + RAID + VM that did it in.  I never got around to trying BTRFS + VM without RAID so I can't say if that is safe or not.

Under what distro?

If I had more time I would have given it a try.

If you find the time I can only recommend to give openSUSE Leap (*not* Tumbleweed, which is rolling release!) a try. Aside BTRFS, there are other things which make it a great distro, like YAST.

[radar_macgyver]

@BravoV you may want to test how much memory ZFS takes up for your intended data set. It is known to be a memory hog. All my ZFS instances are pure storage servers (NAS) so this doesn't matter so much, but if it's in a desktop box it may need even more RAM to keep your average web browser happy. Have you considered a NAS but with 10G networking? It's become fairly inexpensive these days, particularly with SFP+ switches/NICs instead of 10Gbase-T. I have a FreeNAS box at home (internally uses ZFS, based on a Xeon 1541-D with 10G NICs) and it's about as fast as my desktop SSD over NFS.

@Wuerstchenhund this was on a Centos 7 box, and when I had first put it together btrfs was not considered deprecated. They announced that it was later on. Centos 8 (and its corresponding RHEL) no longer offer btrfs at all, and in fact default to xfs on LVM (used to be ext4 on LVM). Now I know why. Time to switch distros, I guess. I still don't trust btrfs; that even the official FAQ mentions "the usual errors" does not inspire confidence.

[BravoV]

@BravoV you may want to test how much memory ZFS takes up for your intended data set. It is known to be a memory hog. All my ZFS instances are pure storage servers (NAS) so this doesn't matter so much, but if it's in a desktop box it may need even more RAM to keep your average web browser happy. Have you considered a NAS but with 10G networking? It's become fairly inexpensive these days, particularly with SFP+ switches/NICs instead of 10Gbase-T. I have a FreeNAS box at home (internally uses ZFS, based on a Xeon 1541-D with 10G NICs) and it's about as fast as my desktop SSD over NFS.

Yes, I dropped the idea of RAIDZ-1 on desktop, just don't want the complexity on managing/tuning/trouble-shooting, if somehow it starting to cripple the main desktop "interactive activity", this ZFS should goes only to NAS/server role for this current state, cmiiw.

Regarding the wired networking, yes, we've discussed it, the cost of rewiring is clearly off budget and off the plan, as she doesn't like the extra network dangling wires except power cables only , we have two stories house, her desktop working room (wifi-ed) is quite far away at 2nd level, while my room, lab ... lair  , which all computing + electronics stuff reside, is at far corner at the 1st level.

Its more like budget & aesthetically constraint rather than technicality.

Thanks for the suggestion. 

[Wuerstchenhund]

@Wuerstchenhund this was on a Centos 7 box, and when I had first put it together btrfs was not considered deprecated. They announced that it was later on. Centos 8 (and its corresponding RHEL) no longer offer btrfs at all, and in fact default to xfs on LVM (used to be ext4 on LVM). Now I know why. Time to switch distros, I guess. I still don't trust btrfs; that even the official FAQ mentions "the usual errors" does not inspire confidence.

BTRFS was deprecated in RHEL 7.4:

https://access.redhat.com/solutions/197643/

However, even before then BTRFS in RHEL has always been a Technology Preview, completely unsupported. BTRFS on RHEL (and clones) was never a good idea

And if some of the issues listed on the BTRFS website frighten you then you better shouldn't have a look at the list of problems of ZoL.

Even ext4 wasn't (and still isn't) free from issues, and it also had some data loss inducing bugs.

There's no perfect file system. Which is why if your data is important you need to have backups.

[BravoV]

Just an update ...

Figured out locally at the used servers merchant, they have tons of used RAID controller, and offered quite cheap a DELL PERC H730 , with 1GB cache, hardware XOR engine and BBU, about $75 each and bought two of them    just to have confidence of backup as these are used.



So , RAID 5 is handled by dedicated controller and still EXT4 it is.

[bill_c]

@ Wuerstchenhund:
Distro was Ubuntu, I will give openSUSE a try one day.

[RoGeorge]

So far, my best experience for a Linux based OS was Gentoo.
However, right now I'm typing from an Ubuntu 20.04 LTS.

You can draw whatever conclusion you may want out of this situation.
 

[mansaxel]

I've seen most of my ideas already mentioned, already, so I'll just chime in one "dreamer" vote for raidz2 on FreeBSD, on a separate storage machine. And cabled network. Wifi is a joke, unless you can pour silly money into it. Especially so if you're trying RPC (like file system access, roughly)  over it, because delay is very stochastic.  My wifi at home is silly money level, Würstchenhund style (previous generation enterprise stuff, an AP in every room, central controller VM managing them, all connected with Gigabit Ethernet, etc.), and it's barely OK.

I'll give our friend from Brazil right on one thing. Systemd is a pile of festering shite. Not that it isn't a good idea, because parts of it is, but the combination of bad NIH attitude, the childish desire to put your name on a complete set of badly "designed" utilities and interfaces that seek to "replace" things that have been carefully built over decades, utterly incompetent handling of DNS and DHCP, security ignorance (user name starting with digit 0 is interpreted as UID 0, "works as designed") and the crown of the dung heap, binary logs so you'll never understand why it broke, makes me install one of the BSDen. And Devuan when I can't avoid penguins.

[madires]

Meanwhile most linux related threads change into systemd bashing. I don't like systemd either, but there's no need to use any possible opportunity for systemd bashing. It's boring and tiring!

[magic]

Hmm, maybe let's talk PulseAudio or GNOME for a change?

 

[PKTKS]

(..)

I'll give our friend from Brazil right on one thing. Systemd is a pile of festering shite. Not that it isn't a good idea, because parts of it is, but the combination of bad NIH attitude, the childish desire to put your name on a complete set of badly "designed" utilities and interfaces that seek to "replace" things that have been carefully built over decades, utterly incompetent handling of DNS and DHCP, security ignorance (user name starting with digit 0 is interpreted as UID 0, "works as designed") and the crown of the dung heap, binary logs so you'll never understand why it broke, makes me install one of the BSDen. And Devuan when I can't avoid penguins.

 

systemd is a monumental pile of bad EGO trip from those goonies behind it

All security measures and good practices were just "dumped"
with those "ready made" slogans  like... "faster", "modern",
"it is here to stay", "industry blah", etc..

Try to place a fully SAFE  SOCKS firewall into a fully operational NAS
with all file system gizmos of today top security (including encrypt
protocols) and put that into a proper "SANELY CRAFTED SERVICE IMAGE"

Nothing fits.

systemd  was designed by retarded imbeciles with no idea
about the amount of effort  placed into SECURITY and best practices
standards in *NIX.

The binary log obviously is meant to introduce their own
systemd.registry and systemd.license_manager like shit..
running like nobody really having a clue what they do...

Paul

[Marco]

SystemD is made to sell Redhat service contracts, an eco system under their exclusive control as all encompassing as they can get away with.

No getting rid of it after Debian folded really, there was a small window where Debian could have gone for runit but it's gone now. You can coast on old SysV scripts slowly getting out of sync with the evolving programs for a while, but eventually you'll have the choice to just start writing your own init scripts for everything or accept the D ... SysV is a dead end. Unfortunately it's the road most the non D distributions follow because it's also the road of least resistance, except Void.

As for file systems, XFS seems to be becoming the central choice for Linux servers. When I make a little time I'm going to try to setup a server with LVM+XFS, using LVM snapshots for incremental backup.