Products > Computers

FreeBSD/ZFS, openSUSE/BTRFS or Ubuntu/ZFS(experimental)?

(1/3) > >>

RoGeorge:
Tried this weekend FreeBSD and openSUSE for the first time.  The machine was an i7 4790k, 32GB RAM, nVidia GTX760, Samsung PRO 512GB SSD and few extra HDDs.

The goal is to find an OS for the main desktop in the lab (at home).
- must have a form of snapshot/restore/backup, preferably ZFS as a root file system
- preferably KDE Plasma for GUI
- must be open source
- must not lock the user inside a single world (no Windows or Mac)

FreeBSD was very nice, manged to make it run with the proprietary nVidia drivers and hardware acceleration, has native ZFS integration, though BSD needs little tinkering even for the most basic things, so I find myself constantly googling how to do things that are usually automated in Linux.  As an example to connect the ADALM-PLUTO SDR (a device using LAN over USB) I had to manually look if a new addapter appeared, and to manually bring up the addapter and manually assign an address to it.  In Ubuntu, you just plug the cable and that's it.

openSUSE was the best surprise, very good documentation, amazingly good looking, and it's for the first time when out of the box settings are just right for me.  So far I've tried already almost any other Linux distros, including Gentoo or Arch, but never before I have seen something like openSUSE before, absolutely love it.  Typing from openSUSE with KDE Plasma right now, and even the forum editor looks better from openSUSE!  :D

openSUSE would be my choice in a heartbeat, but it is using BTRFS instead of ZFS for its root disk, and YaST disk features like snapshot/restore/diffs/pools etc. are not working with ZFS.

That's a pity, openSUSE on ZFS would have made the perfect Linux distro.

But then I've stumbled on this blog from a BTRFS dev,  https://josefbacik.github.io/kernel/btrfs/extent-tree-v2/2021/11/10/btrfs-global-roots.html  and the plan is to make major disk format changes, and that's the last thing anybody would want to hear about a file system.

Ubuntu with experimenta ZFS support seems attractive, too.  They developed their own ZSys tool as a ZFS administering helper, and integrate that into Ubuntu, so there are integrated features to roll back, for example automatic OS snapshots, the ability to reboot the OS using any available snapshot, etc.  That would require some extra steps to add a KDE Plasma on top of the regular Ubuntu distro, and the support and documentation are in format more too popular for my needs, but I'm already used to Debian so less reading required.



I'm tempted to leave Kubuntu and switch to openSUSE/BTRFS from now on, and keep ZFS for storage HDDs only.
Other comments or ideas are welcome.

Any observations from current openSUSE users, or any advice regarding BTRFS?

evb149:
Yeah I've been surprised / frustrated / disappointed about BTRFS. 
Every few years I check and it seems like there is some caveat that it isn't really "production ready" as in rock solid, stable, well tested,
basically robust against bugs / data loss cases etc.  Maybe it has changed and is more stable but your own comment suggests otherwise.
And even at its best of promised attributes  it seemed a poor comparison for what ZFS already was back in '2005 or so.
 
On the one hand ZFS has been free in the wild for some decades now
(though quite a shame about the whole solaris / open solaris / open indiana et. al. thing and the ZFS anti-LINUX licensing stuff by SUN)
and it is just about perfect minus some things that SUN didn't specifically design for / finish.  ZFS-Crypto was an after thought and never really
got first class implementation / design support AFAIK though it was said to be "coming soon".  Deduplication another after thought that seemed to
be problematic in actual systems.   Although not entirely a fault of the ZFS implementation overall the lack of robust ECC was / is a concern for
data integrity in desktop / low end workstation machines like we're discussing -- it may be the data on disc is well protected by ZFS checksums but
if the data gets corrupted at the layers of storage interfacing / drivers / OS / host memory buffers then no longer is the integrity checking
preserved end to end by ZFS from the application to the storage device and apparently problems at one time resulted from that.
Some years back IIRC memory management was an issue too for ZFS deployments where it wasn't uncommon for smallish systems to be able to
deploy several TBys of disk storage but be architecturally limited to a small number of GBys of RAM and the ratio of the two wasn't optimal wrt.
some kinds of intermediate data buffers / structures or some such thing so things could get degraded / broken / fail to scale well or something in
such cases.  On the one hand that's a bit better now that at least typical systems can have NN GBy RAM vs N but at the same time now the discs are
NN TBy / unit so the scaling ratio of RAM to pool size is as bad or worse than ever so I don't know what the latest advice about tuning and scaling is
for small systems like you are discussing; maybe it isn't an issue if you have a "small" pool without tons of devices etc.

The other thing I don't know about is how they've currently engineered and optimized the support for the present kinds of very heterogeneous low end storage devices -- * small NVME 0.5-2TB SSDs; * SATA attached small SSDs; * SATA attached slow NAS or archival type spinning disc drives; USB attached archival spinning disc drives, * remote attached storage (e.g. NAS, cloud, ...).

In the consumer market none of those are particularly high quality, many don't well support the expected diagnostic / self test (SMART, SCSI, ...) modes, many are so slow that just one read or write cycle can take many hours or multiple days, they may not deal with errors / error recovery / error handling well or flexibly, the system may not deal with online failures or hot swapping correctly / well, sometimes they "lie" about and / or "fail" power integrity guarantees (does that write operation the drive PROMISED was "finished / synced" really complete / succeed if the power failed just at the wrong moment..), and they're not necessarily very reliable in general.   So in theory one should be able to set up cache / journaling on the smaller / faster devices for optimizing the reliability of the whole pool but whether that actually works reliably in such small systems these days ... eh...?

In my own past experience I've trusted Solaris derivatives and FBSD more than Suse / Ubuntu et. al. for ZFS since it was a first class citizen on the former and something partially / experimentally tacked on by integration of an out-of-distro subsystem in the case of the LINUXes.  Maybe these days it's good enough for some uses to consider LINUX host based ZFS on a workstation.

The other experience I have is that if one builds a low end server (e.g. desktop PC class) and uses it as a storage server it'll probably be pretty reliable and once the initial problems are sorted out one can have "in service uptime" measurements of many weeks or several months sometimes even years no problem for modest uses.

When I create a workstation desktop that's running some LINUX and runs GUI desktop and general purpose office use cases, typically things may not work nearly as reliably.  Maybe it'll run for many days or a few weeks without a crash or some kind of resource starvation problem but not usually many weeks to several months to a few years. 

For a personal storage server maybe it is forgivable to reboot it cleanly a few times a month or weekly or whatever to allow for OS upgrades and refreshing the system state etc.  But I always worry about what will or could get corrupted / lost in the case of unexpected freeze / crash / power fail / hardware fail / hardware lock up etc. cases which happen many times per year or even per month depending on use cases.

I don't know if there's a good way to have a reliable storage server and a highly capable / modern workstation in the same machine.
Even if one virtualized the two one would still have significant limits due to the effective inability for a GPU to work well in most cases when virtualized
without some very ad-hoc configurations to optimize that.  USB pass through is also somewhat limiting in addition to other advanced kinds of PCIE peripherals.

"must have a form of snapshot/restore/backup" -- the problem is that just because you may have a couple of pools attached, maybe even some hot spare
devices, that's not REALLY a backup.  It is a RAID and/or other levels of redundancy.  But one power surge, flood, fire,  or cosmic ray or buggy OS / FS problem of just the wrong sort can happily overwrite / corrupt / break anything that's actually locally directly attached to the machine and writable.  A true backup is a completely independent system that isn't automatically synchronized / mirrored in a dangerous way that could be blind to data loss / corruption cases.

You might check out fedora or centos or one of the centos derivatives:
https://en.wikipedia.org/wiki/CentOS

They may have more integrated system administration oriented tools / documentation / workflows than one typically sees for some more average user oriented distributions like ubuntu desktop etc.

I'd suggest being careful about the RAIDZ and dedup and other kinds of setups that are presenting more chances of things going amiss and taking especially long times and complex processes for device failure recovery.

Wuerstchenhund:

--- Quote from: RoGeorge on November 14, 2021, 09:05:40 pm ---Tried this weekend FreeBSD and openSUSE for the first time.  The machine was an i7 4790k, 32GB RAM, nVidia GTX760, Samsung PRO 512GB SSD and few extra HDDs.

The goal is to find an OS for the main desktop in the lab (at home).
- must have a form of snapshot/restore/backup, preferably ZFS as a root file system
- preferably KDE Plasma for GUI
- must be open source
- must not lock the user inside a single world (no Windows or Mac)

FreeBSD was very nice, manged to make it run with the proprietary nVidia drivers and hardware acceleration, has native ZFS integration, though BSD needs little tinkering even for the most basic things, so I find myself constantly googling how to do things that are usually automated in Linux.  As an example to connect the ADALM-PLUTO SDR (a device using LAN over USB) I had to manually look if a new addapter appeared, and to manually bring up the addapter and manually assign an address to it.  In Ubuntu, you just plug the cable and that's it.
--- End quote ---

I'm not surprised. FreeBSD is great for many things but not as a replacement for a standard desktop system. Hardware support is also way behind what Linux supports.


--- Quote ---openSUSE was the best surprise, very good documentation, amazingly good looking, and it's for the first time when out of the box settings are just right for me.  So far I've tried already almost any other Linux distros, including Gentoo or Arch, but never before I have seen something like openSUSE before, absolutely love it.  Typing from openSUSE with KDE Plasma right now, and even the forum editor looks better from openSUSE!  :D
--- End quote ---

Yes, openSUSE is great, it's the most easy to use Linux diistro there is, and in my view is vastly underrated (probably because it's European).

Also, there is other stuff like OBS (OpenSUSE Build System) which has lots of additional programs that aren't part of the main repositories.


--- Quote ---openSUSE would be my choice in a heartbeat, but it is using BTRFS instead of ZFS for its root disk, and YaST disk features like snapshot/restore/diffs/pools etc. are not working with ZFS.

That's a pity, openSUSE on ZFS would have made the perfect Linux distro.
--- End quote ---

Why?

BTRFS is not just the default file system in openSUSE, it is also the default and fully supported file system in SUSE Linux Enterprise (SLE), which is SUSE's commercial Linux offering. SUSE is also one of the main contributors to BTRFS (as they are to many other FOSS projects). The fact alone that BTRFS is fully supported by the 2nd largest enterprise Linux vendor (after Red Hat) should already tell you that BTRFS, at least on SUSE, is completely reliable. The only thing that isn't supported are the RAID5/6 modes of BTRFS, which are blocked in SLE (you can do RAID/5 on Linux dmraid, which is supported).

What I can tell you is that at work we have several hundreds of TB of important data on BTRFS, and so far it has been rock-solid. We also use SLE and openSUSE for engineering desktops, and again BTRFS has been perfectly fine for everything we throw at it.

You certainly can get ZFS support into openSUSE, but it makes no sense whatsoever. What you'd get is a file system which is optimized for large amounts of rotational media (hard drives, which is what ZFS was designed for) while losing the great integration of snapshots into openSUSE's management tools like YaST/snapper, and likely other functionality as well.


--- Quote ---But then I've stumbled on this blog from a BTRFS dev,  https://josefbacik.github.io/kernel/btrfs/extent-tree-v2/2021/11/10/btrfs-global-roots.html  and the plan is to make major disk format changes, and that's the last thing anybody would want to hear about a file system.
--- End quote ---

Then you should stay away from ZFS because there have been similar changes in ZFS over the years. Which not only saw major changes but is now split in tow separate branches (Oracle ZFS and OpenZFS).


--- Quote ---Ubuntu with experimental ZFS support seems attractive, too.  They developed their own ZSys tool as a ZFS administering helper, and integrate that into Ubuntu, so there are integrated features to roll back, for example automatic OS snapshots, the ability to reboot the OS using any available snapshot, etc.  That would require some extra steps to add a KDE Plasma on top of the regular Ubuntu distro, and the support and documentation are in format more too popular for my needs, but I'm already used to Debian so less reading required.
--- End quote ---

So you'd rather rely on the experimental implementation of ZFS on Ubuntu (which is known as "the Windows amongst Linuxes" for a reason) than on a fully supported BTRFS implementation?

I also question the wisdom using a desktop computer used as a desktop system as storage server. It's usually a good way to lose your data, even more so if you don't have any real backups (and no, RAID is not a backup!). So unless these drives hold hot data (i.e. data you need when working on that computer) offload them somewhere else, e.g., a NAS.



--- Quote from: evb149 on November 15, 2021, 04:34:42 am ---Yeah I've been surprised / frustrated / disappointed about BTRFS. 
Every few years I check and it seems like there is some caveat that it isn't really "production ready" as in rock solid, stable, well tested, basically robust against bugs / data loss cases etc.  Maybe it has changed and is more stable but your own comment suggests otherwise. And even at its best of promised attributes  it seemed a poor comparison for what ZFS already was back in '2005 or so.
--- End quote ---

I'm not sure what you did "check" but as mentioned above BTRFS is (and has been for several years) the default and fully supported file system of the 2nd largest enterprise Linux vendor. Which it wouldn't be if it wasn't "production ready". That may not be true for other distros but that's an issue with that particular distro, not BTRFS.

Also, back in 2005 ZFS was still in pre-release, and even Sun advised customers to stay away from it for anything important. And I still remember the many issues we had with ZFS on Sun's storage systems back in 2008/2009 (we spent quite a bit of time resolving problems in ZFS together with Sun's support, and it took a lot of effort from Sun and a lot of changes to ZFS to get it where it is today).


--- Quote ---In my own past experience I've trusted Solaris derivatives and FBSD more than Suse / Ubuntu et. al. for ZFS since it was a first class citizen on the former and something partially / experimentally tacked on by integration of an out-of-distro subsystem in the case of the LINUXes.  Maybe these days it's good enough for some uses to consider LINUX host based ZFS on a workstation.
--- End quote ---

It depends if your data is worth something. Pretty much all ZFS support in Linux is still "experimental" which is short for "we have no idea if this setup is stable/reliable and you may well lose all your data for no reason".

Nominal Animal:

--- Quote from: RoGeorge on November 14, 2021, 09:05:40 pm ---but it is using BTRFS instead of ZFS for its root disk, and YaST disk features like snapshot/restore/diffs/pools etc. are not working with ZFS.

--- End quote ---
You could just use any FS on top of LVM, and use LVM for the snapshots, restore, et cetera.

You do need to either use thin provisioning, or not commit all disk space for the actual filesystems.  That extra space is needed to describe the differences between the snapshots and the live system.  I don't know how well LVM support is integrated into YaST, although its documentation does say it does support thin-provisioned LVM volumes.  I myself use the command-line LVM tools, and don't use OpenSUSE, so I cannot say whether the UI tools are any good.

LVM is basically a very thin layer on top of block devices like HDDs and SSDs, and allows combining them into a single logical device.  It does support software RAID, using Linux Device Manager and Multiple Device, DM and MD.  The filesystem on top of that doesn't really matter, because the storage is managed at the underlying block level.

nightfire:
I use FreeBSD since 3.2, so more than 20 years now. Although I love the system, I type this reply from my main Windows machine.
FreeBSD has a very big history and reputation on the server side of things, and can be used as a desktop, but lots of things that one would need in an advanced desktop environment are simply not the strengths of FreeBSD or one of its Desktop-optimised Distros like GhostBSD or FuryBSD.

That said, I used a FreeBSD workstation some years at my previous employer (Datacenter, where we used Unix for our main stuff), but that was mostly used as a console multiplexer and webbrowser station. Currently I have some tinkering project going on to re-use my old desktop as some experimental graphics workstation for photography and picture/video editing (over network with X11 forwarding), and its nice to see what stuff one can do with OpenSource Software.
But as a "daily driver" with lots of graphics stuff going on, especially as a beginner to the whole *NIX stuff I would not necessarily recommend a standard FreeBSD box.
Here has to be kept in mind that the *BSD approach to most things is a more conservative one, so that usually you would get the most advanced desktop stuff on a penguin, not a daemon.

ZFS is a different kind of story- due to licensing issues, Linux folks frowned a long time on ZFS, so the whole implementation on BSD had more time to mature. Back then, when my last employer made some decisions on what Filesystem we would host some filehosting for our customers, and we chose ZFS on (Open)Solaris for the task. As Oracle back then bought out Sun, some colleagues were tasked to have a look at BTRFS as a possible replacement, but back then (7+ years) BTRFS was deemed not to be stable enough in comparison.
Since then lots of things should also have been improved, and I did not follow that development, as in my recent job some tools have radically shifted towards other things.

Navigation

[0] Message Index

[#] Next page

There was an error while thanking
Thanking...
Go to full version