Replacement NAS

[Sal Ammoniac]

I've got an old Drobo NAS here at home that is 15 years old. Now that Drobo is out of business and support is no longer available, I need to replace it with something more modern before it fails and takes my data with it.

Rather than buying another commercial NAS, I've been thinking about building a FreeBSD box, adding a bunch of disks, and configuring them as a ZFS pool. Has anyone done this? Is it a good idea?

[woofy]

Yes, for the last couple of years until a few weeks ago I've been running TrueNAS Core which is a FreeBSD based NAS. I'm now running TrueNAS Scale which is Debian based.
Both have performed well with no issues.

[Halcyon]

+1 for TrueNAS. Very simple out-of-the-box solution with enterprise features. I've been using it since the start of FreeNAS. Absolutely rock solid.

[NiHaoMike]

Just build up a PC running Proxmox. Doesn't have to be very fancy, I have one running on an i5-7500.

[vad]

Another vote for TrueNAS. I’ve been running FreeNAS (the old name of the free community version) and TrueNAS Core for over seven years without any issues or data loss. I currently have two servers: one old rack mounted DIY build and another TrueNAS Mini appliance from iXsystems.

Bear in mind that ZFS, with all its great features, is intolerant to crashes, and there is a near-zero chance of recovering a corrupt ZFS pool. So make sure you back up your unique data to another device or into the cloud. If you build your own NAS, pay close attention to hardware recommendations and best practices, such as ECC RAM, UPS.

[Messtechniker]

+1 for Open Media Vault.
My secondary NAS, with the primary being a Synology NAS.
German: "Kein Backup, kein Mitleid" i.e "No backup, no sympathy"

[dobsonr741]

+1 for TrueNAS. Reliable. So many features, even Apple Time Machine backup server.

[SiliconWizard]

My current NAS is years old and I built it from scratch around a mini-ITX motherboard, WD Red HDDs in RAID, RAID1 for the root partition (so it's mirrored over 2 of the drives) and RAID5 for data, ext4 FS. Using Centos.
It's been running solid for all these years.

My next one is probably going to be a bit different, but likely not *that* different. I'll still likely be building hardware from selected parts, and the OS is probably going to be FreeBSD this time, if just to give FreeBSD a shot.
But I'll still probably use a RAID scheme - while ZFS looks nice and modern, being rather intolerant to corruption, as vad mentioned, is not something I personally want to deal with. But that may work for some others.

I'll also probably configure it myself rather than use TrueNAS, mostly because that'll be an opportunity to discover and learn FreeBSD. But otherwise, TrueNAS is fine.

[Halcyon]

Another vote for TrueNAS. I’ve been running FreeNAS (the old name of the free community version) and TrueNAS Core for over seven years without any issues or data loss. I currently have two servers: one old rack mounted DIY build and another TrueNAS Mini appliance from iXsystems.

Bear in mind that ZFS, with all its great features, is intolerant to crashes, and there is a near-zero chance of recovering a corrupt ZFS pool. So make sure you back up your unique data to another device or into the cloud. If you build your own NAS, pay close attention to hardware recommendations and best practices, such as ECC RAM, UPS.

Eh, I wouldn't bother with ECC RAM personally. It offers very little benefit. Even as a user who has a server rack and enterprise gear at home, if I was to rebuild my NAS, I wouldn't bother with ECC RAM. It really doesn't matter.

As for "intolerant to crashes", I strongly disagree. I've had servers simply switch off after the UPS battery dies, with no impact to the ZFS pool. ZFS is extremely resiliant. Even in cases of the underlying OS crashing, ZFS should remain intact. Severe file system damage (ZFS or otherwise) is usually either the sign of hardware problems, or user error (or both). For example, allowing the pool to degrade to such a point that you're forced to try and restore data from a failing hard disk in order to restore your pool.

ZFS is an extremely robust and resiliant file system. Would highly recommend it, but regardless, always have a backup of your data on seperate media. I still don't consider ZFS as a form of "backup", just like traditional RAID isn't.

[nightfire]

ZFS has gotten better over the years, but still a system with a really bad crash will not be recoverable. With my last employer, we began using ZFS on Solaris practically from the beginning, and in those times they had no fsck utility available- and some time later, lots of people learned stuff the hard way, that after passing some point you had to pull the backups in case of a server crash...

This said, ZFS is as good as its operator that maintains the system- and it should get lots of memory to really leverage its strengths.

[vad]

Another vote for TrueNAS. I’ve been running FreeNAS (the old name of the free community version) and TrueNAS Core for over seven years without any issues or data loss. I currently have two servers: one old rack mounted DIY build and another TrueNAS Mini appliance from iXsystems.

Bear in mind that ZFS, with all its great features, is intolerant to crashes, and there is a near-zero chance of recovering a corrupt ZFS pool. So make sure you back up your unique data to another device or into the cloud. If you build your own NAS, pay close attention to hardware recommendations and best practices, such as ECC RAM, UPS.

Eh, I wouldn't bother with ECC RAM personally. It offers very little benefit. Even as a user who has a server rack and enterprise gear at home, if I was to rebuild my NAS, I wouldn't bother with ECC RAM. It really doesn't matter.

As for "intolerant to crashes", I strongly disagree. I've had servers simply switch off after the UPS battery dies, with no impact to the ZFS pool. ZFS is extremely resiliant. Even in cases of the underlying OS crashing, ZFS should remain intact. Severe file system damage (ZFS or otherwise) is usually either the sign of hardware problems, or user error (or both). For example, allowing the pool to degrade to such a point that you're forced to try and restore data from a failing hard disk in order to restore your pool.

ZFS is an extremely robust and resiliant file system. Would highly recommend it, but regardless, always have a backup of your data on seperate media. I still don't consider ZFS as a form of "backup", just like traditional RAID isn't.

Many decades ago, when I lived in Russia, there was a culture among drivers not to wear seatbelts. At my workplace, the chauffeurs took it as a personal affront if you dared to buckle up in the passenger seat, as if questioning their safe driving skills. They were seasoned professionals, with more years behind the wheel than they could count. They all presented a compelling argument: they claimed they had driven cars throughout their entire careers, and none of them had ever been injured in a car accident.

Can you run TrueNAS on non-recommended hardware and never suffer total loss of a ZFS pool? Sure, you can, if you're lucky enough. Should you run it on a machine with non-ECC memory and without battery power backup? That depends on many factors, such as how much you value your data, whether you have a backup, what your RTO and RPO are for your disaster recovery plan, and how much you value your time, keeping recovery time in mind.

If the NAS crashes before it syncs pending metadata changes to disks (where metadata stands for file allocation tables of the beautiful COW file system), your entire ZFS pool will be lost with little prospect of recovery. And unlike inferior NTFS, FAT32, and EXT file systems, you will not find any tool, commercial or open source, to salvage even a fraction of data from a bad ZFS pool.

But who am I to suggest you wear the seatbelts?

[Halcyon]

If the NAS crashes before it syncs pending metadata changes to disks (where metadata stands for file allocation tables of the beautiful COW file system), your entire ZFS pool will be lost with little prospect of recovery. And unlike inferior NTFS, FAT32, and EXT file systems, you will not find any tool, commercial or open source, to salvage even a fraction of data from a bad ZFS pool.

That's the entire point of copy-on-write file systems, like ZFS. It's designed to withstand sudden power failures or incomplete writes, without catastrophic loss of data. Live, good data is never overwritten and that includes the metadata.

[vad]

If the NAS crashes before it syncs pending metadata changes to disks (where metadata stands for file allocation tables of the beautiful COW file system), your entire ZFS pool will be lost with little prospect of recovery. And unlike inferior NTFS, FAT32, and EXT file systems, you will not find any tool, commercial or open source, to salvage even a fraction of data from a bad ZFS pool.

That's the entire point of copy-on-write file systems, like ZFS. It's designed to withstand sudden power failures or incomplete writes, without catastrophic loss of data. Live, good data is never overwritten and that includes the metadata.

Design and real-world use are different things. The Boeing 737 MAX was supposed to be a superior design over the 737NG. In real life, it proved to be a disaster. Even then, Sun Microsystems never intended for ZFS to run on a recycled, ancient desktop from a dumpster.

You may refer to TrueNAS community forums for hardware guides and horror stories from those who did not RTFM.

Edit: Also, all appliances from TrueNAS vendor iXsystems come with ECC memory.

[Halcyon]

iXsystems' hardware requirements are quite broad. See: https://www.truenas.com/docs/core/gettingstarted/corehardwareguide/
They don't recommend ECC as a requirement, but says you can use it. Since their appliances are designed for enterprise, of course they would come with ECC RAM. Most, if not all, enterprise gear does.

I appreciate your analogies vad, but I don't think you're making a very strong argument. ZFS is a resilient filesystem by design and it's certainly lived up to that design in real-world applications for quite a long time.

If you can demonstrate ZFS pool corruption on a PC by simply yanking the power cord or causing the OS to crash, I'll stand to be corrected. Use the cheapest, crappiest hardware if you like, as long as it meets their requirements. However based on many years of experience in this industry, I haven't seen it happen unless there is some underlying cause that hasn't been addressed (which I usually chop up to user error). As they say, the proof is in the pudding. If you can show me that ZFS is "intolerant to crashes" (your words, not mine), then I will withdraw my earlier statements. And by "show", I don't just mean some vague forum post, anyone can cherry pick comments on the internet, but I mean actual proper evidence of this alleged weakness in ZFS.

You've already claimed that "If the NAS crashes before it syncs pending metadata changes to disks (where metadata stands for file allocation tables of the beautiful COW file system), your entire ZFS pool will be lost with little prospect of recovery" which is demonstrably false, so I'm sorry, I don't have a lot of faith in your advice.

[nctnico]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

Additionally modern nas systems have many extra features. I'm quite happy with a Qnap nas btw but that is only one data point.

[Messtechniker]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

That's precisely why I run a PC NAS once a week for a few hours as a secondary NAS.

[SiliconWizard]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

"Any form" is a vast range. Between a full desktop-like PC purposed for this, with an average idle power consumption of 80W-100W and a small system around a mini-ITX motherboard which will idle at around 15W-20W, there's quite a span.

Other than that, if this is a NAS for personal or small business use, it may not (in the case of personal, most surely not) require being on 24/7, but maybe only a couple hours a day. You don't need to leave it on. My personal NAS is built around a mini-ITX motherboard, draws about 20W when "idle", and I put in in standby most of the time (where it draws only 1W or so) and wake it up with WOL when needed.

[coppice]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

You do need to be careful about power consumption, but saying all PCs are inefficient is misleading. I have have been using the same Intel i3 system as a home server for about 12 years. It idles at 12-13W from the wall, and quite a bit of that is taken by 2 3.5" hard disks. Choosing a suitable motherboard and power supply is a big factor. Some motherboards with massive overclocking potential can have a horrible idle power draw. Some power supplies seem to be designed with no regard to their efficiency outside the range laid down in the 80PLUS spec.

[macboy]

Bear in mind that ZFS, with all its great features, is intolerant to crashes, and there is a near-zero chance of recovering a corrupt ZFS pool. So make sure you back up your unique data to another device or into the cloud. ...

My experience is the exact opposite. I am still using a ZFS pool that I created 15 years ago. The disks in the array are now more than an order of magnitude larger, but the pool and my data has persisted all this time. I have backups, but have never needed to recover a byte of data from them. I have replaced old small disks with new big ones, replaced sick/dying disks with healthy ones, and replaced outright dead disks with new ones. Pool always up and available, data intact.
The difference may be the operating system. I have always used Solaris or a derivative. ZFS was designed on - and for - Solaris.

I do agree though, that just using redundancy (whether ZFS or some other RAID scheme) is not equivalent to having a separate backup.

[nctnico]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

You do need to be careful about power consumption, but saying all PCs are inefficient is misleading. I have have been using the same Intel i3 system as a home server for about 12 years. It idles at 12-13W from the wall, and quite a bit of that is taken by 2 3.5" hard disks. Choosing a suitable motherboard and power supply is a big factor. Some motherboards with massive overclocking potential can have a horrible idle power draw. Some power supplies seem to be designed with no regard to their efficiency outside the range laid down in the 80PLUS spec.

Well, compared to a relatively simple ARM processor optimised for super low power, even an i3 is quite a power hog. For comparison: the Qnap NAS I have consumes little over 7Watts max. including the hard drive  according to the specs. But I'm sure this number is too high as I put a low power (2.9W idle), 5400 rpm hard drive in it.

[coppice]

I'd keep power consumption in mind. A pc in any form isn't particularly efficient.

You do need to be careful about power consumption, but saying all PCs are inefficient is misleading. I have have been using the same Intel i3 system as a home server for about 12 years. It idles at 12-13W from the wall, and quite a bit of that is taken by 2 3.5" hard disks. Choosing a suitable motherboard and power supply is a big factor. Some motherboards with massive overclocking potential can have a horrible idle power draw. Some power supplies seem to be designed with no regard to their efficiency outside the range laid down in the 80PLUS spec.

Well, compared to a relatively simple ARM processor optimised for super low power, even an i3 is quite a power hog. For comparison: the Qnap NAS I have consumes little over 7Watts max. including the hard drive  according to the specs. But I'm sure this number is too high as I put a low power (2.9W idle), 5400 rpm hard drive in it.

I used to have power consumption like that with an Atom based home server, before the i3 one I have used since. I can't remember why I chose a higher consumption option for the replacement. There were still plenty of Atom options back then. Its sad they disappeared. The thing I like about these options is anything running on other Linux machines I have runs exactly the same on that server. Pushing functions beyond basic NAS to that server is trivial.

[NiHaoMike]

Well, compared to a relatively simple ARM processor optimised for super low power, even an i3 is quite a power hog. For comparison: the Qnap NAS I have consumes little over 7Watts max. including the hard drive  according to the specs. But I'm sure this number is too high as I put a low power (2.9W idle), 5400 rpm hard drive in it.

Then the vendor does something unfriendly to the user after the device goes out of support and all those savings are gone. Or worse, the device fails and the proprietary RAID effectively has ransomwared your data. At the least, make sure there's a decent aftermarket firmware community for it (even if you have no initial plans to use it) and that there's a way to read off the data by connecting the disks to a regular PC.

[David Hess]

I repurposed my old Phenom 2 940 workstation as a NAS for testing various configurations.

TrueNAS Core (FreeBSD) worked great, but I had problems with Samba where files or directories could be created that could not be deleted.  TrueNAS Scale (Linux) worked but did not support the hardware quite as well.

Since I had a Windows 10 license for it, I ended up just using Storage Spaces for now which is more flexible about modifying the array of drives.  Performance was about the same either way, but getting good performance out of Storage Spaces requires learning how to configure details of the array from the command line.

[vad]

You've already claimed that "If the NAS crashes before it syncs pending metadata changes to disks (where metadata stands for file allocation tables of the beautiful COW file system), your entire ZFS pool will be lost with little prospect of recovery" which is demonstrably false, so I'm sorry, I don't have a lot of faith in your advice.

I don't mean to be rude, but there's an old Russian saying: “A fool learns from his own mistakes, a wise man from the mistakes of others.”

So COW file system is a superior design compared to journaling file systems. I got it. There's another file system called Btrfs. Like ZFS, it's a superior copy-on-write file system “designed to withstand sudden power failures or incomplete writes without catastrophic data loss”. Here's a recent firsthand experience shared by a prominent figure whom I've been following for almost 30 years:

Why is the Btrfs file system as implemented by Synology so fragile?
February 29, 2024 by Phil Greenspun

We had a few seconds of power loss the other day. Everything in the house, including a Windows machine using NTFS, came back to life without any issues. A Synology DS720+, however, became a useless brick, claiming to have suffered unrecoverable file system damage while the underlying two hard drives and two SSDs are in perfect condition. It’s two mirrored drives using the Btrfs file system (the Synology default, though ext4 is also available as an option). Btrfs is supposedly a journaling file system, which should make this kind of corruption impossible. Yet searching the Internet reveals that Synology suicides are commonplace. Here’s one example that pins the blame on the SSDs being enabled as read/write caches (but given that the SSDs are non-volatile why isn’t the Synology software smart enough to deal with the possibility of a power outage even when read/write caching (seems to be the default) is enabled? The Synology web page on the subject says you need two SSDs (which I have) for “fault tolerance” and doesn’t mention that the entire NAS can become a brick after losing power for a few seconds).

Given that Synology has only one job, i.e., the secure storage of data, this strikes me as a spectacular failure of corporate mission.

Readers: Have you seen this kind of failure before? NTFS was introduced by Microsoft in 1993 and I’ve never seen it completely destroyed by a power interruption.

So, perhaps ZFS is the better COW file system compared to Btrfs, and maybe all the folks on the TrueNAS forum are mistaken about ECC memory and battery backups, along with all the TrueNAS guides being wrong. But what if they're not? What if wearing seatbelts really does save your life when you hit a pole at 40 miles per hour?

The full blog post is available here: https://philip.greenspun.com/blog/2024/02/29/why-is-the-btrfs-file-system-as-implemented-by-synology-so-fragile/

[Halcyon]

So, perhaps ZFS is the better COW file system compared to Btrfs, and maybe all the folks on the TrueNAS forum are mistaken about ECC memory and battery backups, along with all the TrueNAS guides being wrong. But what if they're not? What if wearing seatbelts really does save your life when you hit a pole at 40 miles per hour?

The full blog post is available here: https://philip.greenspun.com/blog/2024/02/29/why-is-the-btrfs-file-system-as-implemented-by-synology-so-fragile/

I have little experience with Btrfs, but what I do know is that Synology's offerings shouldn't be compared to TrueNAS. I also can't comment on how Synology implements btrfs or their RAID solution. Maybe in the past it has been a poor design? I don't know? But one thing to keep in mind is that Synology's products are aimed at the consumer and prosumer market and that includes their RackStation products (which I have used before). They are designed to be an easy to use, out-of-the-box solution for individuals and businesses that don't want to mess around with complex configurations, permissions etc... TrueNAS on the other hand is an enterprise solution, but one that can be used in the home in a scaled down version.

iXsystems have been around for a long time and along with a FreeNAS/TrueNAS have a longstanding proven track record, as does ZFS. People aren't necessarily mistaken about ECC, there are use cases for it, but unless you have money to burn, that use case isn't in the home (or even small to medium office environments). None of the TrueNAS guides are wrong either. If you have ECC, you'll get the added benefit from it, if not, it's not a deal-breaker. My advice is, don't compare one product to a entirely different product, it'll lead you to making false assumptions.

I personally use a UPS so that I don't have to login to the server after short power outages and re-input the volume encryption keys. Graceful shutdowns, whilst always recommended, are really just an afterthought for me. I sleep pretty well at night knowing my data is safe.