ZFS scrub repaired 0B

[RoGeorge]

There is a default ZFS scrub scheduled once a months or so, though I never ever seen any errors reported, let alone fixed.  For now it's a single ZFS disk (HDD), and the PC does not have ECC RAM.  There is another backup NAS drive anyway, to backup the most important directories.

The HDD is an 8TB WD, has some years but only light usage, since it's a model for small-enterprise, and was used in a normal workstation, with spin always on to reduce the head landings.

A ZFS scrub takes many hours, almost a day, during which the HDD is visible slowed down.  I'm thinking to reduce the scrub to once or twice a year.  Periodic scrubbing never ever found any errors since I've start using ZFS on this desktop PC, years ago.

Any advice about the periodic ZFS scrubbing?  Should I turn it off?  Did ZFS scrubbing ever detected or repaired any errors for you?  Do you let it scrub monthly?

[magic]

BTRFS scrub found an error I deliberately made by editing the raw disk. So I know it works

Otherwise, disks store data with ECC (both spinners and solid state) and there are checksums and/or correction codes on SATA and PCIe links and AFAIK even on internal CPU data buses, so probability and real world frequency of silent corruption is low. This is mainly a protection from bugs in storage hardware. I started using BTRFS after encountering a second case of data corruption involving a (second different) USB-SATA bridge. Some people worry about RAID controllers, there are horror stories about those writing sectors to wrong component disks and stuff like that.

It doesn't protect you from bit flips in RAM and software bugs striking before the checksum is computer or after it is verified.

I have an old laptop where I saw quite frequent data corruption, but I never found the reason. It didn't use a checksumming filesystem and the disk it used at the time kicked the bucket, and it was IDE so I retired that hardware for lack of good replacements nowadays.

[DiTBho]

I have an old laptop where I saw quite frequent data corruption, but I never found the reason

Hey? I deliberately paid 60 euro (SH included) for a pair of hard disks that manifest exactly that "feature".
Since they manifest frequent data corruption at the low level, the seller had listed them on eBay as  "dead drives for parts-only", but it's not a defect but a great feature!!! when you are developing your own filesystem/RAID1 solution/etc and you need real-life testing patterns.

So ... I envy you for that hardware 

[borjam]

There is a default ZFS scrub scheduled once a months or so, though I never ever seen any errors reported, let alone fixed.  For now it's a single ZFS disk (HDD), and the PC does not have ECC RAM.  There is another backup NAS drive anyway, to backup the most important directories.

Especially in large stores, random error can cause silent data corruption. One day you want to read a file written long ago and it has some glitches.

ZFS uses a very strong checksum protection so that it can detect those glitches much better than the typical CRC used by hard drives. Moreover, it can also detect problems caused by cabling, faulty controllers...

A periodic scrub is always a good idea because it can serve as an early warning. If you don't try to read a file for years, who knows what will happen the day you need it?

As for repairing, ZFS of course needs redundancy. Without redundancy it can certainly detect errors but it won't be able to fix them.

Any advice about the periodic ZFS scrubbing?  Should I turn it off?  Did ZFS scrubbing ever detected or repaired any errors for you?  Do you let it scrub monthly?

I would not turn off ZFS scrubbing.

I would add redundancy.

And yes, ZFS has alerted me several times in the last ten years that a disk was deteriorating. It repaired the errors and I could replace a faulty drive before it was too late.

I even saw a ZFS system with a faulty SAS controller that suffered so much corruption even ordinary reads would report reading errors on "zpool status", yet no single bit of data was damaged and I could solve it by replacing the controller. Again, there was redundancy!

A properly configured ZFS system can help data survive even power supply glitches, faulty cabling or interference. Depending on the amount of errors of course.

The TrueNAS people have an excellent set of hardware recommendations developed over the years thanks to sometimes painful lessons.

[borjam]

I have an old laptop where I saw quite frequent data corruption, but I never found the reason

Hey? I deliberately paid 60 euro (SH included) for a pair of hard disks that manifest exactly that "feature".
Since they manifest frequent data corruption at the low level, the seller had listed them on eBay as  "dead drives for parts-only", but it's not a defect but a great feature!!! when you are developing your own filesystem/RAID1 solution/etc and you need real-life testing patterns.

So ... I envy you for that hardware 

Pity, that defective SAS card I mentioned in my previous post was dumped for recycling. Otherwise it would be yours!

[magic]

I have an old laptop where I saw quite frequent data corruption, but I never found the reason

Hey? I deliberately paid 60 euro (SH included) for a pair of hard disks that manifest exactly that "feature".

Silent corruption or failure to read random sectors?
The former I have never seen, let alone two such disks.
The latter is plentiful and cheap.

[rdl]

I know my FreeNAS install originally was scrubbing very frequently, probably once a week. I changed it to monthly. I've had a couple of errors reported over the years but they were too cryptic for me to understand. On the FreeNAS forums I didn't get any clear explanations. One time I was told I should replace a disk and I remember one error sounded like maybe a cable issue. At any rate none of the errors ever repeated and the disk is apparently still working.

[DiTBho]

Silent corruption or failure to read random sectors?

Both, and - bonus - * both * in the same pair of devices, so it's a known local property!!! 

Each disk silently corrupts and - better still - randomly IO fails; the motor "sounds" ok as it spins without too much ultrasonic noise, so the bearings "sound" in decent state, but - my speculation - the thin layer of protective grease must have worn off in several places r/w heads must have already crashed onto the plates, and survived as "walking deads", and that's great because writing some data and hoping that you will read * the same data * again is really a challenge there.

[DiTBho]

cable issue

yup, from SATA2 and up the cables and connectors must be of high quality, and properly secured in a way they cannot move too much.

This is also a problem for UPSs, therefore not only for data integrity but also for power lines. Yesterday I saw a 500Watt UPS burn in a fire and the reason was a poor Molex connector that did not make complete metal contact, and, worse still not properly secured, so it started making the current density not evenly distributed but concentrated in one point of the connector that overheated beyond specification, ... and ... at the point the PCB cooked of bad smoke, so they said, and something started the fire.

[SiliconWizard]

I had a SATA connector (the power connector) literally break on a hard drive due to heat (this was a 10000 RPM HDD running pretty hot). The drive was RMA'd though and replaced for free.
The data connectors normally have a retaining mechanism (use SATA cables with this), but not the power connectors, which are frankly crap.

[gnif]

Having used ZFS for over 10 years both on home recycled hardware and enterprise configurations, I had often wondered the same. Experience has taught me that scrubbing is crucial for several reasons:

1) The obvious, it will detect minor silent corruptions and repair them.
2) Even if you have ECC and SAS drives, there is still a guaranteed undetectable error rate.
3) The scrub rather heavily exercises all the disks in the array, if a disk is about to fail, this is when its most likely to occur. As scrubs are usually scheduled during periods of low activity, it can be handy to have it fail when it's most agreeable to have a disk go offline (read retry can cause quite a long stall depending on the mode of failure, causing the entire OS to stall on FS accesses)

I schedule a weekly scrub, and perform a scrub on every unsafe shutdown (ie, power outages, failures, etc). I have lost far too much data due to silent errors in the past and wont be bitten by this again if I can at all avoid it.

My recommendations if you need a reliable storage solution are:

1) Use ECC RAM. This is critical for ZFS, many people argue against this today, but with the idiotic overclocking of binned memory being the norm, your rate of error is stupidly high.
2) If it's a home NAS, don't use SATA disks, use SAS. For a home/budget build, I'd use used SAS drives from eBay any day over a brand new SATA disk. If you're worried about the used nature note that enterprise SAS drives have far higher MTBF rates, and as nobody wants a used SAS disk for a enterprise server, can be had very cheap, so buy extras for additional redundancy.
3) Don't use ZFS on Linux... I know it's much better today then it was in the past (I lost a lot of data years ago due to a silent bug in the Linux/BSD compatibility shim for ZFS), it's still IMO not suitable for prime time use. Go for BSD and use ZFS on the platform it was developed for.

[DiTBho]

I lost a lot of data years ago due to a silent bug in the Linux/BSD compatibility shim for ZFS

yup, ZFS is still "rolling" on Linux.
What were you using when yoy lost data? ZFS on the fuse or experimental ZFS kmod?

[gnif]

I lost a lot of data years ago due to a silent bug in the Linux/BSD compatibility shim for ZFS

yup, ZFS is still "rolling" on Linux.
What were you using when yoy lost data? ZFS on the fuse or experimental ZFS kmod?

It was the kernel module, in my experience FUSE file-systems are not suitable for most of my workloads (many thousands of small files).

[borjam]

2) If it's a home NAS, don't use SATA disks, use SAS. For a home/budget build, I'd use used SAS drives from eBay any day over a brand new SATA disk. If you're worried about the used nature note that enterprise SAS drives have far higher MTBF rates, and as nobody wants a used SAS disk for a enterprise server, can be had very cheap, so buy extras for additional redundancy.

I think the key is not SATA versus SAS, but how you connect multiple disks, which makes a very important difference.

If you use some sort of backplane with a single HBA (controller), certainly you need SAS. SAS backplanes can be roughly compared to a network switch, while SATA "multipliers" are more like an Ethernet hub. This means, electrical problems in the interface of one of the SATA disks can corrupt communications with the rest of the units.

However: If, using SATA, you have an AHCI controller per disk instead of a "port multiplier", it should be perfectly reliable unless you get crappy hardware.

Another important recommendation is, avoid "intelligent RAID" controllers. No matter what the manufacturer says, ZFS does a much better job. So many people makes the mistake of defining a RAID 5 logical volume on the controller and creating a single device ZFS pool on top...

As I said, check the TrueNAS forum for hardware recommendations.

3) Don't use ZFS on Linux... I know it's much better today then it was in the past (I lost a lot of data years ago due to a silent bug in the Linux/BSD compatibility shim for ZFS), it's still IMO not suitable for prime time use. Go for BSD and use ZFS on the platform it was developed for.

I prefer FreeBSD myself (been using ZFS on FreeBSD since 2006 or earlier, when it was just released as a beta) but I think ZFS on Linux has matured a lot. Currently even the TrueNAS people have released a Linux based version and I think it works.

I still recommend FreeBSD, but if you want Docker and other Linux goodies FreeBSD will lack some features.

[DiTBho]

If, using SATA, you have an AHCI controller per disk instead of a "port multiplier", it should be perfectly reliable unless you get crappy hardware.

like SiliconImage chips 

[rhb]

I have run ZFS for 20 or so years.  Most of my systems are single parity RAID 5, with a couple of double parity.

I have had at least 3 drive failures. I've  never lost data, though I often can't find it because I can't remember where it is and have to run find(1) on multiple systems.  I need to consolidate it all on one system,  but got so burned out spending a full month doing battle with the installer to provide a 4 disk bootable system by partitioning the disk and running a 4 way mirror for rpool and double parity RAID 5 on the bulk of the disk that I still have not migrated to using it.  The system is now so out of date I need to update first.

Scrubs are important because ZFS can't detect errors in sectors it has not read.  Try setting the scrub to very low priority via  nice(1).

Have Fun!
Reg

[gnif]

2) If it's a home NAS, don't use SATA disks, use SAS. For a home/budget build, I'd use used SAS drives from eBay any day over a brand new SATA disk. If you're worried about the used nature note that enterprise SAS drives have far higher MTBF rates, and as nobody wants a used SAS disk for a enterprise server, can be had very cheap, so buy extras for additional redundancy.

I think the key is not SATA versus SAS, but how you connect multiple disks, which makes a very important difference.

No, SATA drives are not rated for the kind of usage that SAS is unless it's enterprise SATA, but if you're paying the money for that, why not just get SAS and all the added benefits that come with it?

If you read 10TB of data from Consumer SATA drives, the probability of encountering a read error approaches 100% (virtually guaranteed to get an unreadable sector resulting in a failed drive).
...
SATA/SAS Nearline Enterprise drives improve the hard error rate by a factor of 10 but they are still 10 times more likely to encounter a hard read error (inability to read a sector) relative to an Enterprise SAS drive. This is equivalent to reading roughly 111 TB of data (0.11 PB).
...
On the other hand, using Enterprise SAS drives, a bit more data can read before encountering a read error. For Enterprise SAS drives about 1.1 PB of data can be read before approaching a 100% probability of hitting an unreadable sector (hard error).

Ref: https://www.enterprisestorageforum.com/networking/sas-vs-sata/

As for SAS vs SATA communication reliability, read on:

The SATA channel (and the IB channel) has an SDC of about 10E17. If you transfer data at 0.5 GiB/s you will likely encounter 1.4 SDC’s in a year. In the case of faster storage with a transfer rate of 10 GiB/s you are likely to encounter 27.1 SDC’s in a year (one every 2 weeks). For very high-speed storage systems that use a SATA channel with a transfer data rate of about 1 TiB/s, you could encounter 2,708 SDC’s (one every 3.2 hours).

On the other hand, SAS channels have a SDC rate of about 10E21. Running at a speed of 10 GiB/s you probably won’t hit any SDC’s in a year. Running at 1 TiB/s you are likely to have 0.3 SDC’s in a year.

So assuming the drive never has a fault, by using SATA your probability of silent data corruption is orders of magnitude worse then SAS, even with a reliable backplane/cables, whatever.

Edit: Add to the fact that here (Aus) a consumer SATA 8TB disk costs upwards of $250, and a low hours used enterprise 12TB SAS drive can be obtained for about $200, it's kind of a no brainier if you're building a personal use NAS with redundancy. If you want smaller disks, 4TB new old stock/cold spares can be obtained for about $50 each.

[radar_macgyver]

I have a few arrays that use SAS and a few that use NL-SATA ('enterprise grade'). All use ZFS on Linux, capacities ranging from 20 - 50 TB. Based on the recommended 'best practices', I have the SATA arrays doing a scrub once a week, and the SAS ones once a month. I do see the drives on the SATA arrays failing more often, but it's unclear if the reason is that they're being scrubbed more often (and hence have a higher average workload). The arrays have been up for many years, with at least one going back to 2012.

As for why one would use NL-SATA vs SAS - no good reason in my case. All my NL-SATA arrays are older commercial units (with a SATA backplane) that were repurposed to run ZFS on Linux as backup arrays (I run periodic replication jobs from the main arrays). Since these are backup arrays, I care a little less about data loss so after some of the NL-SATA drives started to go bad, I replaced them with the "NAS grade" SATA (Seagate Iron Wolf Pro) as they are considerably cheaper than their NL-SATA or SAS equivalents (Seagate Exos). A former colleague had tried using the workstation grade (Seagate Barracuda) and that did take down an array fairly quick (disk fail, followed by a failure during rebuild). The SATA arrays use independent links per drive (no port multipliers).

I dunno about used disks, though. @gnif do you have a preferred vendor? Amazon listings are filled with reviews indicating that disks advertised as NOS showed up with SMART data indicating thousands of power-on hours. I'm also aware that some vendors' SMART stats 'roll over' and very old drives sometimes indicate low hours.

To @RoGeorge 's question, ZFS has emailed me on occasion that it found and corrected errors. The sequence of events is usually that I get such an error on a drive, then a couple more of them on the same drive, then either the drive fails completely, or I get a SMART warning of impending failure and I swap out the disk with a new one. I suspect that's because the drive firmware ECC is correcting a bunch of read errors, and over time there are one or two that have just the right statistics to pass through the ECC without being flagged as an error.

[ejeffrey]

BTRFS scrub found an error I deliberately made by editing the raw disk. So I know it works

Otherwise, disks store data with ECC (both spinners and solid state) and there are checksums and/or correction codes on SATA and PCIe links and AFAIK even on internal CPU data buses, so probability and real world frequency of silent corruption is low. This is mainly a protection from bugs in storage hardware. I started using BTRFS after encountering a second case of data corruption involving a (second different) USB-SATA bridge. Some people worry about RAID controllers, there are horror stories about those writing sectors to wrong component disks and stuff like that.

People think of "bit rot" as degradation of the magnetic storage medium.  Due to the extensive use of reed-solomon error correction codes, I think it is extremely unlikely that ever happens silently.  A lot of people claim it is, but the data they show doesn't support that.  Maybe people like backblaze or google have internal data that is better, but I doubt they will ever see that.

Specifically, what I think has never been observed is: 1) Write data to a drive.  2) read it back successfully, long after the data is gone from the controller's cache, 3) wait, 4) read it back and get silent corruption, 5) read it back again, get the same silent corruption.  With modern controller managed flash wear leveling or controller managed SMR, this isn't necessarily even possible to verify as the controller can move the data around without telling you.  But I'd be very interested and surprised to hear from someone that they saw zfs-scrub detect silent corruption in a block which had previously been scrubbed and not written to since.

I strongly suspect that silent data corruption instead happens in-flight during the write or read.  While the PCIe and SATA links have checksums, and the cache memory in the HDD and CPU may have ECC, each controller along the way that decodes and re-encodes is a point of corruption that won't be detected by those checksums, and the incorrect data will then be encoded for the next step and from there on be blessed as "good".  By attaching a checksum in userspace, you have an end-to-end integrity guarantee against errors in the PCIe controller, SATA controller, disk controller, and the reverse trip through all of those.

Of course it can't catch every possible error, in particular it won't catch memory errors that happen before or during checksum computation.    But it at least makes the storage system round trip protected.

[DiTBho]

Yup, I do think so, silent data corruption has a higher probability to happen in-flight during the write or read.

[DiTBho]

People think of "bit rot" as degradation of the magnetic storage medium.  Due to the extensive use of reed-solomon error correction codes.

In my case, it was simpler to find a few HDDs with an a lot of knwon "bit rot" (known from SMART, badblocks, etc), than trying to model in software (testbench) the "in-flight data corruption" during data write or read.

Sure, I somehow modeled both, then I was lucky that the two HDDs I found manifest both the synthoms.

[gnif]

I dunno about used disks, though. @gnif do you have a preferred vendor? Amazon listings are filled with reviews indicating that disks advertised as NOS showed up with SMART data indicating thousands of power-on hours. I'm also aware that some vendors' SMART stats 'roll over' and very old drives sometimes indicate low hours.

I generally do not use Amazon at all. If I need a set of disks (10 or more) I usually will contact one of the local data centres that house government servers as they often have a surplus of disks that are now considered obsolete for government applications, but were bought a spares for servers that they were managing. Often they are cold spares and have < 100 hours on them.

For single disks I will just use eBay, but I will contact the seller first and check if the disk is low hours, or new old stock. I have had a few disks that were high hours but I was able to obtain full refunds for all of them except one that was replaced with a still sealed new old stock drive. As the disks are low value the sellers didn't want the old disks back, so I ended up with some high mileage spares for non-important tasks.

[macboy]

...
3) Don't use ZFS on Linux... I know it's much better today then it was in the past (I lost a lot of data years ago due to a silent bug in the Linux/BSD compatibility shim for ZFS), it's still IMO not suitable for prime time use. Go for BSD and use ZFS on the platform it was developed for.

So... Solaris then, rather than BSD or anything else  . ZFS was not developed for BSD nor on BSD, but for Solaris and on Solaris. BSD has had ZFS for many more years than Linux only because its license was deemed compatible with the CDDL used to open source the ZFS and other Solaris code. It also may have been easier to integrate into BSD kernel due to similarities with Solaris kernel, but I don't know anything about that. The GPL of Linux was deemed to be so very incompatible that no in-kernel implementation was possible. Even when it did appear as a kmod, it was only through a compatibility layer (SPL - Solaris Portability Layer). SPL just adds more complexity and more opportunity for bugs. Even now the license remains a grey area. I've tried ZFS on Linux and I don't like it. It does not instill confidence. I understand that ZFS on BSD is much better.

I've used ZFS on Solaris (and Solaris-like) systems for over 15 years. I've never lost a byte of data, even with dodgy SATA cables and both gradual and sudden hard drive failures. I've twice expanded the pool size by replacing disks with larger ones, and I've replicated the pool to a new set of disks (zfs send/receive). But it is fundamentally the same pool that I've used for 15 years. Remarkable.

I have had data repaired by a scrub, or just by a read operation. Occasional scrub is probably a good idea, but as mentioned, it stresses the disks. I now use NAS grade SATA disks (24/7 rated and 5 year warranty) but I still don't like undue stress. A scrub is equivalent to reading all allocated blocks of storage. If a checksum doesn't match, the software tries to recover using any redundancy available (mirrors, parity disks, or copies on the same disk if zfs copies property is >1). Scrub is more useful the more disks in the pool, since it will hopefully catch and repair data errors before the amount of bad data exceeds the amount of redundant data. Other than the slight delay incurred by the repair, catching errors during a normal read is fully equivalent to catching them during a scrub.

[DiTBho]

I've used ZFS on Solaris (and Solaris-like) systems for over 15 years. I've never lost a byte of data, even with dodgy SATA cables and both gradual and sudden hard drive failures.

well, talking about that ... I could say

• I have been using my own made personal multi node NAS since 2008
• embedded uc-libc-profiled, based on GNU/Linux stuff
• running on experimental kernles (without any official support, and easily prone to crash/panic) for PowerPC 4xx
• with a simple RAID1, 2 Hdds in mirroring on XFS, + userspace checksuming and automate-rcopy

and I have never lost a single byte with - what people in this forum think are the wrorst hdds ever made - the infame Seagate Barracuda ST3500418AS 


so I don't think you can say... "ZFS on Solaris is better than on FreeBSD"; the only thing that can be said is: on Linux there is much less attention because ZFS (talking about the kmod version of ZFS, and not the FUSE one) is not part of the kernel, therefore it has much lower priority, at least as regards system integration tests, which NOT they are made by the Linux team, but by third parties, often by those who manage the distributions!