Low-cost SATA SSDs for ZFS

[olkipukki]

I'm looking to buy 4 (four) SATA SSDs to be used for independent FreeBSD ZFS RAIDZ-2 pool with max storage up to 0.5TB (reality ~200GB)
Nothing too serious, considering consumer grade drivers, so max 250GB per drive should be okay 

Any recommendations what (not) to buy?

[wraper]

Don't buy DRAMless models. Also avoid QLC NAND.

[olkipukki]

Any particular model you have in mind?

For example, WDS240G2G0A (WD Green) - not clue if it fail into DRAMless and QLC categories

[Haenk]

Keep in mind the standard SSDs have a rather low lifespan when used in a RAID (i.e. constantly reading/writing), there is a reason for a MTBF based on data amount written.
We have one RAID 50 setup running for a couple of years now, using Crucial MX500 SSDs - those are very slow now, in fact, incredible slow. So you should replace them once every couple of years, at least.
And yes, avoid QLC as the plague, TLC+DRAM Cache is the way to go, if you want it cheap. Don't fall for those very cheap Amazon/ebay/Aliexpress SSDs from dubious chinese sources. Those are made from B-grade NAND and therefor not very reliable.

[David Hess]

Don't buy DRAMless models.  Also avoid QLC NAND.

I agree with both.  Besides the performance difference, models with DRAM seem to suffer from less write amplification and the added DRAM is not expensive.  QLC NAND has much worse endurance than TLC NAND.  I do not know how other people are using their drives but at the rate I am going, I expect to burn through the endurance of my TLC drives in less than 5 years.

Keep in mind the standard SSDs have a rather low lifespan when used in a RAID (i.e. constantly reading/writing), there is a reason for a MTBF based on data amount written.

RAID does not have to mean constant writing.  It is often used in standby and bulk storage applications with a majority of accesses being reads.

We have one RAID 50 setup running for a couple of years now, using Crucial MX500 SSDs - those are very slow now, in fact, incredible slow. So you should replace them once every couple of years, at least.

What does Crucial's Storage Executive utility say about the health of the drives?  Are they just worn out?

The Crucial MX500 series is my preferred drive right now but you can sometimes find deals on enterprise drives like the Intel D3-S4510.

[wraper]

We have one RAID 50 setup running for a couple of years now, using Crucial MX500 SSDs - those are very slow now, in fact, incredible slow. So you should replace them once every couple of years, at least.

What does Crucial's Storage Executive utility say about the health of the drives?  Are they just worn out?

I suspect simply lack of TRIM and drives not having known free space which was pre-erased, thus now they need to erase sectors before any write happens. Which slows them down dramatically. Can be cured with secure erase within seconds. Also some old drives may become slow during reading data written a long time ago as their cells discharge, and drive has harder time to determine actual data written. Can be cured by rewriting the data. Upditing the firmware may help as well, as newer firmware may have added functionality to refresh old data.

[David Hess]

I suspect simply lack of TRIM and drives not having known free space which was pre-erased, thus now they need to erase sectors before any write happens. Which slows them down dramatically. Can be cured with secure erase within seconds. Also some old drives may become slow during reading data written a long time ago as their cells discharge, and drive has harder time to determine actual data written. Can be cured by rewriting the data. Upditing the firmware may help as well, as newer firmware may have added functionality to refresh old data.

I have not encountered either problem yet on my BX500s or MX500s, even the largely full ones, although they all take advantage of TRIM.

The MX500 drives support Over-Provisioning accessed through the Crucial utility that I mentioned previously but I do not think it works with RAID.  It seems that Flexcap which does the same thing and would work with RAID is only supported on Micron branded drives and not Crucial ones, so maybe the MICRON 1300 series would be more suitable.

I have not yet had any reason to use RAID with SSDs.

[Daixiwen]

AFAIK when using ZFS RAID arrays you can turn on an autotrim option that will use the trim instruction on free spaces.

[Kjelt]

The most important question has not been asked, how many GB per day will you write on average ?
If you mainly read and occasionally write that is a total different situation than when you use ssd's as write cache in a small business.

[ejeffrey]

I wouldn't use RAIDZ(2) for 200-500 GB of space.  I would either use a single 500 GB-1TB drive or if I really wanted the availability of online redundancy I would use two drives in a ZFS-mirror.  I would only consider any form of parity raid when I needed more storage than I can get in a single drive at reasonable $/TB.  This advice isn't true for 100% of workloads, but the fact that you say "nothing too serious, considering consumer grade drives" makes me think that the added complexity, cost, and overhead of any type of RAID is possibly not warranted.

As for SSD choices, mostly what people say here: a non-QLC drive from a reputable manufacturer is what you want.  Beyond that it depends on your workload. For light to medium duty read-heavy workload, a TLC drive from Samsung or Crucial is probably fine.  For heavier use with lots of writes, MLC drives from Intel or Samsung or even Intel Optane SSDs might be a better choice if you can afford it.  Enterprise targeted drives will have higher write endurance and more reliability under heavy use but may fare no better than good consumer drives under light use.

QLC doesn't make much sense for any application. The capacity/$ advantage over TLC is marginal especially with the size SLC caches they use to get adequate performance.  The reliability and performance are much worse even than TLC.

[wraper]

I wouldn't use RAIDZ(2) for 200-500 GB of space.  I would either use a single 500 GB-1TB drive or if I really wanted the availability of online redundancy I would use two drives in a ZFS-mirror.  I would only consider any form of parity raid when I needed more storage than I can get in a single drive at reasonable $/TB.  This advice isn't true for 100% of workloads, but the fact that you say "nothing too serious, considering consumer grade drives" makes me think that the added complexity, cost, and overhead of any type of RAID is possibly not warranted.

Yeah, especially considering that 256GB drives cost more per GB and usually have worse performance. And more drives also means more points of failure.

[ejeffrey]

Also they don't change product names during iteration. You have WD Green for 10+ years yet the technology iterates, making pin pointing what exactly a unit is very hard. Same goes for Sandisk. Anyone tell me what "Extreme Pro" means in 2020, and what it meant in 2019?

Unfortunately Samsung is now doing that too.  It used to be that the Evo and Pro lines were high performance desktop/workstation drives and the difference was TLC vs MLC.  But the new 980 Pro is TLC just like the 970 Evo...

[olkipukki]

I wouldn't use RAIDZ(2) for 200-500 GB of space.  I would either use a single 500 GB-1TB drive or if I really wanted the availability of online redundancy I would use two drives in a ZFS-mirror.  I would only consider any form of parity raid when I needed more storage than I can get in a single drive at reasonable $/TB.

Thought that four drives will buy some time if some drive will fail (max 2 per RAIDZ2 spec) and don't require to deal with an issue immediately.
The mirroring still on cards, but I threw away the idea to have just one drive.

[olkipukki]

But the new 980 Pro is TLC just like the 970 Evo...

Really?* 

https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/980pro/

STORAGE MEMORY
Samsung V-NAND 3-bit MLC

* still awaiting 980 Pro 1TB availability to utilize PCIe4 on a desktop and never considered that they will downgrade Pro series

[olkipukki]

The most important question has not been asked, how many GB per day will you write on average ?
If you mainly read and occasionally write that is a total different situation than when you use ssd's as write cache in a small business.

from zero write to max 50GB daily, but not more than approx 200GB weekly.

[wraper]

Really?* 

STORAGE MEMORY
Samsung V-NAND 3-bit MLC

* still awaiting 980 Pro 1TB availability to utilize PCIe4 on a desktop and never considered that they will downgrade Pro series

MLC, if you are blind. Samsung calls everything as MLC because technically it is.

[nightfire]

I agree with going for used Intel or Samsung Server/Enterprise SSDs- they are very good quality.

On the consumer end, Crucial MX series deliver good quality performance and have some techniques embedded like thermal sensor ans some tiny capacitors to keep at least the filesystem intact in case of power loss...

At Work, I have some legions of Crucial M500 up to MX500 SATA drives, and they are running rock-solid and without big issues.
If real server-grade stuff is needed, I would look out for some old Intel S3500, if there are no big amounts of data to write.

[tszaboo]

I was considering doing this, as my age old SSD dies on me this summer.
But it is easier to do regular backups to the spinning rust. PCIe-4 is also a lot faster, and it is hardly possible to do raid on that. Unless you buy some of thes 16x to SSD adapters. Extra points of failure.

[Kjelt]

The most important question has not been asked, how many GB per day will you write on average ?
If you mainly read and occasionally write that is a total different situation than when you use ssd's as write cache in a small business.

from zero write to max 50GB daily, but not more than approx 200GB weekly.

Ok so about 70TB/year means you need a big standard ssd or an industrial ssd.
Check the datasheets of your ssd of interest and make sure the TBW within warranty matches your needed lifespan.
For example a 256GB Samsung 270EVO SSD has a warranty upto 150TB TBW or 5 years which is the first reached. In your case the warranty ends after two years and your ssd might fail within 3 to 4 years.
However the same ssd in a 1TB version has a 600TB TBW warranty which in your scenario the warranty of five years is the girst reached.

Edit: mistakenly calculated your weekly usage as dayly, in any case you can calculate it yourself and make a decision.

[ejeffrey]

But the new 980 Pro is TLC just like the 970 Evo...

Really?* 

https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/980pro/

STORAGE MEMORY
Samsung V-NAND 3-bit MLC

* still awaiting 980 Pro 1TB availability to utilize PCIe4 on a desktop and never considered that they will downgrade Pro series

3-bit MLC is a term Samsung uses for TLC, everyone else in the industry only calls 2-bit cells "MLC".  Samsung is 100% right here BTW, as "TLC" has 8 levels, not three.  But still: the 970 Pro uses 2-bit cells, the 980 pro uses 3-bit cells just like the 970 Evo.

[David Hess]

Unfortunately Samsung is now doing that too.  It used to be that the Evo and Pro lines were high performance desktop/workstation drives and the difference was TLC vs MLC.  But the new 980 Pro is TLC just like the 970 Evo...

At least you know it is a different product, 980 is not 970 after all.

With WD and SD, you need to dig into the fine prints (actual SKU) to tell what exactly it is.

The same applies to Samsung.  They helpfully point you to their application note about power loss protection on SSDs, and imply that all of their SSDs have it, while if you check the actual datasheets, only specific ones do.  At best, Samsung is deliberately misleading on the subject.

[SarahSimm]

Samsung is good, but it's kinda pricy
Crucial is decent and affordable

Also, why not buying a M2 NVME SSD? They have a higher speed

[olkipukki]

3-bit MLC is a term Samsung uses for TLC, everyone else in the industry only calls 2-bit cells "MLC".  Samsung is 100% right here BTW, as "TLC" has 8 levels, not three.  But still: the 970 Pro uses 2-bit cells, the 980 pro uses 3-bit cells just like the 970 Evo.

Ah, 3-bit MLC (aka TLC) 
Is it reason why 970 Pro cost more than 980 Pro?!

If I'm not mistaken, all Samsung Enterprise SSDs (except SM833 SM883) are TLC, right?

[olkipukki]

Is there way to verify Samsung drives by S/N or any text available on a label?
Something like if exist,region etc.

[nightfire]

An ordinary mainboard provides usually not enough M-2 slots to go for real redundancy like the OP specified. In that price region is a software volume manager coupled with some affordable, reasonable quality SSDs a very good way to go. And 70TB/Year is also no big deal when it comes to bandwidth. If there are no additional needs like low (and guaranteed) latency, also usually no Enterprise SSD with optimized firmware are needed.