Come diss my NAS/Server idea!

[paulca]

I have a 24/7 busy box.  I want my normal use desktops/media centres to be small, fast, low power SSD'd devices that I can switch off and walk away from.  But I need somethings to be available to all of them.

So I downgraded my previous gaming machine by removing it's video card and youtubing "Worst ever PCIe GPU" and picked a winner.  NVidia 720

But... the machine is from the previous generation before power efficiency was a big seller.  It has a AMD Piledriver FX in it.  Even idle, under volted and under clocked the pulls 90W.  My entire house electric over night is only 180W.... half of which is that server.  Granted the 6 HDDs in it don't help, but they do spin down... or they are supposed to, but I think something keeps polling them, so they don't.  It has to go.

Oh and it coil whines and fan whines constantly.

So I need a machine which is.
Cheap
Low power
Quiet
Robust
Vaguely capable of saturating 1Gb Ethernet file transfer.
Capable of handling 3 or 4 HDDs and an SSD or two.
Running a docker host for a few dozen services.

Solution 1:  Just buy a NAS.
Rejection:  Not cheap, running a docker host could get complex, especially if it's not AMD64 arch.  I don't really want a micro-managed bespoke OS.

Solution 2:  Buy an old server.
Rejection:  Large.  Probably noisy.  The generation for my budget would probably not be power efficient either.  Can't be absolutely sure what I'm going to get S/H.

Solution 3:  Use a little PI that could.
Rejection.... please.  Just No.

Solution 4:  Buy a beefy mini-pc and use external USB drives.
hmm...  close, but multiple USB drives with power adapters and usb hubs is messy and not that robust.  Mini-PC architectures are often odd.  Good ones are not cheap.

So, what I'm going with is....

A refurbished corporate Dell small form factor PC because they are reliable, robust and the "bang for your buck" on the second hand market is amazing at the minute.  2nd gen i5 3200, 8Gb RAM, 256Gb SSD will cost about £120.

A PCIe x4 USB3.1 card.  Providing theoretical 10GB/s bandwidth.

A 4 or 5 bay USB3.1 drive enclosure.

(Might downgrade to USB 3.0 for cost)

Two small boxes ... with fans which can be tamed or just removed.  It will take a linux install easily, run docker with enough memory, etc.

Power consumption of these machines idle is around 19W.  Add the disk array and call it 25W. 

What could go wrong?  Right?

I do need to put the 6HDDs on a diet though.  Half of the older ones are legacy Linux and Windows installs.  I think there is even a stray Windows XP partition on one of them.

Not sure I need RAID or if it is even wise over USB.  I could look at 3 1Tb in RAID or similar with part reliability part performance.  And a 6Tb drive for bulk stuff and media.

The question is... do I try and set that RAID up native on the Linux or hand those drives over to a NAS controller running on Linux?... is that even available?

[newbrain]

Eh, tough choice.
I agree that an RPi is pointless for this purpose.

At the same time, I don't very much like using USB for mass storage, even 3.1 or faster especially if it's only one link serving a number of devices, and having two boxes is in any case cumbersome.

How many and how fast network interfaces are you considering?

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

My solution has been an HPE Microserver Gen 8, the base model with a Cleleron 1610T (but a Xeon E3-1220lv2 is now crawling from China to Sweden...).

• 4×2 TB HDD (2 pairs of different WD models)
• 1×256 GB SSD
• boot device is an internal 32 GB USB key (there's an internal USB 3.0 socket and a Micro-SD one)
• 4+8 GB ECC RAM

FreeNAS and of course ZFS, volumes are mirrored and striped on the HDDs, the SSD is used for less important or temporary stuff.
Administration is reasonably simple, though it took some attempts to have it join the AD Domain.

The server comes with two 1 GB/s NICs - three if you count the one for the lights out, iLO, management.
They can easily be saturated towards my main PC (two NICs there, too), using the native SMB multichannel in Windows (no setup whatsoever needed) for about 200 MB/s of total transfer rate.

There's an internal PCIe×16 low profile expansion slot.

Epochal uptimes, never a glitch and it's relatively silent even in a living room.

Only power and two Ethernet cables are connected, as with iLO there's no need for a physical console.
IIRC correctly, power consumption is about 30W at idle (the coming Xeon sports half the TDP, so I hope to decrease even further), but I don't feel like turning it off to measure it...

Very good build quality and maintainability, small enough (about 25×25×25 cm) to sit on a bookshelf.

Can be found on Ebay for not much money, even decently equipped.

In the pictures a file copy and the HPE server, below my open-air Hyper-V server (picture is terrible)

[dnwheeler]

I suggest you look into UnRaid - it will give you a parity-protected drive pool without the limitations of RAID (you can mix-and-match drive sizes, and even in a worst-case scenario, all of the drives are readable in a standard Linux system). It also offers great support for hosting Docker containers and VMs. It is Linux based, but has a nice Web-based management UI (or you can dive in and customize whatever you want). It also boots directly from a USB thumb drive so you don't need to dedicate any hard drive space for the OS.

[PlainName]

Can be found on Ebay for not much money, even decently equipped.

Not any time I've been looking for one! I'd happily take one off your hands for £100.

[Jr460]

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

Maybe you could explain why EMC, HDS, IBM, LSI Logic, HP are all wrong by making and using hardware RAID controllers.

[paulca]

Can be found on Ebay for not much money, even decently equipped.

Not any time I've been looking for one! I'd happily take one off your hands for £100.

Think I bought the last one.  This is a really good bargain, if it's not too good to be true.  Gen3 i5 3470, 8Gb DDR3, 256Gb HDD, not SSD, but I'll be depositing that at my local recycling centre anyway.

https://www.amazon.co.uk/gp/product/B01E97IVD2/

EDIT:  Oops, you meant the HPE Microserver Gen 8.  Yes.  Looking like £300+ on Amazon.

[paulca]

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

Maybe you could explain why EMC, HDS, IBM, LSI Logic, HP are all wrong by making and using hardware RAID controllers.

Not wrong.  However, they bring out different RAID controllers every few years.  If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller.  So these business will happily sell you one and support your recovery.... at professional service rates.

As I understand it anyway.

[paulca]

At the same time, I don't very much like using USB for mass storage, even 3.1 or faster especially if it's only one link serving a number of devices, and having two boxes is in any case cumbersome.

How many and how fast network interfaces are you considering?

I'm not that heavy a user and I'm the only user in the house.  I currently have SATA3 drives served over a single 1Gb link and it's absolutely fine for my purposes.

As long as the USB link is stable and mounted drives stay mounted and don't go catatonic it will be fine.

In the end I went for this enclosure.
https://www.amazon.co.uk/gp/product/B07QJVNBKL/

And this USB card:
https://www.amazon.co.uk/gp/product/B00JEVLEFQ/


UnRaid.  I considered it, but I think I'll be fine with a normal OS.  That way I can always use a vanila boot USB drive to repair it without having to provide my UnRaid config etc.  I have been in the pain for a machine failing and swapping the HDs over to find out they were all layered LVM volumes.  Wasted more than a day.

[Halcyon]

What's your budget?

[PlainName]

If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller

You shouldn't be trying to recover data from failed RAID. RAID is not a backup - it merely helps prevent downtime when you have a disk issue. Instead of being offline until a new drive turns up, or you get a spare from your stock, you keep going and get a bit of breathing space while you replace it at your leisure. You must still have backups and a disaster recovery procedure.

Thus, in the situation you describe, the correct process is to bin the old kit, install the new hardware and restore from backup. Bosh.

[newbrain]

As long as the USB link is stable and mounted drives stay mounted and don't go catatonic it will be fine.

Yes, that was my point, I still find it a big if, I'll be happy to know how it goes after some time!
In fact, there's people lamenting that enclosure's noise, heat and disconnection under load...for as much as Amazon reviewers can be trusted 

You also got my intentions in not recommending HW RAID. EMC DELL is very happy, financially speaking, to support their old gear, but we are talking a completely different target market!
I should have better qualified my statement, though.

As for Unraid, I've never tried it - I feel at home with FreeNAS - but your idea of using a general purpose OS is also a good compromise.
If you think zfs might be interesting, there's a number of articles by Jim Salter on Ars Technica about zfs on Linux.

[paulca]

In fact, there's people lamenting that enclosure's noise, heat and disconnection under load...for as much as Amazon reviewers can be trusted 

I didn't see the under load complains, I did see the noise complaints.  But I got bored after the 10th, "I'm an idiot, this didn't work!"  Reviews.

The fan is fixable with a pair of wire snips.   Or a £10 nice silent fan and a bi-metallic strip.  I think I have a 40*C one here already.  A bit high, but, my HDDs have suffered far, far, worse before.  I can always use a little MCU board with a temp sensor, and to spin a fan if I feel the need to tinker.

Anyway.  So far I have spent £245.  Let's see how much that grows making it work well.

Drives.  I am somewhat tempted to just go for 3x1Tb RAID5 (or similar ZFS if I don't need any custom partitions).  1Tb Seagates are £35.  2Tbs are £55 ish.  + my existing 6Tb and 2Tb drives... emptying and binning the rest.... or overwriting them and selling them on ebay on auction but I doubt anyone wants a S/H HDD under 1Tb.

[Monkeh]

If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller

You shouldn't be trying to recover data from failed RAID. RAID is not a backup - it merely helps prevent downtime when you have a disk issue. Instead of being offline until a new drive turns up, or you get a spare from your stock, you keep going and get a bit of breathing space while you replace it at your leisure. You must still have backups and a disaster recovery procedure.

Thus, in the situation you describe, the correct process is to bin the old kit, install the new hardware and restore from backup. Bosh.

Or, when you're not reliant on a proprietary controller and its firmware, just plug the drives in and continue operation with zero interruption. If you can't see how that's preferable to a 'solution' which costs more money and time for no added performance at this scale..

[Jeroen3]

There really isn't any reason anymore to require hardware raid. The software solutions do everything and more.
Why spend money on hardware you do not need? Those cards are expensive!

I think you'd like the microserver approach.
DIY-ing a server from consumer parts will most likely give you a higher energy usage anyways (like 30-50 W idle).
And a rack with blowymatrons will make you lose your mind.

[BravoV]

For home NAS I deliberately use proprietary RAID controller ... BUT ... used as new one cost arm & leg, and I bought two identical cards, as spare one is for backup.

Here where I live, used server's grade controllers or server parts which usually has powerful features such as big cache, battery backup, high IOPS and etc are relatively cheap.

Bought this used Dell Perc H730 , which has 1GB cache with battery backup, just google for the complete spec, only $75 a piece, and I bought two.

[PlainName]

Or..

Well done for taking down the straw man you inserted, allowing a bit of ad hom on top. Class.

[dnwheeler]

UnRaid.  I considered it, but I think I'll be fine with a normal OS.  That way I can always use a vanila boot USB drive to repair it without having to provide my UnRaid config etc.  I have been in the pain for a machine failing and swapping the HDs over to find out they were all layered LVM volumes.  Wasted more than a day.

That's one of the great benefits of UnRaid - each drive in the array can be removed and read outside of the array without UnRaid. They have a standard Linux filesystem and all the files are available and readable on any Linux system.

[madires]

Try a standard ATX mainboard with a low TDP CPU with integrated graphics. MIni PCs and servers a nice, but often lack ports, interfaces  and drive bays, and have a poor performance/price ratio. A NAS able to deliver nearly 1Gbps while running a few containers won't be cheap either.

[PKTKS]

2x for the latest MOBO with IGPUs are best choice.

Ive been there quite a while trying things...

- USB sucks - not even the best USB will be fine handling Terabytes
 or even a dozen Gigas. just bottlenecks

- You definitely want a MOBO with SATA 3.0 support may be
a server mobo to support a SATA 6Gbs RAID. That makes sense
if you plan to handle Terabytes or dozen Gigas.

- You want Linux. The file systems supported and the inherent
capability to build your own custom rescue disk/system is invaluable
Windows is a deep shit toy with single display and pathetic remote access
you also want ssh for remote secure purposes

- Your network should be fault tolerant anywhere you want
safety for data backup  just in case some RJ45 breaks you still
keep online...  Port trunking bonded ifaces.

Paul

[David Hess]

But... the machine is from the previous generation before power efficiency was a big seller.  It has a AMD Piledriver FX in it.  Even idle, under volted and under clocked the pulls 90W.  My entire house electric over night is only 180W.... half of which is that server.  Granted the 6 HDDs in it don't help, but they do spin down... or they are supposed to, but I think something keeps polling them, so they don't.  It has to go.

I usually figure that hard drives take 7 watts each, and double that on startup.  That makes 42 watts at idle so half of your measured power leaving half for everything else which is not too bad.

Make sure CPU power management is enabled.  Also, you did not say what GPU you are using but they can be real power hogs at idle.

I just built a close to minimum system suitable for server use with a Ryzen 2 3100 and 16GB of ECC RAM.  Excluding the hard drives and GPU, it still came out to $600.

[paulca]

It's not a server for a enterprise rack.  It's cheap, cheerful, functional, hopefully dust collecting.

The machine arrived today.  Looks legit.  Dell 7010, 3rd Gen i5, 3.4Ghz, 8Gb RAM.  It powered on, but I have not booted it yet.  I love the Dells and how they are built to be dismantled by people being paid minimum wage.  USB card went in without a single screw.... except fixing the low-profile bracket to the card.

The USB enclosure arrived yesterday.  It's PSU "claims" 12V 6500mA.  That should handle the spin up torque.  I can see the cheap ass chinsy fan...  not seen how to open it up yet, but I bet that fan will be swapped soon.  If it's loud, I'm sticking it with tape in the mean time.

Linux.  Please.  Why would I ever install Windows on a server?  I'd have to go and do 100 hail Linus's if I did that.

The choice that's more difficult is... do I go custom minimal Gentoo which I'm familiar with and can go NON-systemd.... or just plop a Ubuntu on it.

EDIT:  Amazon reviews had people screaming disappointment the USB enclosure had no UK plug.  But for £3 you can get a UK plug delivered, next day.
I also spent a whole £30 on a SanDisk 240Gb SSD for it.

Actual "NAS" drives will come next month.

[newbrain]

I found the invoice for my Microserver Gen 8, I bought it from Dustin end of 2016.
That might have been a good deal (considering the usual prices at Dustin's): 1858 SEK ~= 180€. Probably a black Friday sale.

Linux.  Please.  Why would I ever install Windows on a server?  I'd have to go and do 100 hail Linus's if I did that.

Oh, so I'm past due with my penance!
I run Windows Server 2016 Essentials in a VM on Hyper-V server 2019, so that would make 200 hail Linus.

Plus a number of other FreeBSD and Linux stuff, of course...I'll settle for 150.

[tkamiya]

I have such a box, or rather boxes on my home network.

They are both Dell T20 with 8GB of memory, SSD for boot drive.  Main box has mirrored hard disk for storage.  Backup box has a couple of 8TB disks for backup of the first.  I took care so that booting is contained within SSD, and does not depend on anything on storage HDs.  CPU is Dual Core Pentium of some kind, running Ubuntu Linux.  (not the Xeon version of T20).  OS has been stripped to run in text mode only.  Quite low power but, it spends 20 watts or less idling, and not much more on full bore transfers. CPU utilization is like 5% most of the time.  I can pretty much max out theoretical max on 1GB network for file transfers.  When I tried 10GB connection, it topped out at 2GB/s or so.  Quite enough for personal use.  My home network is redundant and ruggetized with multiple path.  Oh, yeah, it's almost noiseless.  I have quite sensitive ears.

I nixed the customized networked storage offerings for the fact that if anything is to fail, I will be depending on manufacturer to RMA or otherwise do something.  If something fails in few years, who knows if anything is supported.  I didn't want to DIY for reliability reasons, and frankly, it will cost more to do it that way.  I used everything off-the-shelf, and I have two more stand-by boxes ready for swap out.  I also bought them new rather than reusing tired old boxes.  (they were quite cheap)  First box is the main nfs/samba/ntp box, and the second box backs up the first nightly.  It has multiple backup schedules, all written by me using simple scripts and cron.

It has been working fine like this for past few years.  It's always available.  I don't depend on anyone else or any other organization to help in case of catastrophic failures.  Everything stays inside my little subnet.  I wouldn't want to do it any other way.

[perieanuo]

hi,
second hand servers are cheap, I used to have a dell T110 pretty silent. compairing to this, a workstation is child stuff, not the same endurance.
your 3-4 hdd requirement demands nas or server
ns doesn't take much power and really silent
small mips: choose nas
 higk mips: take sh server
regards, pierre

[paulca]

Well all the parts arrived and I bought 3 x 2 Tb Barracuda drives.

The PC boots, works perfectly.  Might need the fans replaced, but it's still quieter than the current machine.

The USB enclosure, the fan was way too loud.  The airflow non-existant anyway.  I got a Noctua 120mm fan off Amazon and went to fit it today.

Found a surface mount cap broken off and one of the clips on the lid broken.  Looks like transit damage.  So I soldered a through hole 220uF in it's place and wired in the Noctua fan.  Much quieter and I'm still testing thermals with the lid on, versus off, but I expect it will end up off.

RAID.

Pfffffblblblblblb.  I don't know.  I went for 3x in RAID5, then changed my mind and went 2 in RAID1, but I'm still not sure.

Really I'm procrastinating and trying to avoid the migration exercise as long a possible.

Currently the two in raid1 is resyncing the new pair... basically creating a complete copy of a blank drive.  Stupid, but it's testing the thermals nicely.

Code: [Select]

      [=>...................]  resync =  9.7% (189941184/1953382464) finish=186.7min speed=157352K/sec


[paulca]

Noctua fans rock once again.

With the enclosure running 2 drives flat out syncing for hours, the HDD temps have stabilised at 44*C +-1*C and there is significant air flow out the back of the unit now.... and it's much quieter.  The HD bearing whine and the head chatter are much louder than the fan.

Other than the audibles, I think the fan provides more pressure to force air through the narrow grilling.

EDIT: When I did this with 3 drives, no fan and no lid earlier all 3 drives hit "Too hot to hold" and reported 52*C.

[Mr. Scram]

Eh, tough choice.
I agree that an RPi is pointless for this purpose.

At the same time, I don't very much like using USB for mass storage, even 3.1 or faster especially if it's only one link serving a number of devices, and having two boxes is in any case cumbersome.

How many and how fast network interfaces are you considering?

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

My solution has been an HPE Microserver Gen 8, the base model with a Cleleron 1610T (but a Xeon E3-1220lv2 is now crawling from China to Sweden...).

• 4×2 TB HDD (2 pairs of different WD models)
• 1×256 GB SSD
• boot device is an internal 32 GB USB key (there's an internal USB 3.0 socket and a Micro-SD one)
• 4+8 GB ECC RAM

FreeNAS and of course ZFS, volumes are mirrored and striped on the HDDs, the SSD is used for less important or temporary stuff.
Administration is reasonably simple, though it took some attempts to have it join the AD Domain.

The server comes with two 1 GB/s NICs - three if you count the one for the lights out, iLO, management.
They can easily be saturated towards my main PC (two NICs there, too), using the native SMB multichannel in Windows (no setup whatsoever needed) for about 200 MB/s of total transfer rate.

There's an internal PCIe×16 low profile expansion slot.

Epochal uptimes, never a glitch and it's relatively silent even in a living room.

Only power and two Ethernet cables are connected, as with iLO there's no need for a physical console.
IIRC correctly, power consumption is about 30W at idle (the coming Xeon sports half the TDP, so I hope to decrease even further), but I don't feel like turning it off to measure it...

Very good build quality and maintainability, small enough (about 25×25×25 cm) to sit on a bookshelf.

Can be found on Ebay for not much money, even decently equipped.

In the pictures a file copy and the HPE server, below my open-air Hyper-V server (picture is terrible)

It'd be interesting to see how much of a difference that Xeon makes for the power consumption. I tried to look it up but most people report full system numbers with varying amounts of HDDs included.

[indeterminatus]

RAID1 is fine, I think. It should also allow you to extend the storage later on without needing a complete replica of your setup to migrate the data.

Depending on the data you keep there, please also consider a propery backup strategy (if not already done so). RAID is no backup!

Also, 44°C sounds a bit high for my taste, I like my drives between 25°C and 40°C. Do you expect the strain on the disks to go down in "normal" operation? If so, you're probably fine. If not, you're probably fine too, but your drives might fail earlier. (Cannot cite the statistics, so please take this with a grain of salt; I think to remember it was Backblaze to post the drive failures and as a result of that, I think to remember that the ideal operation temperature would be between 25°C and 40°C ... but again, I fail to find the quote. If my memory does not serve me right, please, someone, correct me!)

[paulca]

I'm expecting much less activity in operation.  Right now drive 1 is copying to drive 2 (oversimplification), so both hammering.  It's peaking, 200Mbyte/s, troughing at 130.  For a spinner that's not bad for write perf.... unless that's Mbits....

Resilience.  If it all went on fire tomorrow, I'd be heart broken, but my life would function.

I want to minimise the return to full operation time for failures and add another safety net to/before backups.

EDIT:
If it was 200Mb/s 2Tb would copy in 10,000 seconds.  That's 3 hours.  Yes?  (+1 bottle wine)

[David Hess]

Below is a photo of the system I just built.  It is not quite a server but it could be.

- 4 x 4TB HGST drives with RAID6 boot, RAID10 swap, and RAID6 data volumes.  Can convert RAID6 volume to RAID5 later to increase available space if necessary.
- Spare Areca 1210 RAID HBA
- Spare Radeon R7 240
- AMD Zen 2 3100 CPU - Originally wanted one of the pro series APUs but availability was a problem and the 3100 ended up being less expensive.  Could not find 3300X for sale anywhere.
- 16GB ECC RAM - 2 DIMM slots still open to increase memory later if needed.
- Crucial MX500 for swap drive.
- Relatively cheap ASRock B450M Steel Legend motherboard.  Was originally going to get B450M Pro4 but availability was problematical.
- Silverstone case has air filtering, easy to clean filters, and positive pressure.  Drives operate at 40C.
- EVGA SuperNOVA 550 GA power supply - In retrospect, should have gotten power supply with 4 SATA connectors per lead to make wiring easier.

[rdl]

Beyond simple mirroring for backup purposes I don't really see any need for RAID at all.

[David Hess]

Beyond simple mirroring for backup purposes I don't really see any need for RAID at all.

It provides redundancy against simple failure and it increased I/O throughput over single or spanned disks.

[dnwheeler]

Beyond simple mirroring for backup purposes I don't really see any need for RAID at all.

My UnRaid server currently has 17 hard drives. With that number of drives, the odds are very high that at least one drive will fail every year. I am currently running dual parity, which allows for any two drives to fail at the same time with no data loss. If more than two drives fail at once, my losses are limited to just the drives that failed.

[PlainName]

What happens if you have a fire or the PSU puts 110V on the 5V rail?

[BravoV]

What happens if you have a fire or the PSU puts 110V on the 5V rail?

That risk is no greater than accidentally delete the partition, or infected by ransomware that locked all files in it.

RAID is NOT a backup, period

[Mr. Scram]

That risk is no greater than accidentally delete the partition, or infected by ransomware that locked all files in it.

RAID is NOT a backup, period

It is a backup but not a backup solution. You literally have a hardware backup and no more.

[Berni]

Yep RAID is not a definitive backup solution but then again even backup solutions come in various levels.

You could have a 2nd identical server sitting next to it that gets mirrored over every so often.
But what if a lightning strike takes out both? Okay then unplug the 2nd server after a backup is done
But what if the house burns down? Okay then put the 2nd server in a different building.
But what if a flood sweeps the area? Okay then put the 2nd server 300km away.
But what if a virus corrupts the data before mirroring starts? Okay then the 2nd server is bigger in storage and makes versions.
But what if a virus slowly corrupts the data? Okay make the 2nd server have even more storage to hold a year worth of history.
But what if a hacker targets both servers and blows them up? Okay then make a 3rd secret server that i manually type the address in even month to sync data to.
...
etc

RAID is just the first line of defense in the chain to defend from hardware failures. Sure maybe you never had a HDD failure in 15 years. but if you have 15 drives in that server that 15 year mean time between failures becomes just 1 year. So it becomes pretty likely that you will see some drive failure and its much nicer if you can simply replace the drive and move on with no downtime. Rather than bring the whole system down and rebuild from a backup that might be days or even weeks old, then bring it back up.

[PlainName]

So it becomes pretty likely that you will see some drive failure and its much nicer if you can simply replace the drive and move on with no downtime

Indeed, yes. It is hard to overstate how useful that is

But... for most purposes isn't a mirror fine for that? Obviously, if you have enough drives (not needed for the array per se) that you'll see one or two failures quite often, an array that can cope with more than one failure at a time would be useful. But if you have, say, four drives, a mirror will give you twice as much storage as the full thing and as close as dammit robustness to boot.

[Berni]

Yes mirror is useful when you have a small number of drives, but once you get past 3 drives the other parity based RAID methods are better because they can give more redundancy for wasting less space.

If you have 4x1TB drives you could have 2+2 mirroring to get 2TB of usable space with 1 allowed failure if you are unlucky or 2 if lucky.
If you have 4x1TB drives and use 3+1 parity you get 3TB of usable space with 1 allowed failure. Or 2+2 parity to give you the same 2TB of usable space but grantee 2 allowed failures (any 2 combination of drives)

Once you get to more drives say 10x1TB you can get 8+2 parity to have 8TB of usable space with guaranteed 2 allowed failures versus 5+5 mirror giving you 5TB of usable space with only 1 guaranteed allowed failure.

[PlainName]

OK, thanks

On reason I like the mirror is because when, say, an OS update goes wrong you can whip one drive out and keep it as the authoritative source whilst trying to fix things on the other.

[amspire]

I am a fan of the commercial NAS boxes like the Netgear Readnas 6 boxes. They are not cheap, but I have managed to pick up some cheap second hand and they tend to be very reliable and very quiet. Don't get the older Netgear boxes that do not run ReadyNAS 6.

They are a commercial distro, but they are basically a well configured Debian Linux box with SSH access. You can update via the Netgear ReadyNAS upgrades, but you can also use apt-get commands for upgrading and for installing your own Linux apps. I have often used my own scripts to do daily rsyncs to remote cloud sites or to remote NAS boxes.

Netgear uses BTRFS for the disk system and snapshotting, and mdadm for the raid. They do not use the RAID of BTRFS.

I like the low power - my 2 disk ReadyNAS  box is something 12W average and it handles power failures and reboots very well. The boxes are also very compact with excellent hot-swappable hardware. I got my current NAS box for about A$120 with a single WD 4G RED NAS drive. Had to buy a second RED drive.

Richard

[PlainName]

the Netgear Readnas 6 boxes

Have to say that if I were in the market for off-the-shelf NAS, Netgear wouldn't even get on the long list never mind the shortlist. They make some decent non-managed hardware, but anywhere firmware is involved they are terrible. No, terrible it too kind. Atrocious is more appropriate.

[Berni]

OK, thanks

On reason I like the mirror is because when, say, an OS update goes wrong you can whip one drive out and keep it as the authoritative source whilst trying to fix things on the other.

Well... RAID will most likely not help with that.

When the OS does a write to the drive that write gets committed to both disk drives once there is free disk time available to do so. So if the OS shoots its own foot it will do so on both mirrored drives at nearly the same time. The actual advantage of mirroring is that the two mirrors are just two identical normal disk drives that will boot in any PC. But any more fancy RAID arrays absolutely requires a working RAID controller to get any useful data from it.

For this case i actually use incremental backups. Using Veeam Agent ( https://www.veeam.com/windows-endpoint-server-backup-free.html ) i have a scheduled task that runs once per week, making a complete backup of my entire PC and storing that on the NAS server. To save space it creates a full backup every so often but then just makes incremental backup files every week. So with this i have the complete state of my entire PC stored for about 3 months into the past. If my PCs SSD explodes all i have to do is insert a new one, boot from Veeams USB recovery image, show it the path to the backup file on the network and about 1 hour later the PC will boot up in just the same state it was 1 week ago. Or if a randsome ware has trashed the drive i can go back 2 weeks or 3 or more. Or if i overwrote an important file then i can just go open the backup file like a ZIP archive and browse trough my entire HDD as it was 1 month ago to retrieve an old non overwritten version of a file.

[Mr. Scram]

I am a fan of the commercial NAS boxes like the Netgear Readnas 6 boxes. They are not cheap, but I have managed to pick up some cheap second hand and they tend to be very reliable and very quiet. Don't get the older Netgear boxes that do not run ReadyNAS 6.

They are a commercial distro, but they are basically a well configured Debian Linux box with SSH access. You can update via the Netgear ReadyNAS upgrades, but you can also use apt-get commands for upgrading and for installing your own Linux apps. I have often used my own scripts to do daily rsyncs to remote cloud sites or to remote NAS boxes.

Netgear uses BTRFS for the disk system and snapshotting, and mdadm for the raid. They do not use the RAID of BTRFS.

I like the low power - my 2 disk ReadyNAS  box is something 12W average and it handles power failures and reboots very well. The boxes are also very compact with excellent hot-swappable hardware. I got my current NAS box for about A$120 with a single WD 4G RED NAS drive. Had to buy a second RED drive.

Richard

12 watt is very low. Both disks should consume as much when in use. What kind of processor is in your system?

[Monkeh]

But any more fancy RAID arrays absolutely requires a working RAID controller to get any useful data from it.

And that's the great thing about software RAID, you're not tied to a specific model and firmware version of controller which may or may not still be available.

[PlainName]

I think you haven't grasped what I was getting at.

So if the OS shoots its own foot it will do so on both mirrored drives at nearly the same time.

Yes, exactly that. Or if you do a fat fingered delete/format/whatever. The point is that the system is borked,  but you might be able to fix it. Do you want to try on the one copy you have? No, you take one half of the mirror and store it somewhere safe, then you can try fixing the other half. If it works, fine, but if it doesn't you haven't made things worse.

With a setup where your data is striped over more than one drive, you can't do that. If that happened, then you'd have to revert to backups, but there are good reasons why that might be the last option.

[Monkeh]

I think you haven't grasped what I was getting at.

So if the OS shoots its own foot it will do so on both mirrored drives at nearly the same time.

Yes, exactly that. Or if you do a fat fingered delete/format/whatever. The point is that the system is borked,  but you might be able to fix it. Do you want to try on the one copy you have? No, you take one half of the mirror and store it somewhere safe, then you can try fixing the other half. If it works, fine, but if it doesn't you haven't made things worse.

With a setup where your data is striped over more than one drive, you can't do that. If that happened, then you'd have to revert to backups, but there are good reasons why that might be the last option.

In those instances I generally resort to making an image before messing with things. Small drive for OS, image if you need to mess with it, generally treat as disposable.

[PlainName]

the great thing about software RAID

It's swings and roundabouts. Software RAID isn't transparent. Hardware RAID doesn't lend itself to SMART monitoring of individual drives. Etc. Depends on what the implementor/user considers important to them.

[Berni]

But any more fancy RAID arrays absolutely requires a working RAID controller to get any useful data from it.

And that's the great thing about software RAID, you're not tied to a specific model and firmware version of controller which may or may not still be available.

Yep for this reason i use unRAID on my homebuilt NAS server.

It provides the redundancy of RAID while even on a failed array i can pull the drives out and read the files from each drive individually on any Linux machine. Tho as a consequence i also loose the speed advantage of using RAID, but i only have a 1Gbit LAN network anyway, so it bottlenecks me to about half the sequential read speed of a single drive. Totally worth the price of unRAID for my home server use case. Pretty much the reason why i didn't go with FreeNAS, since im an electronics engineer, not a IT professional, so i want to avoid the chance of doing something really stupid and corrupting a more fragile RAID array to an unrecoverable state.

[Monkeh]

the great thing about software RAID

It's swings and roundabouts. Software RAID isn't transparent. Hardware RAID doesn't lend itself to SMART monitoring of individual drives. Etc. Depends on what the implementor/user considers important to them.

Software RAID is transparent to everything but the OS, and most hardware controllers do allow SMART access these days. The only real upside I've ever seen to hardware RAID is performance in certain applications, for which consumer drives need not apply anyway.

It is, and this is my opinion you may disagree with if you wish, a needless expense and unnecessary liability in small scale bulk storage.

[PlainName]

In those instances I generally resort to making an image before messing with things.

I run one off every night as a matter of course. I try to have a recent one if I know I'm going to do something dicky, but sometimes these things happen out of the blue, just when you missed last night's job, etc. Don't get fixated on OS stuff - that's just an example off the top of my head. There have been other instances that definitely didn't involve the OS.

Small drive for OS, image if you need to mess with it

Yes, that's what I do. And my small OS drive is 90GB with 15GB free. Thanks Microsoft - even telling Visual Studio to sit on a different drive, it won't fit because the fat-arsed git will plonk 10's of GB in system32 for no good reason.

Sorry, you hit a nerve there

[paulca]

Thanks Microsoft - even telling Visual Studio to sit on a different drive, it won't fit because the fat-arsed git will plonk 10's of GB in system32 for no good reason.

Reminds me of installing SQL Server or Exchange.  "All your computer belong to us now, ok?"

[amspire]

the Netgear Readnas 6 boxes

Have to say that if I were in the market for off-the-shelf NAS, Netgear wouldn't even get on the long list never mind the shortlist. They make some decent non-managed hardware, but anywhere firmware is involved they are terrible. No, terrible it too kind. Atrocious is more appropriate.

Sounds to me like you have never used ReadNAS 6. You cannot compare ReadNAS to, say, modem firmware. It is Debian Linux that has been updated for something like the last 8 years probably at least monthly. I have used at least 8 of the ReadyNAS boxed running 24/7 with never a software issue or hardware failure.

[PlainName]

Sounds to me like you have never used ReadNAS 6

You are correct. My comments are based on many years experience of Netgear products, including NAS kit, but not RN6. Nevertheless, incompetence seems to be baked into the corporate culture there, and unless their brand has been bought to to stick on someone elses OEM badge stuff I see no reason why RN* should be any different.

with never a software issue or hardware failure

When a university had an IP address DDOS'd, the Netgear kit end users didn't notice a thing wrong.

Edit: just stumbled across this:

Neatgear has cocked up its cloud management service, losing data stored locally on ReadyNAS devices' shared folders worldwide – and customers have complained to The Register about only being informed four weeks later.

https://www.theregister.com/2017/04/26/netgear_customer_backups_fried/

[tkamiya]

I've been in this business for long time, as a customer and as a company rep.  Just until recently, I worked for one of the biggest cloud storage/computing supplier.  Proper cloud service of any kind is expensive.  Free service or something given to you as part of hardware purchases aren't.  But they do cut corners everywhere they can, and hide them in fine prints, which you ok'd without reading. 

Personally, not that I have any top-secret content, but I don't trust cheap suppliers ability, budget, or willingness to keep and keep secure my files for free.  I just don't trust them.  My files are valuable to me.  If I have to keep it elsewhere, I will pay premium for it.  Since it's not a possibility, I do it in-house.  (literally....)

Also, I found using old desktop or servers can be a false economy.  Today's machines are quite efficient in terms of power consumption. It's far cheaper to run 2 disks mirrored than 10 disks in raid 6. 

I also try not get attached to company's proprietary anything.  I considered it but abandoned those NAS box ideas early on.  Can I recover data if CPU board dies?  What kind of filesystem are they?  Can "secure protection" be bypassed by me, the owner of the file, to read content on another machine?  All those were considerations.

In the end, I ended up with a new generic Linux server, spare parts, and software raid.  Having worked in hardware/software support for total of 20 some years, I don't want to depend on them to get my precious files back.  I am my own support.

[amspire]

Neatgear has cocked up its cloud management service, losing data stored locally on ReadyNAS devices' shared folders worldwide – and customers have complained to The Register about only being informed four weeks later.

https://www.theregister.com/2017/04/26/netgear_customer_backups_fried/

The last thing I ever wanted to even look at was Netgears cloud service.

It was always a dubious service that was designed to make cloud storage look easy rather then secure.

It was totally optional and unnecessary.

The nice thing about ReadNAS is all the NAS functionality is done using open source tools and you have access to it all.

I don't know of any Netgear family for which you have such complete access to the workings.

It is also pretty nice that rather then have different ReadyNAS's for all the different models, they have one ReadyNAS for all platforms from singe core Marvell processors with 500M RAM to multi-core rack units Intel processors and many gigs or RAM. They just have the same O/S + ReadyNAS available in distro repositories for all different platforms. They are all kept in sync - even the platforms that have been phased out for years.

[madires]

In the end, I ended up with a new generic Linux server, spare parts, and software raid.  Having worked in hardware/software support for total of 20 some years, I don't want to depend on them to get my precious files back.  I am my own support.

A wise conclusion!

[paulca]

I've settled with a 2x2Tb RAID 1 array, called vault.  Backups of bulk storage, like photos/videos/unreplaceable media get's stored or RSync'd here.  My home automation system will run from this drive.  Software "devel" folder also.

OS and /home is on the 256Gb Sandisk SSD.  Rsyncing key config files to vault.

My existing 6Tb "storage" drive will just move over.
My older "bulk" 2Tb storage drive will move over too.
That leaves me a blank 2Tb drive as spare.

I also have another USB enclosure with a 2Tb drive which I use as "offline" backups.  I turn it on, I rsync things to it and I turn it off.  Minimising the drive wear time.

Cloud... is just someone else's computer.
If the service is free.  YOU are the product being sold.

[Mr. Scram]

I've settled with a 2x2Tb RAID 1 array, called vault.  Backups of bulk storage, like photos/videos/unreplaceable media get's stored or RSync'd here.  My home automation system will run from this drive.  Software "devel" folder also.

OS and /home is on the 256Gb Sandisk SSD.  Rsyncing key config files to vault.

My existing 6Tb "storage" drive will just move over.
My older "bulk" 2Tb storage drive will move over too.
That leaves me a blank 2Tb drive as spare.

I also have another USB enclosure with a 2Tb drive which I use as "offline" backups.  I turn it on, I rsync things to it and I turn it off.  Minimising the drive wear time.

Cloud... is just someone else's computer.
If the service is free.  YOU are the product being sold.

2 Tb? That's only like 250 GB. Could've gone for a bigger drive. Even those 750 GB drives aren't huge.

[paulca]

2 Tb? That's only like 250 GB. Could've gone for a bigger drive. Even those 750 GB drives aren't huge.

 

[Mr. Scram]

Putting in extra effort to do it wrong is facepalm worthy indeed. 

[Berni]

You start to take the difference between MB and Mb very seriously once you have dealt with enough memory chip manufacturers.

They never just give you the number of bytes, but always the number of bits cause its the bigger number. So when you are buying a 64Mb flash chip you better be aware this is actually only 8MB

Then again if we are pedantic saying you have 8GB of RAM is technically wrong and should have said 8 GiB (Gibibyte) since RAM uses prefixes of 1024 while calling a 500GB hard drive is correct since they use 1000 prefixes so its actually 465.6 GiB while Windows will then present this as having a 465.6 GB partition. But lets be real nobody uses mibibytes

[Mr. Scram]

You start to take the difference between MB and Mb very seriously once you have dealt with enough memory chip manufacturers.

They never just give you the number of bytes, but always the number of bits cause its the bigger number. So when you are buying a 64Mb flash chip you better be aware this is actually only 8MB

Then again if we are pedantic saying you have 8GB of RAM is technically wrong and should have said 8 GiB (Gibibyte) since RAM uses prefixes of 1024 while calling a 500GB hard drive is correct since they use 1000 prefixes so its actually 465.6 GiB while Windows will then present this as having a 465.6 GB partition. But lets be real nobody uses mibibytes

One is a few percent out and one is an order of magnitude out. The latter is a big difference. Using the incorrect notation seems to be mostly accidental too as typing it wrong isn't less effort, so pointing it out may help. If you order hardware with the wrong prefix there's a good chance you'll get exactly that and there's little you can do at that point. Manufacturers aren't using one or the other consistently either.

Whether not using Gibibytes in incorrect is a bit of a difficult discussion as it was appended after the fact and there's a good argument that GB can mean either. It's good practice to use the ibi notation to avoid the confusion though.

[paulca]

So I got everything up and running.  I had the thing all prepped weeks ago, but was procrastinating and putting off migration, trying to work out if I could do it without downtime.

In the end I said, sod it, downtime will be incentive to get it working.  Took 2 hours.  Biggest delays where messing around with Ubuntu's NetworkManager and finally uninstalling it completely for wasting time.

Performance wise, it's fine.  Disk bandwidth terms...

From internal SSD to USB3 HDD it took 8 minutes to copy 33Gb.

33/8 = 4.1Gb per minute =  68Mbyte per second.  Sounds low to me, but it was writing real work load files, like 10s of thousands of small files and a few large ones.

Network writes from a Windows 10 machine on 1Gbit/s LAN started out around 112Mbyte/s, which is a saturated gigabit link more or less.  When the memory caches filled up that dropped to the 30s and 40s, which again looks pretty low and might need some tuning.  So copying anything less than 4Gb it's flat out, over that it's slow.

Final disk config is TBD, but currently I have:
120Gb SSD OS boot drive Ubuntu headless
240Gb SSD /home [Shared]
2x2Tb RAID 1 "Vault" where current and past projects get rsynced to daily and backed up periodically to external HD kept offline normally.  Stuff I can't just download again goes here.
6Tb bulk storage / media.  [Shared]

Still to allocate:
2Tb brand new
2Tb reclaimed

Probably for the bin, or wiped and sold/donated to good homes.
2x 1Tb SeaGate (6 yo )
2x 500Gb Hitachi (10 yo )

Next project will be to try and change the fan in the SFF PC to a Noctua and maybe consider if the PSU fan can be replaced or if a new PSU is expensive.  Just aiming for silence, at least when the drive bay spins down.

It seems happy anyway.

[nightfire]

Sounds like a nice setup you have done here!

Also, my 0.02 bucks: The HP Microserver is quite popular with lots of "build-your-own-NAS" scenarios.

My own NAS I built years ago consisted of a Silverstone case, where I put originally a 12V Mainboard with the first Intel Atom N270 (notebook low voltage) version in. Booted FreeBSD from a 16GB CF card, had 2x 1TB Seagate Constellation (Video recording HDD) in it, in a mirrored config. Under full load, that rig consumed 24 Watts...
Idle, I came with both HDD suspended to 12W.
Meanwhile, as that board died, an old desktop board with an old i3 Sandy Bridge went in, and boot from SSD- 2 internal HDD, that soon will be pumped up to 3 (have only 4 SATA connectors) and an external USB drive that gets connected from time to time to do an external backup.

In my experience, a NAS with a RAID over several USB devices is doable, but there are so many variables where it may go wrong or cause issues, that I really do not like that.
Internal HDD/SSD are the way to go for me, and some external backup on a USB drive that gets physically disconnected after backup is a way of protecting valuable data.
RAID can sometimes work wonder, but does not protect from fire, overvoltage or a crypto trojan.

[paulca]

an external USB drive that gets connected from time to time to do an external backup.

This is something I do too.  It's much cheaper than using optical or tape.  Keeping it off line means it will last a lot of years.... and you can't accidentally delete or infect files if they aren't connected.

Fine for home use.

For small business use I have provided people with a swappable drive bay that is backed up to every night and they are instructed to swap it with a duplicate in their fire safe every week.