Come diss my NAS/Server idea!

[paulca]

I have a 24/7 busy box.  I want my normal use desktops/media centres to be small, fast, low power SSD'd devices that I can switch off and walk away from.  But I need somethings to be available to all of them.

So I downgraded my previous gaming machine by removing it's video card and youtubing "Worst ever PCIe GPU" and picked a winner.  NVidia 720

But... the machine is from the previous generation before power efficiency was a big seller.  It has a AMD Piledriver FX in it.  Even idle, under volted and under clocked the pulls 90W.  My entire house electric over night is only 180W.... half of which is that server.  Granted the 6 HDDs in it don't help, but they do spin down... or they are supposed to, but I think something keeps polling them, so they don't.  It has to go.

Oh and it coil whines and fan whines constantly.

So I need a machine which is.
Cheap
Low power
Quiet
Robust
Vaguely capable of saturating 1Gb Ethernet file transfer.
Capable of handling 3 or 4 HDDs and an SSD or two.
Running a docker host for a few dozen services.

Solution 1:  Just buy a NAS.
Rejection:  Not cheap, running a docker host could get complex, especially if it's not AMD64 arch.  I don't really want a micro-managed bespoke OS.

Solution 2:  Buy an old server.
Rejection:  Large.  Probably noisy.  The generation for my budget would probably not be power efficient either.  Can't be absolutely sure what I'm going to get S/H.

Solution 3:  Use a little PI that could.
Rejection.... please.  Just No.

Solution 4:  Buy a beefy mini-pc and use external USB drives.
hmm...  close, but multiple USB drives with power adapters and usb hubs is messy and not that robust.  Mini-PC architectures are often odd.  Good ones are not cheap.

So, what I'm going with is....

A refurbished corporate Dell small form factor PC because they are reliable, robust and the "bang for your buck" on the second hand market is amazing at the minute.  2nd gen i5 3200, 8Gb RAM, 256Gb SSD will cost about £120.

A PCIe x4 USB3.1 card.  Providing theoretical 10GB/s bandwidth.

A 4 or 5 bay USB3.1 drive enclosure.

(Might downgrade to USB 3.0 for cost)

Two small boxes ... with fans which can be tamed or just removed.  It will take a linux install easily, run docker with enough memory, etc.

Power consumption of these machines idle is around 19W.  Add the disk array and call it 25W. 

What could go wrong?  Right?

I do need to put the 6HDDs on a diet though.  Half of the older ones are legacy Linux and Windows installs.  I think there is even a stray Windows XP partition on one of them.

Not sure I need RAID or if it is even wise over USB.  I could look at 3 1Tb in RAID or similar with part reliability part performance.  And a 6Tb drive for bulk stuff and media.

The question is... do I try and set that RAID up native on the Linux or hand those drives over to a NAS controller running on Linux?... is that even available?

[newbrain]

Eh, tough choice.
I agree that an RPi is pointless for this purpose.

At the same time, I don't very much like using USB for mass storage, even 3.1 or faster especially if it's only one link serving a number of devices, and having two boxes is in any case cumbersome.

How many and how fast network interfaces are you considering?

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

My solution has been an HPE Microserver Gen 8, the base model with a Cleleron 1610T (but a Xeon E3-1220lv2 is now crawling from China to Sweden...).

• 4×2 TB HDD (2 pairs of different WD models)
• 1×256 GB SSD
• boot device is an internal 32 GB USB key (there's an internal USB 3.0 socket and a Micro-SD one)
• 4+8 GB ECC RAM

FreeNAS and of course ZFS, volumes are mirrored and striped on the HDDs, the SSD is used for less important or temporary stuff.
Administration is reasonably simple, though it took some attempts to have it join the AD Domain.

The server comes with two 1 GB/s NICs - three if you count the one for the lights out, iLO, management.
They can easily be saturated towards my main PC (two NICs there, too), using the native SMB multichannel in Windows (no setup whatsoever needed) for about 200 MB/s of total transfer rate.

There's an internal PCIe×16 low profile expansion slot.

Epochal uptimes, never a glitch and it's relatively silent even in a living room.

Only power and two Ethernet cables are connected, as with iLO there's no need for a physical console.
IIRC correctly, power consumption is about 30W at idle (the coming Xeon sports half the TDP, so I hope to decrease even further), but I don't feel like turning it off to measure it...

Very good build quality and maintainability, small enough (about 25×25×25 cm) to sit on a bookshelf.

Can be found on Ebay for not much money, even decently equipped.

In the pictures a file copy and the HPE server, below my open-air Hyper-V server (picture is terrible)

[dnwheeler]

I suggest you look into UnRaid - it will give you a parity-protected drive pool without the limitations of RAID (you can mix-and-match drive sizes, and even in a worst-case scenario, all of the drives are readable in a standard Linux system). It also offers great support for hosting Docker containers and VMs. It is Linux based, but has a nice Web-based management UI (or you can dive in and customize whatever you want). It also boots directly from a USB thumb drive so you don't need to dedicate any hard drive space for the OS.

[PlainName]

Can be found on Ebay for not much money, even decently equipped.

Not any time I've been looking for one! I'd happily take one off your hands for £100.

[Jr460]

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

Maybe you could explain why EMC, HDS, IBM, LSI Logic, HP are all wrong by making and using hardware RAID controllers.

[paulca]

Can be found on Ebay for not much money, even decently equipped.

Not any time I've been looking for one! I'd happily take one off your hands for £100.

Think I bought the last one.  This is a really good bargain, if it's not too good to be true.  Gen3 i5 3470, 8Gb DDR3, 256Gb HDD, not SSD, but I'll be depositing that at my local recycling centre anyway.

https://www.amazon.co.uk/gp/product/B01E97IVD2/

EDIT:  Oops, you meant the HPE Microserver Gen 8.  Yes.  Looking like £300+ on Amazon.

[paulca]

DO NOT rely on any HW RAID driver, is in general a good recipe for disaster.

Maybe you could explain why EMC, HDS, IBM, LSI Logic, HP are all wrong by making and using hardware RAID controllers.

Not wrong.  However, they bring out different RAID controllers every few years.  If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller.  So these business will happily sell you one and support your recovery.... at professional service rates.

As I understand it anyway.

[paulca]

At the same time, I don't very much like using USB for mass storage, even 3.1 or faster especially if it's only one link serving a number of devices, and having two boxes is in any case cumbersome.

How many and how fast network interfaces are you considering?

I'm not that heavy a user and I'm the only user in the house.  I currently have SATA3 drives served over a single 1Gb link and it's absolutely fine for my purposes.

As long as the USB link is stable and mounted drives stay mounted and don't go catatonic it will be fine.

In the end I went for this enclosure.
https://www.amazon.co.uk/gp/product/B07QJVNBKL/

And this USB card:
https://www.amazon.co.uk/gp/product/B00JEVLEFQ/


UnRaid.  I considered it, but I think I'll be fine with a normal OS.  That way I can always use a vanila boot USB drive to repair it without having to provide my UnRaid config etc.  I have been in the pain for a machine failing and swapping the HDs over to find out they were all layered LVM volumes.  Wasted more than a day.

[Halcyon]

What's your budget?

[PlainName]

If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller

You shouldn't be trying to recover data from failed RAID. RAID is not a backup - it merely helps prevent downtime when you have a disk issue. Instead of being offline until a new drive turns up, or you get a spare from your stock, you keep going and get a bit of breathing space while you replace it at your leisure. You must still have backups and a disaster recovery procedure.

Thus, in the situation you describe, the correct process is to bin the old kit, install the new hardware and restore from backup. Bosh.

[newbrain]

As long as the USB link is stable and mounted drives stay mounted and don't go catatonic it will be fine.

Yes, that was my point, I still find it a big if, I'll be happy to know how it goes after some time!
In fact, there's people lamenting that enclosure's noise, heat and disconnection under load...for as much as Amazon reviewers can be trusted 

You also got my intentions in not recommending HW RAID. EMC DELL is very happy, financially speaking, to support their old gear, but we are talking a completely different target market!
I should have better qualified my statement, though.

As for Unraid, I've never tried it - I feel at home with FreeNAS - but your idea of using a general purpose OS is also a good compromise.
If you think zfs might be interesting, there's a number of articles by Jim Salter on Ars Technica about zfs on Linux.

[paulca]

In fact, there's people lamenting that enclosure's noise, heat and disconnection under load...for as much as Amazon reviewers can be trusted 

I didn't see the under load complains, I did see the noise complaints.  But I got bored after the 10th, "I'm an idiot, this didn't work!"  Reviews.

The fan is fixable with a pair of wire snips.   Or a £10 nice silent fan and a bi-metallic strip.  I think I have a 40*C one here already.  A bit high, but, my HDDs have suffered far, far, worse before.  I can always use a little MCU board with a temp sensor, and to spin a fan if I feel the need to tinker.

Anyway.  So far I have spent £245.  Let's see how much that grows making it work well.

Drives.  I am somewhat tempted to just go for 3x1Tb RAID5 (or similar ZFS if I don't need any custom partitions).  1Tb Seagates are £35.  2Tbs are £55 ish.  + my existing 6Tb and 2Tb drives... emptying and binning the rest.... or overwriting them and selling them on ebay on auction but I doubt anyone wants a S/H HDD under 1Tb.

[Monkeh]

If you have a HW RAID failure you can only recover your data if you have EXACTLY the same model of RAID controller

You shouldn't be trying to recover data from failed RAID. RAID is not a backup - it merely helps prevent downtime when you have a disk issue. Instead of being offline until a new drive turns up, or you get a spare from your stock, you keep going and get a bit of breathing space while you replace it at your leisure. You must still have backups and a disaster recovery procedure.

Thus, in the situation you describe, the correct process is to bin the old kit, install the new hardware and restore from backup. Bosh.

Or, when you're not reliant on a proprietary controller and its firmware, just plug the drives in and continue operation with zero interruption. If you can't see how that's preferable to a 'solution' which costs more money and time for no added performance at this scale..

[Jeroen3]

There really isn't any reason anymore to require hardware raid. The software solutions do everything and more.
Why spend money on hardware you do not need? Those cards are expensive!

I think you'd like the microserver approach.
DIY-ing a server from consumer parts will most likely give you a higher energy usage anyways (like 30-50 W idle).
And a rack with blowymatrons will make you lose your mind.

[BravoV]

For home NAS I deliberately use proprietary RAID controller ... BUT ... used as new one cost arm & leg, and I bought two identical cards, as spare one is for backup.

Here where I live, used server's grade controllers or server parts which usually has powerful features such as big cache, battery backup, high IOPS and etc are relatively cheap.

Bought this used Dell Perc H730 , which has 1GB cache with battery backup, just google for the complete spec, only $75 a piece, and I bought two.

[PlainName]

Or..

Well done for taking down the straw man you inserted, allowing a bit of ad hom on top. Class.

[dnwheeler]

UnRaid.  I considered it, but I think I'll be fine with a normal OS.  That way I can always use a vanila boot USB drive to repair it without having to provide my UnRaid config etc.  I have been in the pain for a machine failing and swapping the HDs over to find out they were all layered LVM volumes.  Wasted more than a day.

That's one of the great benefits of UnRaid - each drive in the array can be removed and read outside of the array without UnRaid. They have a standard Linux filesystem and all the files are available and readable on any Linux system.

[madires]

Try a standard ATX mainboard with a low TDP CPU with integrated graphics. MIni PCs and servers a nice, but often lack ports, interfaces  and drive bays, and have a poor performance/price ratio. A NAS able to deliver nearly 1Gbps while running a few containers won't be cheap either.

[PKTKS]

2x for the latest MOBO with IGPUs are best choice.

Ive been there quite a while trying things...

- USB sucks - not even the best USB will be fine handling Terabytes
 or even a dozen Gigas. just bottlenecks

- You definitely want a MOBO with SATA 3.0 support may be
a server mobo to support a SATA 6Gbs RAID. That makes sense
if you plan to handle Terabytes or dozen Gigas.

- You want Linux. The file systems supported and the inherent
capability to build your own custom rescue disk/system is invaluable
Windows is a deep shit toy with single display and pathetic remote access
you also want ssh for remote secure purposes

- Your network should be fault tolerant anywhere you want
safety for data backup  just in case some RJ45 breaks you still
keep online...  Port trunking bonded ifaces.

Paul

[David Hess]

But... the machine is from the previous generation before power efficiency was a big seller.  It has a AMD Piledriver FX in it.  Even idle, under volted and under clocked the pulls 90W.  My entire house electric over night is only 180W.... half of which is that server.  Granted the 6 HDDs in it don't help, but they do spin down... or they are supposed to, but I think something keeps polling them, so they don't.  It has to go.

I usually figure that hard drives take 7 watts each, and double that on startup.  That makes 42 watts at idle so half of your measured power leaving half for everything else which is not too bad.

Make sure CPU power management is enabled.  Also, you did not say what GPU you are using but they can be real power hogs at idle.

I just built a close to minimum system suitable for server use with a Ryzen 2 3100 and 16GB of ECC RAM.  Excluding the hard drives and GPU, it still came out to $600.

[paulca]

It's not a server for a enterprise rack.  It's cheap, cheerful, functional, hopefully dust collecting.

The machine arrived today.  Looks legit.  Dell 7010, 3rd Gen i5, 3.4Ghz, 8Gb RAM.  It powered on, but I have not booted it yet.  I love the Dells and how they are built to be dismantled by people being paid minimum wage.  USB card went in without a single screw.... except fixing the low-profile bracket to the card.

The USB enclosure arrived yesterday.  It's PSU "claims" 12V 6500mA.  That should handle the spin up torque.  I can see the cheap ass chinsy fan...  not seen how to open it up yet, but I bet that fan will be swapped soon.  If it's loud, I'm sticking it with tape in the mean time.

Linux.  Please.  Why would I ever install Windows on a server?  I'd have to go and do 100 hail Linus's if I did that.

The choice that's more difficult is... do I go custom minimal Gentoo which I'm familiar with and can go NON-systemd.... or just plop a Ubuntu on it.

EDIT:  Amazon reviews had people screaming disappointment the USB enclosure had no UK plug.  But for £3 you can get a UK plug delivered, next day.
I also spent a whole £30 on a SanDisk 240Gb SSD for it.

Actual "NAS" drives will come next month.

[newbrain]

I found the invoice for my Microserver Gen 8, I bought it from Dustin end of 2016.
That might have been a good deal (considering the usual prices at Dustin's): 1858 SEK ~= 180€. Probably a black Friday sale.

Linux.  Please.  Why would I ever install Windows on a server?  I'd have to go and do 100 hail Linus's if I did that.

Oh, so I'm past due with my penance!
I run Windows Server 2016 Essentials in a VM on Hyper-V server 2019, so that would make 200 hail Linus.

Plus a number of other FreeBSD and Linux stuff, of course...I'll settle for 150.

[tkamiya]

I have such a box, or rather boxes on my home network.

They are both Dell T20 with 8GB of memory, SSD for boot drive.  Main box has mirrored hard disk for storage.  Backup box has a couple of 8TB disks for backup of the first.  I took care so that booting is contained within SSD, and does not depend on anything on storage HDs.  CPU is Dual Core Pentium of some kind, running Ubuntu Linux.  (not the Xeon version of T20).  OS has been stripped to run in text mode only.  Quite low power but, it spends 20 watts or less idling, and not much more on full bore transfers. CPU utilization is like 5% most of the time.  I can pretty much max out theoretical max on 1GB network for file transfers.  When I tried 10GB connection, it topped out at 2GB/s or so.  Quite enough for personal use.  My home network is redundant and ruggetized with multiple path.  Oh, yeah, it's almost noiseless.  I have quite sensitive ears.

I nixed the customized networked storage offerings for the fact that if anything is to fail, I will be depending on manufacturer to RMA or otherwise do something.  If something fails in few years, who knows if anything is supported.  I didn't want to DIY for reliability reasons, and frankly, it will cost more to do it that way.  I used everything off-the-shelf, and I have two more stand-by boxes ready for swap out.  I also bought them new rather than reusing tired old boxes.  (they were quite cheap)  First box is the main nfs/samba/ntp box, and the second box backs up the first nightly.  It has multiple backup schedules, all written by me using simple scripts and cron.

It has been working fine like this for past few years.  It's always available.  I don't depend on anyone else or any other organization to help in case of catastrophic failures.  Everything stays inside my little subnet.  I wouldn't want to do it any other way.

[perieanuo]

hi,
second hand servers are cheap, I used to have a dell T110 pretty silent. compairing to this, a workstation is child stuff, not the same endurance.
your 3-4 hdd requirement demands nas or server
ns doesn't take much power and really silent
small mips: choose nas
 higk mips: take sh server
regards, pierre

[paulca]

Well all the parts arrived and I bought 3 x 2 Tb Barracuda drives.

The PC boots, works perfectly.  Might need the fans replaced, but it's still quieter than the current machine.

The USB enclosure, the fan was way too loud.  The airflow non-existant anyway.  I got a Noctua 120mm fan off Amazon and went to fit it today.

Found a surface mount cap broken off and one of the clips on the lid broken.  Looks like transit damage.  So I soldered a through hole 220uF in it's place and wired in the Noctua fan.  Much quieter and I'm still testing thermals with the lid on, versus off, but I expect it will end up off.

RAID.

Pfffffblblblblblb.  I don't know.  I went for 3x in RAID5, then changed my mind and went 2 in RAID1, but I'm still not sure.

Really I'm procrastinating and trying to avoid the migration exercise as long a possible.

Currently the two in raid1 is resyncing the new pair... basically creating a complete copy of a blank drive.  Stupid, but it's testing the thermals nicely.

Code: [Select]

      [=>...................]  resync =  9.7% (189941184/1953382464) finish=186.7min speed=157352K/sec