How to do NAS?

[legacy]

... it uses btrfs (pronounced ButterFS) which is now very solid...

Well, almost. BTRFS will be great but some RAID modes are broken at the moment:

https://btrfs.wiki.kernel.org/index.php/Status
https://btrfs.wiki.kernel.org/index.php/RAID56

Cerberus is running kernel 2.6.39
I guess it's time to update it

[jnz]

I love engineers, but some of you guys go out of your way to make future headaches for yourselves. 

[Monkeh]

I love engineers, but some of you guys go out of your way to make future headaches for yourselves. 

Like what?

[amspire]

The use of btrfs in a commercial product at this stage is.. well, just what I'd expect of Netgear: Stupid.

Don't get me wrong, I like btrfs, but.. it's not ready for that sort of application.

I don't get your point. Who uses btrfs raid56 anyway? Netgear doesn't use any btrfs raid. Netgear froze btrfs at version 0.2 for years until they were confident with the latest updates, so I believe they have been intelligently managing the reliability of their btrfs. They obviously had done a lot of testing on btrfs 0.2.

The really important thing is to get people using NAS boxes with a totally solid daily snapshot option as it is the best way to protect against cryptoviruses and accidental file corruption or deletion. A btrfs NAS is a million times better then a daily data sync to a ntfs/ext4 disk system, a USB drive, or a cloud storage lacking snapshots.

ZFS was great on Solaris, but it is not nearly as stable on linux and freebsd. The ZFS license is incompatible with the Linux GPL license so ZFS can never be fully integrated into Linux. It is vulnerable to powerfail issues on linux or freebsd just like btrfs raid56 is. It is vulnerable to metadata corruption just like btrfs. The solution is to have a proper UPS so the system always shuts down properly and also to provide multiple data backups including off-site backup.

Nothing is perfect, so it is all about  managing risks. Every time you update software in an operating system, a disaster could happen, no matter how "production ready" the software is.

What is your "production ready" solution?

[Monkeh]

It's not feature complete yet. It's still a moving target with new issues appearing and old ones not disappearing.

Compression apparently still has some issues (although I'll be honest, I haven't encountered any lately), quotas have interesting quirks (actually, the entire filesystem has interesting quirks which don't quite align with standard tools), apparently it requires underwear changing when encountering bad sectors (what is this, Windows?).

In its current state I would much rather use ZFS (on Linux or say, FreeBSD) or a more traditional setup over a normal RAID array (no hardware please, I value being able to access my data) and LVM for substantial storage (that is, bulk storage I literally cannot afford duplicates of, which is a reality I'm afraid, as much as people love to deny it).

The really important thing is to get people using NAS boxes with a totally solid daily snapshot option as it is the best way to protect against cryptoviruses and accidental file corruption or deletion. A btrfs NAS is a million times better then a daily data sync to a ntfs/ext4 disk system, a USB drive, or a cloud storage lacking snapshots.

Snapshots are not backups! They're a convenient way of providing history. If the data actually gets corrupted on the disk, you're boned. As for cryptoviruses, the average user of a cheap NAS box has admin access to it: Snapshots won't save you.

btrfs is the future: right now it's an ongoing experiment (one I'm quite impressed with in my usage).

[amspire]

The really important thing is to get people using NAS boxes with a totally solid daily snapshot option as it is the best way to protect against cryptoviruses and accidental file corruption or deletion. A btrfs NAS is a million times better then a daily data sync to a ntfs/ext4 disk system, a USB drive, or a cloud storage lacking snapshots.

Snapshots are not backups! They're a convenient way of providing history. If the data actually gets corrupted on the disk, you're boned. As for cryptoviruses, the average user of a cheap NAS box has admin access to it: Snapshots won't save you.

btrfs is the future: right now it's an ongoing experiment (one I'm quite impressed with in my usage).

I hope I have never suggested in any way a snapshot is a backup. I personally use my NAS boxes as a backup, but if you use it as your working drive, a single NAS box is not a backup. That is why I keep pushing for multiple synched NAS boxes so you do have backups. I do not think one backup is enough. Snapshots mean that there is only one copy of a file but that is the way snapshots are meant to work. The difference of a btrfs file (with snapshots) from a NTFS file is you have one file that contains every daily (or hourly) change to the file and none of the sectors containing the file will ever be deleted until you start deleting snapshots. Even deleting the file does not touch the snapshot copies in any way. The snapshots cannot be changed or deleted through a Windows share access.

Netgear NAS boxes with snapshots do keep previous days/weeks/months data completely safe even when a cryptovirus finds a share for which it has full rights and encrypts the NAS share - I know from direct experience.  It is the best solution for cryptoviruses.

We have also had unhappy sacked staff deliberately deleting all their files before being shown the door. No problem - go to an earlier snapshot and get them all back.

[legacy]

btrfs is the future: right now it's an ongoing experiment (one I'm quite impressed with in my usage).

which kernel version, which configuration and on which system ?
I am really tempted to rebuild my stage4 for Cerberus

[legacy]

yesterday I bought a pair of tulip 1000BASE-T Twisted-pair cabling Cat-6
and a pair of fiber optic Gigabit Ethernet 1000BASE-SX over multi-mode fiber
made by Phobos using a 770 to 860 nanometer, near infrared light wavelength

every node in Cerberus comes with a limited bandwidth due to the PCI
which is 32 bit (instead of 64 bit) and 32 Mhz (instead of 66 Mhz)

Code: [Select]

32 MHz × 32 bits × 1 byte / 8 bits = 128 Mbyte/s

therefore the bandwidth is strictly close to the bandwidth of the controller
with is close to 100Mbyte/sec 

fiber optic Gigabit cards are faster than Twisted-pair when they come
with larger ethernet buffer along with smarter controller aboard
and this is the lucky case, 10% faster, if you buy cards from Phobos

therefore I can reduce the bottleneck of the 100Base-T cards
(Digital is the best producer, in this case) currently used in Cerberus

This standard is highly popular for intra-building links in large office buildings,
co-location facilities and carrier-neutral Internet exchanges.

In my case it's used only for its property of being 10% faster (as controller)
than every 1000Base-T I have ever tried.

I will used a couple of them inside the main nodes of cerberus, node0 and node2
giving them the possibility to "resync" each to the other

externally, to connect Cerberus to the home-lan …well I don't have Cat5 cables
in my house, therefore I am limited to Ethernet 100Base-T (~10Mbyte/s)

but it's Cat6 in the white room where Cerberus silently works,
therefore NFS and CIFS rock, actually  faster than a FSCK 


it's a shame that laptops don't come with a fiber optic plug 

[legacy]

with a "network attached storage"
a part of the property of being fast implies
how fast the network can go

how fast do you expect your NAS should be ?

[ChunkyPastaSauce]

it's a shame that laptops don't come with a fiber optic plug 

https://www.transition.com/products/network-adapters/nec-gxe-xx-01/

[Monkeh]

The really important thing is to get people using NAS boxes with a totally solid daily snapshot option as it is the best way to protect against cryptoviruses and accidental file corruption or deletion. A btrfs NAS is a million times better then a daily data sync to a ntfs/ext4 disk system, a USB drive, or a cloud storage lacking snapshots.

Snapshots are not backups! They're a convenient way of providing history. If the data actually gets corrupted on the disk, you're boned. As for cryptoviruses, the average user of a cheap NAS box has admin access to it: Snapshots won't save you.

btrfs is the future: right now it's an ongoing experiment (one I'm quite impressed with in my usage).

I hope I have never suggested in any way a snapshot is a backup. I personally use my NAS boxes as a backup, but if you use it as your working drive, a single NAS box is not a backup. That is why I keep pushing for multiple synched NAS boxes so you do have backups. I do not think one backup is enough. Snapshots mean that there is only one copy of a file but that is the way snapshots are meant to work. The difference of a btrfs file (with snapshots) from a NTFS file is you have one file that contains every daily (or hourly) change to the file and none of the sectors containing the file will ever be deleted until you start deleting snapshots. Even deleting the file does not touch the snapshot copies in any way. The snapshots cannot be changed or deleted through a Windows share access.

Netgear NAS boxes with snapshots do keep previous days/weeks/months data completely safe even when a cryptovirus finds a share for which it has full rights and encrypts the NAS share - I know from direct experience.  It is the best solution for cryptoviruses.

We have also had unhappy sacked staff deliberately deleting all their files before being shown the door. No problem - go to an earlier snapshot and get them all back.

Current cryptoviruses are essentially unaware of this. How long before they correct that? If the machine is compromised and admin credentials were stored on it, they can quite happily go around deleting snapshots. They may handle accidents and malicious users, they are not the best solution to external attacks.

[Bicurico]

I have had the same considerations as the OP.

I tried FreeNAS (great in general, but could not put USB connected HDD's into standby and uses a filesystem unknown to Windows - in case of problems you are doomed to read the data).

I tried a LACIE NAS (original firmware did not included PnP service, which I need for TV's - so I used an alternative firmware from http://plugout.net/. This is not bad, the authors did a great job, but it is still not satisfactory. The NAS shuts itself down occasionally, the PNP service is primitive, etc.).

I tried a dedicated Atom based PC in a small box, but that meant to have a second PC turned on all the time.

In the end I am using my regular home PC, which is always turned on. This way I can do RDP, have Samba, PNP, etc. I can run servers for HTTP, FTP, etc. HDD's are correctly put into standby. Everything works.

If you want a propper solution: either purchase a propper NAS like the ones from QNAP or build a propper PC.

Do NOT go the route of building some Raspberry Pi or similar NAS! They are not robust and stable. Speed is an issue (even with the Lacie NAS I have).

Regards,
Vitor

[legacy]

it's a shame that laptops don't come with a fiber optic plug 

https://www.transition.com/products/network-adapters/nec-gxe-xx-01/

thanks
do you know if they also exist for pcmcia || cardbus ?

[Dave]

I recently had to figure out this exact problem.
We previously had a WD Mybook Live NAS, which was a complete piece of shit. The transfer speeds were downright awful, you always had to wait for the disk to spin up when you tried to access the files, terrible UI and on top of that, it bricked itself when I tried updating the firmware.
I ended up pulling the HDD out of the enclosure and had to jump through a bunch of hoops to recover the files from the disk (many thanks to the good people on #eevblog for helping me with that).

We finally ended up buying a HP Gen8 Microserver (the exact model that someone already linked in this thread). I added 8GB of RAM for a total of 12GB, a 32GB microSD card for booting the OS (freeNAS) and 4x 3TB hard disks. Two WD Reds and two Seagate NAS. I put them in RAID 10, configured them so that the mirrors are done between different brands. My logic behind it is that if the disks came off the assembly line one after the other, they are going to behave very similarly and likely die at approximately the same time, so you'd want to avoid mirroring data between two of the same model, because if one disk fails, the second one might die during a rebuild and you'd lose your data.

I've had it for a month now and so far it has been working very well. I'm getting around 90 MB/s on a wired connection, but that is very likely the router capping the transfer speed. I'm fairly certain it could saturate the gigabit ethernet connection if the router could handle the load.
It does make a little bit of noise (the disks spinning and a slight fan hum), but you could quite comfortably sleep next to it if you wanted to.

[legacy]

I'm getting around 90 MB/s on a wired connection

impressed by your GigaE throughput

[NiHaoMike]

If you just need one drive, take a look at Pogoplug. You can often find one for $10 or less and it's pretty easy to hack it to add a SATA port and install OpenWRT or Arch Linux.

For multiple drives, an Atom based board is probably going to be the most cost effective. (Only because all the affordable ARM platforms I know of only have at most 2 SATA ports.) Make sure it has AES-NI if you need encryption (probably not a concern for home use) and PCIe to add another SATA controller if you decide to expand it later on.

If you also want to do video transcoding, then you'll have to think beyond a NAS and go with a full fledged server.

[Brumby]

At the outset, my question was really feeling out the DIY ideas as the content is just media files that can be recovered from other sources.  As such, posting it on this forum was more appropriate than on any computer oriented one - but, in this day and age, I was expecting a lot of additional experience and opinion would also be offered.  I was not disappointed - but that is good.

I also appreciate the various input on backups.  While this was not particularly critical for my immediate task, the topic of backups cannot be stressed enough.  I've been involved in DRP with a couple of corporate environments - and they take this very seriously.

As I expected, the extra information offered has been extremely useful in not only establishing the practicality of DIY options, but of the wisdom of doing so ... and that once venturing out into this arena, extending my thinking to take in ideas like a fully blown server have given me a broader outlook and visions of selecting a direction that is not only going to fulfill the current requirements, but put me in an advanced standing for the future.


I am very happy for anyone to offer their experiences.  It is so much more helpful than reading the marketing spiel or specification sheets.  Those are useful - but they rarely tell you about the 'gotchas'.

[Fsck]

You can get a used server for ~400-450$ USD with 12-16 bays using DP westmere or sandy bridge and enough ecc memory (>24GB) to cache. Add 30$ for a connect-x 2, chelsio or other old PCIE gen 2 card and you have a system that can push 10G onto your lan and enough memory to write-cache most personal transfers.

The real question is budget and scalability. Anything short of a "modern" PC-type thing will not be able to push 10G or 40G.

note: I use array to refer to a zpool. (I use 1 vdevs per zpool)

I have 2 NAS boxes, both using freenas and both in 4U 24-bay chassis, (I live alone, but move a ton of data around) and I use 10G connections for the server and my primary/secondary desktops. Laptops and other things are still on old gigabit.
1: i3 2120, 32GB ECC, in a Norco chassis with a normal ATX PSU, kind of struggles with 50TiB of usable storage over 3 arrays under heavy load. Got it in 2012 and it has served faithfully since then.
2: 2*E5-2670 v1, 256GB ECC, in a Supermicro chassis w/ redundant PSUs, got it a few months ago, it's not populated with drives yet, but I have a 16-bay expander chassis that it is hooked up to, migrated from NAS 1.

The hard drives will probably be your largest cost in any given setup.

If you were going dirt cheap. A 200$ old SP sandy bridge/ivy bridge era server will give you 6-8 sata ports and ecc memory. 3TB drives in canada are about 110$ a piece, so 6*110$ for 12TB of storage using raidz2. Or a single 8TB for 350$.


There really isn't any better bang for buck performance than the SB/IVB era servers right now, sometimes you can find a C2750 for cheap but they're quite rare. If your arrays are not too large in capacity and you have enough arrays, you can still push 10G out with an SB/IVB server, and you get ECC to boot.

There's also a crashplan plugin for freenas which makes backup kind of easy for bulk storage. Obviously, if you care about your data, you should mirror it in as many places as you can put it.

So, how much do you care about performance, the availability of your data and future scalability?

[Kilrah]

The hard drives will probably be your largest cost in any given setup.

Indeed... The lack of real progress and price reduction (I swear prices have actually gone up over the past couple of years) is disappointing
I usually would completely replace my drives when needing more space every 3-4 years to start with fresh, unworn ones but this time I've had to think a bit more on a way to cleverly reuse the current ones I have for the upgrade.

My setup was a single 6TB HDD in my main PC, plus 2 of the same (different variants to avoid the batch effect) as external backups with one stored offsite, with both being rotated every week. Given that I have a "main PC" in 2 locations (spending summer/winter in different places due to work) I actually own 4 6TB drives.
I need more space and will stop the 2 location thing, so instead of ditching all 6TB drives and buying 3 10TB ones or so I'll just group them and make RAID0s. I buy 2 new 6TB drives which will be the (now single) main PC's internal 12TB RAID0 array, and the existing 4 will get put in 2 external RAID0 enclosures for the rotating backups. So 2 drives and 2 external cases to buy, which is still a bit more than $1k...

[Kalvin]

I am worried about physical or electrical failure of the NAS device, you know like power supply failure or something else which will destroy the disks. Also, I have a limited budget to spend. That is why I am thinking about using a physically distributed NAS so that the data is being mirrored and duplicated across few cheaper devices. One of the devices will be the main server, but there will be one or more cheaper servers [with less performance] which will be used as redundant backup device(s). The cheap backup devices could be as simple as Banana Pi with Gigabit Ethernet and a Sata hard disk. With two Banana Pi based, cheap backup devices there would be quite good redundancy should any other device fail. The network is not a business environment but only a home environment with modest need for the throughput but I'd like to see that my data is safe and it will survive major hardware failures. Any comments on this?

[Kilrah]

As said what's currently expensive is usually the hard drive itself more than whatever you use to access the data on it. If you want (and you DO) want backups it's going to cost, no miracle.

[Bicurico]

Kalvin: What you describe is not a backup solution and makes no sense at all.
A NAS provides storage spare. It should be fast and reliable.
To have backups, you need to periodically save the data to a storage which is kept in a safe place. These storages, tapes or HDD, are cycled. Depending on your work, you make a new daily backup, cycling through 5 different takes or disks. After rotating a storage a given amount of times it should be permanently stored.
Sounds too complicated? Give each worker an external HDD to do their own backups.
But remember that a backup is never permanently hooked up.

Regards,
Vitor

[amspire]

Any form of copy is a backup whether online or not.
Having a single backup or onsite backups only is a dangerous strategy.

For individuals or small businesses, multiple Nas including one off site backup can be a good strategy. You can be continually monitoring the health of all the Nas boxes so you can fix problems as soon as they happen. It is also easier keeping the technology up to date over time.

It all comes down to having a good strategy. For some people, offline backups with backup media taken offsite is the best solution. For other people, it may not be the best choice

[Bicurico]

Latest ransomware encrypting networkshares have proven how bad a NAS is as a backup strategy.

Regards,
Vitor

[Kalvin]

Latest ransomware encrypting networkshared have proven how bad a NAS is as a backup strategy.

Regards,
Vitor

What do you think, would the ZFS file system with snapshots have helped?