Linux Print Server/Storage Server

[agehall]

And before you do this and put something that you might want to access tomorrow on your new storage server - MAKE SURE YOU USE A RAID SETUP!

I'm not going to argue which RAID level/setup you should use (well, don't use RAID0 - but that should be obvious) but use one! Otherwise the harddrive WILL fail and you WILL lose your data. From experience, I also know that in home systems, you might want to have a bit of extra redundancy as you may not always have spare drives to replace failing drives right away.

If you are serious about doing this, I would highly recommend looking into ZFS. (I guess BTRFS is maturing as well, but I've never used it, so I can't speak for it)

Sorry for all the caps, but this stuff is important if you care about your data.

[stevelup]

RAID is far less useful than a robust backup strategy.

[Naguissa]

RAID is far less useful than a robust backup strategy.

They are not the same. RAID (except 0) is for performance and availability (you can delegate maintenance when a disk fails). Backup is for security.

Maybe RAID gives you a little more of security compared to a single disk, but still is not a backup strategy and never should replace it.


Said that, most people are lazy (me included) and our home backup strategy is defficient. Being so, RAID, cloud and so one mitigates a little bit (VERY LITTLE!) our lazyness....

[agehall]

RAID is far less useful than a robust backup strategy.

Depends on the scenario tbh. I have about 1TB of data that I just cannot lose. If it is destroyed, it would be a disaster. That data is backed up in multiple ways and the chances of it getting lost is virtually nil. The rest of my data, which would be painful to lose but not a total disaster, (somewhere in the neighbourhood of 30TB) is not backed up but resides on a raidz2 pool with two spare drives in the pool. I would like to move up to a raidz3 pool, but I don't have the gear to do it (yet!). With raidz2 and spare drives, I feel rather comfortable not backing it up.

The main problem with backups is that most people tend to get sloppy at home and when disaster strikes, you realize that your backup is 6 months+ old. With a RAID, you normally have a chance to act and recover before the total disaster is a fact. But it's all a balance act - if you just can't lose data, backup and do it to multiple destinations/media.

[stevelup]

RAID won't help you if you do rm -rf / as root

[Naguissa]

RAID won't help you if you do rm -rf / as root

Also on criptolockers (as NAS is also encrypted), or accidentally deleted // overwritten files....

BUT, it protects a little against disk failures.

[agehall]

RAID won't help you if you do rm -rf / as root

ZFS snapshots will though. They are cheap and fast and I tend to use them a lot as a backup light to avoid stupid mistakes like that.

[sleemanj]

Since we have drifted to backup systems.  Let me put in a vote for borg.

https://borgbackup.readthedocs.io/en/stable/index.html

Here's my stats for the latest (nightly) backup of my system and home data...

                       Original size      Compressed size    Deduplicated size
This archive:              368.43 GB            199.56 GB            321.68 MB
All archives:                4.42 TB              2.39 TB            160.62 GB

The current raw-data size is 368.43 GB, but only takes 160.62 GB of backup disk space because borg compresses and deduplicates (at a chunk level not file level, so it's good for things like vmware drive images as well as long as the machines are powered off when you backup them of course for consistency), and in that is for 12 incremental but effectively independent archives totalling 4.42TB.

Any one of of those archives I can mount as a filesystem and pull stuff out of if I so choose, and it's pretty fast to backup once the initial archive is done.

I used to use rsnapshot, but it was soooooooo slooooooow and used way too much space.  Borg can do in 3 hours what rsnapshot was taking 12 to do, in a third of the backup disk space, for about twice as many archives.

[agehall]

What kind of data are you backing up using that? I've never really seen deduplication make such a difference in real life...

[alm]

I used to use rsnapshot, but it was soooooooo slooooooow and used way too much space.  Borg can do in 3 hours what rsnapshot was taking 12 to do, in a third of the backup disk space, for about twice as many archives.

I wonder about a system like that on a CPU and memory constrained low-end SBC. I use the rsync-based backuppc on a quad core Xeon (Core 2 generation) server with 16 GB RAM, and even that is not particularly fast. Now I am willing to believe that Borg is more efficient, but a Raspberry Pi is probably more than an order of magnitude slower than even an old Xeon.

[bitman]

And before you do this and put something that you might want to access tomorrow on your new storage server - MAKE SURE YOU USE A RAID SETUP!

Let's note that this question was about a small embedded device like a Raspberry PI and not a full blow computer with 6-8 parallel IO SATA channels available.  Heck, some options just have USB available. So RAID by any means is not an option here. The hardware we're talking about can barely keep up with a single IO channel. 3-5 parallel channels? No way.  REAL LIFE file servers of course have dozens of disks and RAID is one of the many features that allows optimal IO and redundancies to be build into the hardware. But not here - not this example. Just running a single small USB stick is about the expectations here.

Also, as other posts here sorta indicates, RAID isn't "backup". Deleting a file still gets deleted. If you accidentally delete a directory full of thousands of files, they too are deleted. RAID or no RAID. Gone. Linux comes with quite a few tools like LVM and Dejavu to deal with this, but again for a small embedded device that seems overdoing/overtaxing the hardware we can get.

Bottom line is, that for REAL file servers you need more hardware than laid out. On those you can do all the stuff suggested here, and a lot more (look at FreeNAS for instance), and you can make it really smooth so you can loose whole hosts/nodes and not loose availability, automatic cloud backup etc. etc.  There are appliances today for consumers that do this (most are Linux based) and for people who just want it to work, that's where I'll go. Plug-in and forget. Cheaper than trying to build it yourself, smaller form factor etc.

As to designing your own NAS, there's plenty of options. Gluster is one of them which I mentioned which does striping/mirroring over multiple hosts as well as disks. XFS/GFS have very interesting file system features for file servers, LVM can handle adding/removing capacity to a running system, consistent snapshots, cached block devices etc. etc. - and with a good web frontend you have a very powerful set of servers that can compete with enterprise/cloud solutions you pay a lot of money from. But it will cost you to get there in hardware alone.  It is probably cheaper to just use a cloud provider as your long term storage provider and just download/mirror the data you work on temporarily from the backend storage, automatically sync that across devices etc - and no major hardware investments are needed at all.

Finally, RAID isn't always good. It can make performance worse and make recovery quite complex.  I've seen lots of systems running in degraded mode using software raid because it wasn't configured to notify anyone, and the admin didn't know how to tell what the state was, meaning it was actually preforming slower than a single disk would.  So it's not a simple choice.

[sleemanj]

What kind of data are you backing up using that? I've never really seen deduplication make such a difference in real life...

/usr/local, /etc and /home, minus vmware images, there is a fair amount of repetition in some of it to be fair (svn repository branch checkouts)

My vmware (disk/suspended state) images dedup by about 50% as well but in a different backup set

Borg's dedup is really good, it works on a chunk level, so changing for example 1 byte in a file doesn't mean the entire file changed, only a chunk.

[eugenenine]

For someone not familiar with linux, install http://www.webmin.com/ on your server.  It gives a simple web interface that can configure samba, nfs, etc, then you can look at the config files it created to learn how its done.

I've used a Pi for a long time, one for my internet server with 1TB drive hanging off of it and another for my web facing server running owncloud.
I wanted a little more performance so little by little I watched for sale prices and put together a $200 system in a small mini-itx case.

[Naguissa]

I link a thread I wrote for that purpose:

https://translate.google.es/translate?hl=es&sl=es&tl=en&u=https%3A%2F%2Fwww.foroelectro.net%2Felectronica-digital-raspberry-y-otros-socs-f9%2Fnas-casero-barato-y-bajo-consumo-con-un-soc-banana-t162.html

FS may be opinable (RAID + ext4 or XFS, reiser, LVM....), but overall, good price / performance / options ratio.

[agehall]

Finally, RAID isn't always good. It can make performance worse and make recovery quite complex.  I've seen lots of systems running in degraded mode using software raid because it wasn't configured to notify anyone, and the admin didn't know how to tell what the state was, meaning it was actually preforming slower than a single disk would.  So it's not a simple choice.

Well, consider that case without a RAID setup - your NAS would be dead as the drive. There is no such thing as a fire and forget solution that just works once set up - everything breaks eventually and all we can do is mitigate the effects and delay the event itself.

Granted the hardware suggested by the OP isn't anywhere near enough to run a proper file server, I still think it's appropriate to think about redundancy for the spinning rust. Otherwise a USB drive is probably a better solution imho.

[bitman]

Finally, RAID isn't always good. It can make performance worse and make recovery quite complex.  I've seen lots of systems running in degraded mode using software raid because it wasn't configured to notify anyone, and the admin didn't know how to tell what the state was, meaning it was actually preforming slower than a single disk would.  So it's not a simple choice.

Well, consider that case without a RAID setup - your NAS would be dead as the drive. There is no such thing as a fire and forget solution that just works once set up - everything breaks eventually and all we can do is mitigate the effects and delay the event itself.

If the assumption is that the alternative is no redundancy, then yes you're absolutely right. With my rather long entry I was trying to explain that md/dm raid isn't the only option here. There are plenty of other ways to create redundancy across multiple devices.

Granted the hardware suggested by the OP isn't anywhere near enough to run a proper file server, I still think it's appropriate to think about redundancy for the spinning rust. Otherwise a USB drive is probably a better solution imho.

Exactly - the idea that you would use a PI as a target for VM image backups just bugles the mind. You don't have to spend thousands to get this done, but it will be more than $60 to get the hardware (the 3-5 disks alone will be far more).

[rdl]

Do you have a real need for a print server? I have an old Brother laser printer that's plugged directly into a wireless router. All my computers can print to it. I use FreeNAS for file storage. Disks are in mirrored pairs and the data on each pair is duplicated on two other machines.