Products > Computers
LTO tape usage (modern tape drives)
exuvo:
Seeing as there was some interest in LTO on the forum i will start a thread with my journey using it. All costs include 25% VAT and transport as i live in Sweden. Almost everything bought on ebay.
I recently bought a refurbished tape library (autoloader) Dell PowerVault TL2000 on ebay for $194 and a used LTO-5 SAS tape drive for $199.
First problem: the drive is a HP drive and due to vendor lock in HP drives only work on HP libraries. My options were now to buy a HP library or a IBM drive (resold by Dell and Lenovo).
For some reason IBM drives cost 3 times as much as HP drives with equivalent specifications. So i bought a refurbished HP Storever MSL2024 library for $487 (starting to go downhill here :D) which was cheaper than a refurbished IBM LTO-5 drive at $800. The refurbished libraries came with 6 months warranty.
Both of these libraries are almost identical (dell one has a 4 line graphical display, hp has text 2 lines) with 24 slots for tapes split into 2st 12 bay sides that can be easily removed for bulk changing of tapes. There are also 4U models with 48 slots for just a bit more money.
Both my HP and Dell libraries have good web UIs. I suggest separating them with firewalls and/or VLAN as they probably have security holes. Or just don't connect the network cable for normal use.
If you want to spend way more money LTO-6 drives can be had for around $580 for a used HP and $1470 for IBM. They only store 1TB more than than LTO-5, 1.5TB vs 2.5TB, and are not worth it in my opinion.
I you somehow get your hands on a LTO-7 drive for less than a kidney they can store 6TB and have twice the speed (300MB/s) as LTO-5/6 (140/160MB/s).
For tapes i bought a new cleaning tape for $22 (50 uses), 10st new LTO-5 tapes from china for $225 and 100 used LTO-5 tapes for $490.
Each LTO-5 tape is guaranteed to work for at least 200 full writes.
My use case is monthly rotating backups on the new tapes, and archival backups on the used tapes (i.e. write a few tapes once a year and never reuse).
Archival backups are safe from crypto viruses and accidental deletes, rotating backups are mostly safe from crypto viruses and accidental deletes.
For tape barcodes the new tapes came with a sheet (100x96mm) of 4st empty stickers each. Used tapes already had stickers. Using the program glabels on linux i was able to print onto these with my laser printer (just about minimum paper size it works with, must go short side in first or it gets stuck). Use barcode style 39 in glabels, barcode size is 65.36x11.1mm, sticker size is 78x16mm, center barcode horizonally at the bottom and put whatever text you want above. Printing settings were paper type labels and monochrome i.e. black only as the text did not stick well if it used color. Glabels supports generating from csv files which makes it easy to print many.
I also bought an old 3G/s SAS 2xSFF-8088 PCIe x4 card for $46 and a 1m SAS cable for $30. 3G/s should be good up to LTO-6. LTO-7 is much faster and needs a 6G card to work at full speed.
The LTO drives have incredibly noisy tiny fans, i changed mine to a larger quieter fan using a 3D printed adapter, see https://www.eevblog.com/forum/repair/replacing-the-cooling-fan-in-an-lto-2-tape-drive/msg4021672/#msg4021672 . Fan noise is barely audible now. Drive makes some noise when running but quite pleasant tones so i don't mind :D
Firmware for these things is a bit difficult to get.
IBM requires a service account but dell and lenovo use the exact same drive firmware and have it for free. My refurbished libraries were already up to date.
HP has a few years old firmware for free but i found the latest one on a shared google drive.
PM me if anyone wants copies, i got libraries and drives LTO-5 to LTO-8.
Programs i used on arch linux: mtx-git mt-st-git mbuffer amanda hpltt (itdt manually downloaded for IBM drives) tapestat (from package sysstat) maminfo-git
mt controls the drive and tape positioning, only useful if you want to do things manually. Point environment variable TAPE to your tape drive to ease mt usage.
mtx controls the library and allows listing tapes and loading/unloading into the tape drive.
mbuffer provides large RAM buffering so the drive does not stall if disk IO is intermittently slow.
hpltt or itdt is used for drive diagnostics and firmware updating (firmware can also be updated from the libraries web UI).
tapestat -m -1 shows tape read and write speeds.
maminfo shows tape cartridge stats like times loaded and total data written/read. Good if you buy used tapes to know how much life they have left.
Basic usage: https://www.cyberciti.biz/hardware/unix-linux-basic-tape-management-commands
I access my drive via /dev/tape/by-id/scsi-HUE2530E22-nst and the library via /dev/tape/by-id/scsi-DEC8320699 .
You can also use /dev/nst0 for drive and /dev/sgNumber for the library (if you see /dev/chNumber instead of sg then blacklist module ch).
For manual backups and restore i use:
backup: tar cf - /aDirectory | zstd -qc -3 -T0 | openssl enc -pbkdf2 -e -aes-256-ctr -salt -pass fd:3 3< passwordFile | mbuffer -f --tapeaware -m 2G -L -P 80 -o $TAPE
restore: mbuffer -i $TAPE -s 1024k -b 2048 -L | openssl enc -pbkdf2 -d -aes-256-ctr -salt -pass fd:3 3< passwordFile | zstd -dqcf | tar -xf -
Amanda is a pretty old software (development started in the 80s) for tapes which is very noticable in that all the default settings are absolute garbage for modern hardware. The wiki is a mess but man pages are good.
But the software works well, it can do multi tape backups, networked backups (its own client or samba), (multi level) incremental backups, has good reporting and keeps a local file list so you can easily find the correct tape to restore from if you only need to restore a few files.
I will attach sensible settings that use aes-256-ctr and zstd compression. Put them in /etc/amanda/
A temporary fast storage disk can be useful if the disk you are backuping from is way too slow to write straight to tape (my LTO-5 drive seems to handle write speeds down to 50% of max speed just fine) or if you are doing many small backup sets that would make the tape drive start and stop a lot between them.
To start a backup i run "sudo -u amanda amdump archive localhost" or in user amanda's crontab "/usr/bin/amdump backup localhost; /usr/bin/amtape backup eject".
When it is done i get a email with which tapes it wrote to and other details. For the archive backups i then remove the right 12 slot bay and remove the tapes it used and put in new ones.
To register the new tapes i run:
single tape: sudo -u amanda amlabel --barcode ROT001L5 backup ROT001
multiple tapes: for i in {2..9}; do sudo -u amanda amlabel --barcode ROT00${i}L5 backup ROT00$i; done
Replace ROT001 with whatever the new tapes are called. My own printed tapes i use ROT01 to ROT09. My used ones all start with A and then 5 numbers.
This writes a small header to file 0 on the tapes marking it as a amanda tape registering it to a specific backup config so it does not accidentally get overwritten by amanda if you do something stupid.
Add a -f if a tape was previously used in an amanda config and you are sure nothing important is on it.
Amanda backups can be recovered easily without the software itself.
Position tape at the correct file using "mt fsf and mt status", file 1 if you only did a single backup set (1 DLE).
First block is text information from amanda about what this backup is and how to restore it. Either skip this block with mt fsr or read it "dd if=$TAPE bs=1024k count=1 of=header".
You should now be at block 1 according to mt status, everything following is the backup data, assuming you used my settings you can restore with the same line as up above or:
mbuffer -i $TAPE -s 1024k -b 2048 -L | /etc/amanda/encrypt -d | /etc/amanda/zstd-compression -d | tar -xf -
which uses the password stored in /var/lib/amanda/
Skip zstd-compression if you did not compress this backup (the header will tell you if you did).
Slight security warning here is that the header leaks some details about your backup like the backup name (DLE name) and settings used. So maybe don't label the backup my-banking-passwords if you care about not making it easier for physical attackers.
Another alternative to amanda is bakula but i have not tried it.
Assuming i used all my tape storage, 150TB used tapes and 15TB new, the cost is $11.14/TB including the cost for the extra library i don't use. Sum total i have spent is left as an exercise to the reader as i dare not write it T_T. Planning to allow friends to do archival on the used tapes if they want for a small cost.
-edit-
Had a parsing error in amanda.zip/backup/disklist on line 3, moved "tar-comp" down a line. Zip updated.
-edit2-
Updated config to include amanda-client.conf used for restores via amrecover. See https://www.eevblog.com/forum/general-computing/lto-tape-usage-(modern-tape-drives)/msg4025251/#msg4025251 for usage.
Added tapestat and maminfo to list of useful programs.
-edit3-
Fix wrong dumptype used for raw disks in example config.
EHT:
Thanks for starting this thread exuvo!
I have somewhat similar to you. Its not easy but far superior than what I had before – an assortment of crappy consumer grade external HDDs.
Reason I chose this solution
• I have 10s of TB of data. Cloud storage for this is a significant cost. Cant remember the details though
• I realised drives such as the WD ones use the lowest grade consumer drives (WD Green etc) which are not designed for long term use and will fail. Happened to my external HDDs and internal non-enterprise HDD & SSD, so I got tired of this.
My setup
• I have a multi-TB RAID5 setup with Adaptec cards, which I backup from and also offline some data to save having to keep growing the arrays
• I have a Quantum Superloader3 LTO-5, 16 tape library in 2U rack connected to HBA in the same machine as the RAID arrays
• I take two backup copies with one periodically stored elsewhere
Problems:
• Non-enterprise use of LTO is uncommon. For this reason, I couldn’t find a working cheap/free software backup solution which would properly support the tape library since these are enterprise devices, and enterprise backup licences are multiple $k. I will check out the above recommendation. Good to start this thread!!
• The LTO and HBA drivers are not supported on Windows 10, only Windows server. I need to shift it all to Linux but that is a rainy day project I haven’t got around even with COVID lockdown! I presume the tapes, being LTFS formatted can equally be read/written from Windows or Linux.
• If you’re not careful accessing the drive you can end up grinding to a halt seeking. If you try to move a tape whilst it is not unloaded, the loader still tries and then jams.
• Despite the claimed longevity of the media there is a nasty built-in obsolesce. LTO-5 is not that old, but on its way out. The newer LTO-7 drives won’t write to it. For example, in the other thread, the OP had LTO-2. That means they cannot upgrade beyond LTO-4 to be able to read the existing tapes, otherwise have to go through and duplicate all of them using the old & new drives together. Not a problem if these are purely a backup though.
• I’m not sure what is involved in upgrading the drive in the Quantum tape loader as there is some software / firmware dependency. I guess I’ll wait until LTO-7 is cheap.
• Media is quite expensive, especially since I’m doing a double backup. Buying used tapes is hit/miss.
• It is noisy – both the fan and the rotation of tapes. Still, I don’t leave it on all the time so haven’t bothered thinking about changing the fans.
exuvo:
Thanks for your reply.
--- Quote from: EHT on February 21, 2022, 11:05:34 pm ---I presume the tapes, being LTFS formatted can equally be read/written from Windows or Linux.
--- End quote ---
Yes on arch linux there are packages in AUR named "ltfs" and "ltfs-quantum" (i think quantum has their own variant) and i found some helper scripts at https://github.com/amiaopensource/ltopers but i don't think i will use LTFS for anything as amanda backup software is working well for me.
--- Quote ---• If you try to move a tape whilst it is not unloaded, the loader still tries and then jams.
--- End quote ---
This probably depends on the library, my HP MSL2048 unloads from the drive first and then moves it away. Never managed to make it jam.
--- Quote ---• Despite the claimed longevity of the media there is a nasty built-in obsolesce. LTO-5 is not that old, but on its way out. The newer LTO-7 drives won’t write to it. For example, in the other thread, the OP had LTO-2. That means they cannot upgrade beyond LTO-4 to be able to read the existing tapes, otherwise have to go through and duplicate all of them using the old & new drives together. Not a problem if these are purely a backup though.
--- End quote ---
While true it is not much of a problem for me as I will renew my backups yearly anyways. If you just want to keep the possibility to access old tapes i say just keep the old drive next to the new one. Most if not all libraries has place for multiple drives and even if not they are quick to change.
--- Quote ---• I’m not sure what is involved in upgrading the drive in the Quantum tape loader as there is some software / firmware dependency. I guess I’ll wait until LTO-7 is cheap.
--- End quote ---
Unfortunately i don't have any quantum firmware but i assume the process is similar to mine. Log into the library with the service account using the web interface and there should be menus for updating both the library firmware and connected drives. From the various change logs i found as long as your library firmware is from after when that LTO generation was released it should support it.
--- Quote ---• Media is quite expensive, especially since I’m doing a double backup. Buying used tapes is hit/miss.
--- End quote ---
I seem to have been lucky with my 100 used tapes buy, using https://github.com/arogge/maminfo i was able to read the EEPROM inside a cartridge and it said one of them had only been loaded 15 times and barely written to. Assuming the whole lot is similar i have at least a hundred uses left on each.
Where do you buy from? Even the new cartridges i bought are still half the $/TB price of CMR hard disk drives i could find (i would not want to use SMR in RAID).
--- Quote ---• It is noisy – both the fan and the rotation of tapes. Still, I don’t leave it on all the time so haven’t bothered thinking about changing the fans.
--- End quote ---
Check out the fan replacement i did in the other thread, it really helped for me and was easy to do (assuming you have a 3D printer). I can leave it on all the time now if i want to without it disturbing and i sit 1m away from it daily. The only too noisy part left is when it changes tapes but that is a once every 3 hours during writing and i can live with that. The noise while writing or reading i find is an acceptable level/sound to not be distracting.
You could also just buy long cables for it. I saw SAS cables that were 10 meters for sale.
exuvo:
I only have around 10TB i need backed up so this is not really a cost effective solution for me from that standpoint but i really want an offline backup and this seemed the best option.
Tapes are very resilient to bumpy transports which is good as i want to keep the archive backups in a distant building and cheap enough that i don't need to bring them back for reuse.
Theft risk, if i for example stored my archives in my summer house, would also be much lower than with hard disks as the thief would not recognize them as valuable/resellable nor easily be able to read them even before encryption comes into play. Obscure technology has its advantages.
LTO tapes are mostly sealed and if you keep them in the tight plastic case they come in i think they would be almost immune to humidity problems.
jh15:
Thanks for starting the thread I tried a few years ago, in my linux forum. I can ride on your shoulders. Will help others. Now to read the above posts, wanted to say thanks while you were online. It is quite complex, even for an old tech/engineer. Prefer Linux, but keep a windows drive around for ease of use.
Navigation
[0] Message Index
[#] Next page
Go to full version