A few years ago I decided I needed to implement a proper backup solution for my 72 TB Unraid server. While some of the data (the irreplaceable critical stuff) was already backed up elsewhere in multiple locations, I had well over 65 TB of entertainment media that would be a total pain to recreate from original discs. Additionally, with the rise of ransomware that encrypts your data, I wanted an offline backup that would be safe from savage beasts.
My original idea was to use tape backup. In the 90's I had a tape backup solution, and while it was a hassle it was very cost effective. I also used to work in an IT operations center, and we always did backups to tape. I figured it would likely still be one of the cheapest options today. Surprisingly, tape backup solutions large enough to back up this amount of data are prohibitively expensive.
My next idea was to stand up a 2nd Unraid server. But this seemed like an overkill solution, as I would have to buy the entire server, plus the storage media. Cost was more than I wanted to outlay, portability seemed a challenge, and in general I wasn't crazy about supporting an additional Unraid server just for backups.
What I really wanted was just a single large external HDD, but they aren't quite big enough yet. The best I could do in 2020 was a 16TB external USB drive, a far cry from what I needed. I figured I could buy several of these, but juggling drives to do a backup seemed like a real chore. There had to be a better way.
And there is.
I discovered on the Unraid forums that it is easy to use Unassigned Devices to create a drive pool that combines the drives into a very large storage array. Similar to an Unraid array but without the parity protection, but it provides the ability to plug it in and remove it as needed. This sounded perfect. It would allow me to easily back up my entire array using the cheapest possible solution. And as my needs grow, I can expand the pool with new drives.
So I bought 5 external Seagate 16TB USB drives, which would give me an 80TB storage pool. I was able to buy these for $310 each, which was an amazing deal in the summer of 2020. I also bought a nice 7-port switchable USB hub, which allows me to connect or disconnect each drive at the press of a button, some short mini-USB 3.0 cables, and a more powerful 10A 12V power supply brick that output the correct voltage for these drives, plus had enough power for at least 7 drives.
I went the extra mile, and designed and 3D-printed some drive stackers that allowed me to use zip-ties assemble everything into a single stack for portability, while allowing for some airflow between the drives. I also 3D printed a small project box style enclosure to hide all the power connections, since I cut the power cords for all 5 drives and connected them to the single larger power brick.
Here's the end result:
It's not pretty, but barebones form follows barebones function.
Here's a link to the USB hub I used. It looks exotic and expensive, but it's actually a good value:
https://www.amazon.com/gp/product/B086S ... UTF8&psc=1
I can unplug the single USB cable and power cable, and carry the entire unit quite easily, and store it in a safe spot. When I need to do a backup, I plug the USB cable into my Unraid server, power it up, and sequentially press the buttons on the USB hub to do a manual staggered spin up from bottom to top (not sure if that matters, but it's what I do to ensure the drives always connect in the same order, in case that affects anything, and to minimize power draw on spin-up).
The drives appear in Unassigned Devices. The first time I had to create the pool manually, but now that always shows back up each time they are plugged in, so all I have to do is mount them.
You can ignore the first drive in the picture above, that's my 4TB video surveillance drive. Dev 2-6 are the 5 drives for my backup solution. I named the main drive Frankenstore, and then each additional drive with the drive number, i.e. Frankenstore_D2. The pool data lives on the main Frankenstore drive, so when I hit Mount it mounts all 5 together into the 80 TB large pool. When I'm done with the backup, I click Unmount, turn off the USB ports, and unplug for storage.
I did notice that the drives got too warm for my liking, so I currently have a fan aimed at the drives while doing a backup, which dropped temps tremendously. At some point, I plan to 3D print some brackets and mount a couple 120mm fans permanently as part of the solution.
The last part of the setup is the backup script itself. To keep the backups as quick as possible, I only want to do a differential backup (only back up what has changed since the last backup). I already had the basis of a backup script that I used to backup my cache drive's contents to the array, so I copied that and used it to write my array backup script, below.
Code: Select all
#!/bin/bash
LogFile=/var/log/array_backup.log
BackupDir=/mnt/disks/Frankenstore
Notify=/usr/local/emhttp/webGui/scripts/notify
echo `date` "Starting Array backup to " $BackupDir >> $LogFile
#Backup 4K via rsync
$Notify -i normal -s "Beginning 4K Backup" -d " 4K Backup started at `date`"
rsync -avrtH --delete /mnt/user/4K $BackupDir >> $LogFile
sleep 2
$Notify -i normal -s "Finished 4K Backup" -d " 4K Backup completed at `date`"
sleep 2
$Notify -i normal -s "Beginning Blu-Rays Backup" -d " Blu-Rays Backup started at `date`"
rsync -avrtH --delete /mnt/user/Blu-Rays $BackupDir >> $LogFile
sleep 2
$Notify -i normal -s "Finished Blu-Rays Backup" -d " Blu-Rays Backup completed at `date`"
sleep 2
$Notify -i normal -s "Beginning DVDs Backup" -d " DVDs Backup started at `date`"
rsync -avrtH --delete /mnt/user/DVDs $BackupDir >> $LogFile
sleep 2
$Notify -i normal -s "Finished DVDs Backup" -d " DVDs Backup completed at `date`"
sleep 2
$Notify -i normal -s "Beginning TV_Series Backup" -d " TV_Series Backup started at `date`"
rsync -avrtH --delete /mnt/user/TV_Series $BackupDir >> $LogFile
sleep 2
$Notify -i normal -s "Finished TV_Series Backup" -d " TV_Series Backup completed at `date`"
## RESTORE
## /usr/bin/rsync -avrtH --delete $BackupDir /mnt/cache/
echo `date` "backup Completed " $BackupDir >> $LogFile
# send notification
sleep 2
$Notify -i normal -s "Array Backup Completed" -d " Array Backup completed at `date`"
Most of the script is variables, sleep statements, and notify commands (so I see progress notifications in the Unraid GUI). The sleeps statements are necessary to ensure the notifications works, since Unraid seems to miss displaying notifications if you send them too quickly. That means the actual backup statement is just those "rsync" line, and I do that once for each of my shares that I backup.
The rsync options I pass, -avrtH, are for archive (backup mode), verbose (more logging output), recursive (subdirectories), times (preserves the original modification times instead of changing them to the backup time), hard-links (which preserves any hard-links that exist). The --delete option gets rid of any previously backup up files that are no longer in the source, so I'm replicating the file deletions in my array to the backup, which may not be to everyone's preferences but matched my goals. Everything gets logged to the $LogFile.
The RESTORE command line is commented out, and only serves as a reference for how I could do a restore. That's actually a command for my cache backup restore, so I would obviously have to rewrite it to use the correct source and target if I was restoring one of my array shares. If I actually had to do a restore, I would probably just run the command directly, instead of running it inside a script like this.
The btrfs drive pool I created is not striped or RAID'ed in any way, making it very similar to Unraid in that each file only exists on a single drive, and if I loose a backup drive I only loose the data that was backed up to that drive, plus potentially 1 or 2 files that happened to be spanning the gap from one drive to the next. If I do lose and replace a drive, with the way the rsync works then the next backup will simply re-copy those files that no longer exist in the backup, making it very easy to maintain this backup solution.
For me, the hardest part of adding a new drive to expand this pool will be 3D printing a couple more stackers, cutting the zip ties, inserting the new drive, and pushing new zip ties through the 3D-printed channels to hold everything together. Then I just plug in a USB cable, splice in the power cable to my power distribution bus, and finally add the drive to the pool. I'm extremely comfortable doing electrical wiring, if you're not then you can always just use the individual power plugs that come with these drives.
I will probably be expanding mine within the next 6 months, as I'm quickly running out of backup space. When I do that I'll take some more pictures showing the process.
Even with my "luxury" upgrades to do the single power supply, switchable USB hub, and 3D printed stacking, my total cost for this backup solution is just under $1600. That's only $20/TB, which is dirt cheap for a backup storage solution, and quite a bit cheaper than my Unraid setup. On average that's about $1 per 4K UHD Blu-ray, $0.50 for regular Blu-ray, and a nickle per DVD. But the best part is knowing I have an actual backup, which is priceless...