Decorum for the Forum:
  • Be nice. If you want to be mean, try Reddit.
  • No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
  • No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
  • No Spamming. If you want to make a buck, work smarter... somewhere else.
  • No Adult Content. Half the internet is dedicated to adult content. This half isn't.

Privacy Policy: Click Here to Review (updated September 30, 2020)

My Unraid NAS Backup Solution

Show off your HTPC builds, NAS Servers, and any other hardware. Great place to ask for hardware help too.
Post Reply
User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

My Unraid NAS Backup Solution

Post by Pauven » Wed Apr 06, 2022 7:05 pm

I thought I had posted about this before, but I guess I never got around to it. Here goes, hope it helps someone.

A few years ago I decided I needed to implement a proper backup solution for my 72 TB Unraid server. While some of the data (the irreplaceable critical stuff) was already backed up elsewhere in multiple locations, I had well over 65 TB of entertainment media that would be a total pain to recreate from original discs. Additionally, with the rise of ransomware that encrypts your data, I wanted an offline backup that would be safe from savage beasts.

My original idea was to use tape backup. In the 90's I had a tape backup solution, and while it was a hassle it was very cost effective. I also used to work in an IT operations center, and we always did backups to tape. I figured it would likely still be one of the cheapest options today. Surprisingly, tape backup solutions large enough to back up this amount of data are prohibitively expensive.

My next idea was to stand up a 2nd Unraid server. But this seemed like an overkill solution, as I would have to buy the entire server, plus the storage media. Cost was more than I wanted to outlay, portability seemed a challenge, and in general I wasn't crazy about supporting an additional Unraid server just for backups.

What I really wanted was just a single large external HDD, but they aren't quite big enough yet. The best I could do in 2020 was a 16TB external USB drive, a far cry from what I needed. I figured I could buy several of these, but juggling drives to do a backup seemed like a real chore. There had to be a better way.

And there is.

I discovered on the Unraid forums that it is easy to use Unassigned Devices to create a drive pool that combines the drives into a very large storage array. Similar to an Unraid array but without the parity protection, but it provides the ability to plug it in and remove it as needed. This sounded perfect. It would allow me to easily back up my entire array using the cheapest possible solution. And as my needs grow, I can expand the pool with new drives.

So I bought 5 external Seagate 16TB USB drives, which would give me an 80TB storage pool. I was able to buy these for $310 each, which was an amazing deal in the summer of 2020. I also bought a nice 7-port switchable USB hub, which allows me to connect or disconnect each drive at the press of a button, some short mini-USB 3.0 cables, and a more powerful 10A 12V power supply brick that output the correct voltage for these drives, plus had enough power for at least 7 drives.

I went the extra mile, and designed and 3D-printed some drive stackers that allowed me to use zip-ties assemble everything into a single stack for portability, while allowing for some airflow between the drives. I also 3D printed a small project box style enclosure to hide all the power connections, since I cut the power cords for all 5 drives and connected them to the single larger power brick.

Here's the end result:

image.png
image.png (938.04 KiB) Viewed 18781 times

It's not pretty, but barebones form follows barebones function.

Here's a link to the USB hub I used. It looks exotic and expensive, but it's actually a good value:
https://www.amazon.com/gp/product/B086S ... UTF8&psc=1
image.png
image.png (306.75 KiB) Viewed 18781 times

I can unplug the single USB cable and power cable, and carry the entire unit quite easily, and store it in a safe spot. When I need to do a backup, I plug the USB cable into my Unraid server, power it up, and sequentially press the buttons on the USB hub to do a manual staggered spin up from bottom to top (not sure if that matters, but it's what I do to ensure the drives always connect in the same order, in case that affects anything, and to minimize power draw on spin-up).

The drives appear in Unassigned Devices. The first time I had to create the pool manually, but now that always shows back up each time they are plugged in, so all I have to do is mount them.

image.png
image.png (218.46 KiB) Viewed 18781 times

You can ignore the first drive in the picture above, that's my 4TB video surveillance drive. Dev 2-6 are the 5 drives for my backup solution. I named the main drive Frankenstore, and then each additional drive with the drive number, i.e. Frankenstore_D2. The pool data lives on the main Frankenstore drive, so when I hit Mount it mounts all 5 together into the 80 TB large pool. When I'm done with the backup, I click Unmount, turn off the USB ports, and unplug for storage.

I did notice that the drives got too warm for my liking, so I currently have a fan aimed at the drives while doing a backup, which dropped temps tremendously. At some point, I plan to 3D print some brackets and mount a couple 120mm fans permanently as part of the solution.

The last part of the setup is the backup script itself. To keep the backups as quick as possible, I only want to do a differential backup (only back up what has changed since the last backup). I already had the basis of a backup script that I used to backup my cache drive's contents to the array, so I copied that and used it to write my array backup script, below.

Code: Select all

#!/bin/bash

LogFile=/var/log/array_backup.log
BackupDir=/mnt/disks/Frankenstore
Notify=/usr/local/emhttp/webGui/scripts/notify

echo `date` "Starting Array backup to " $BackupDir >> $LogFile

#Backup 4K via rsync
$Notify -i normal -s "Beginning 4K Backup" -d " 4K Backup started at `date`"
rsync -avrtH --delete /mnt/user/4K $BackupDir  >> $LogFile
sleep 2
$Notify -i normal -s "Finished 4K Backup" -d " 4K Backup completed at `date`"

sleep 2
$Notify -i normal -s "Beginning Blu-Rays Backup" -d " Blu-Rays Backup started at `date`"
rsync -avrtH --delete /mnt/user/Blu-Rays $BackupDir  >> $LogFile
sleep 2
$Notify -i normal -s "Finished Blu-Rays Backup" -d " Blu-Rays Backup completed at `date`"

sleep 2
$Notify -i normal -s "Beginning DVDs Backup" -d " DVDs Backup started at `date`"
rsync -avrtH --delete /mnt/user/DVDs $BackupDir  >> $LogFile
sleep 2
$Notify -i normal -s "Finished DVDs Backup" -d " DVDs Backup completed at `date`"

sleep 2
$Notify -i normal -s "Beginning TV_Series Backup" -d " TV_Series Backup started at `date`"
rsync -avrtH --delete /mnt/user/TV_Series $BackupDir  >> $LogFile
sleep 2
$Notify -i normal -s "Finished TV_Series Backup" -d " TV_Series Backup completed at `date`"

## RESTORE
## /usr/bin/rsync -avrtH --delete  $BackupDir  /mnt/cache/

echo `date` "backup Completed " $BackupDir >> $LogFile

# send notification
sleep 2
$Notify -i normal -s "Array Backup Completed" -d " Array Backup completed at `date`"
I have the script named array_backup.sh in my /boot/config folder, and to run it I use the command "bash array_backup.sh". That's it, easy peasy.

Most of the script is variables, sleep statements, and notify commands (so I see progress notifications in the Unraid GUI). The sleeps statements are necessary to ensure the notifications works, since Unraid seems to miss displaying notifications if you send them too quickly. That means the actual backup statement is just those "rsync" line, and I do that once for each of my shares that I backup.

The rsync options I pass, -avrtH, are for archive (backup mode), verbose (more logging output), recursive (subdirectories), times (preserves the original modification times instead of changing them to the backup time), hard-links (which preserves any hard-links that exist). The --delete option gets rid of any previously backup up files that are no longer in the source, so I'm replicating the file deletions in my array to the backup, which may not be to everyone's preferences but matched my goals. Everything gets logged to the $LogFile.

The RESTORE command line is commented out, and only serves as a reference for how I could do a restore. That's actually a command for my cache backup restore, so I would obviously have to rewrite it to use the correct source and target if I was restoring one of my array shares. If I actually had to do a restore, I would probably just run the command directly, instead of running it inside a script like this.

The btrfs drive pool I created is not striped or RAID'ed in any way, making it very similar to Unraid in that each file only exists on a single drive, and if I loose a backup drive I only loose the data that was backed up to that drive, plus potentially 1 or 2 files that happened to be spanning the gap from one drive to the next. If I do lose and replace a drive, with the way the rsync works then the next backup will simply re-copy those files that no longer exist in the backup, making it very easy to maintain this backup solution.

For me, the hardest part of adding a new drive to expand this pool will be 3D printing a couple more stackers, cutting the zip ties, inserting the new drive, and pushing new zip ties through the 3D-printed channels to hold everything together. Then I just plug in a USB cable, splice in the power cable to my power distribution bus, and finally add the drive to the pool. I'm extremely comfortable doing electrical wiring, if you're not then you can always just use the individual power plugs that come with these drives.

I will probably be expanding mine within the next 6 months, as I'm quickly running out of backup space. When I do that I'll take some more pictures showing the process.

Even with my "luxury" upgrades to do the single power supply, switchable USB hub, and 3D printed stacking, my total cost for this backup solution is just under $1600. That's only $20/TB, which is dirt cheap for a backup storage solution, and quite a bit cheaper than my Unraid setup. On average that's about $1 per 4K UHD Blu-ray, $0.50 for regular Blu-ray, and a nickle per DVD. But the best part is knowing I have an actual backup, which is priceless...
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Fri Apr 08, 2022 3:38 am

Great post, and thanks for sharing the rsync script.

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Fri Apr 08, 2022 3:49 pm

Question: to create the pool, do you use the same "create pool" as the one used to create a cache pool, or is it a different one? Not sure how you can use it, unless you create a backup share that uses this pool as a cache with "prefer" or "only".

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: My Unraid NAS Backup Solution

Post by Jamie » Fri Apr 08, 2022 4:44 pm

Hi Paul,

Thanks for sharing this. I was getting ready to ask about your backup pool when you mentioned it the other day. Can you share the unraid link where they talk about manually creating the pool if that is not too hard to remember or find?

By the way I use to do tape backups in the 90s. I was really happy when norton ghost came out. I use acronis backup for my windows stuff now

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sat Apr 09, 2022 11:11 am

Manni wrote: Fri Apr 08, 2022 3:49 pm do you use the same "create pool" as the one used to create a cache pool, or is it a different one?
Tough answer. I used Unassigned Devices, and this was back on Unraid 6.8.x where that was the only option. Unraid 6.9 introduced cache pools, and now you're supposed to use that instead, and you can even import pools created in UD into the standard Unraid pools.

But I've only played with the UD solution, so my experience stops there.

Jamie wrote: Fri Apr 08, 2022 4:44 pm Can you share the unraid link where they talk about manually creating the pool if that is not too hard to remember or find?
Found it. Here's the link on the Unraid Forums where I found the solution, and it has some good tips in it:

https://forums.unraid.net/topic/71315-i ... ent-882824

And that post had a link in it with the actual directions for creating the Unassigned Devices Pool. ARGH - but I just checked and that link is to a FAQ, and the FAQ has since been updated, and the directions I followed are now gone. :cry:

It seems that because Unraid 6.9 can do this now, the information for how to do it with Unassigned Devices has now been wiped off the internet.

I'm trying to recall how I did it a couple years ago. I know I have Unassigned Devices Plus installed, which allows "Destructive Mode", which I believe I needed to do this.

To create the pool, I think I needed to first delete any partitions on those drives. For example, I can't seem to create a new cache pool right now using these two precleared drives (Dev 2 & 3), because they both already have a partition:

image.png
image.png (102.23 KiB) Viewed 18756 times

Clicking the X next to the partition name (i.e. ZA1HH6CD) would remove the partition, and I think that opens up the opportunity to create a new partition. Do this for all drives you will be adding to the pool

When I created the partition, I used btrfs and somehow had it span multiple drives. I remember making sure not to do any type of striping/mirroring/RAID, as I just wanted JBOD at full capacity.

Here's where I get foggy - there is mention of using some command line statements, and I think I did this, but I don't know what they were anymore.

But alas, it seems I have wasted my time researching this, just found this gem in the official Unassigned Devices support thread, detailing how to create pools:
Starting with Unraid 6.9, there is a 'Pool' feature that should be used for pooling disks.
If one of you go down this path, please share your experience here.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sat Apr 09, 2022 12:53 pm

I tried that path but it doesn't work.

Even with the default file system set to xfs, whenever you create a pool with "Add Pool" in UD, it creates a RAID1 pool with btrfs. So for example if I create a pool with 7 WD Red 6TB, it creates a pool of 2x3 disks in raid 1, shows 18TB free and leaves one disk unused. One the main page, the pool is shown with a total size of 21TB and 3TB used.
Pool.JPG
Pool.JPG (186.76 KiB) Viewed 18753 times
Also even if this was to work, the pool is started at the same time at the array, so I'm not sure how happy it would be if the drives were disconnected when you start the array.

I couldn't find a way to select a different fs/disk layout either during or after the creation of the pool.

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sat Apr 09, 2022 1:04 pm

Actually I found the solution :)

If you click on the pool name in the main screen, it takes you to the settings.

In "Balance status", you can change the RAID level. I changed it to "Convert to single mode":
Balance status.JPG
Balance status.JPG (89.5 KiB) Viewed 18751 times
and pronto, 42TB Free:
Full size in pool.JPG
Full size in pool.JPG (190.31 KiB) Viewed 18751 times
:)

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sat Apr 09, 2022 2:07 pm

That's awesome!!! That's pretty similar to what I did with UD. I think Lime-Tech modelled this feature on how it worked in UD.

Before you go too far, do some sanity tests. Are you able to remove the pool from service, and then plug it back in later? In UD, you can Mount/Unmount, but I don't see such an option for the built-in Unraid Pool Devices. After all, if you want to keep your backup offline for safe keeping, you need to have a way to bring it back online to use it for restoring, or to run your incrementals.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sat Apr 09, 2022 2:59 pm

Yes as you know that was one of my worries, given that the pool is part of the configuration, so second step was to check what was possible or not.

I first tried to copy 7TB of data because I wanted to do my tests with data on more than one disk, unfortunately the server frozen before 600GB, so I guess I'll find out more later. I have no idea if it's a one-off or something more nefarious, I'll report back after another attempt.

First observation, I used screen/MC to do the initial copy, and it looks like MC starts randomly (it started at letter "R") and also writes randomly (it wasn't writing in the first drive of the pool, but in the last one, no idea why.

After the server crashed, I cancelled the parity check (I wasn't writing data to the array so I know it's good, I'll check it overnight) and I did test for a few things.

As expected, if you shutdown, take the pool drives out and boot, unraid isn't happy and reports them missing with big red warnings:
1. Boot without the pool.JPG
1. Boot without the pool.JPG (351.29 KiB) Viewed 18744 times
This is to be expected.

However, you can start the array as normal, as the array doesn't use the backup pool as a cache, and all is fine.

If you then shutdown, put the backup disks back (I put them all in at the same time, just to make sure that the disk order doesn't matter), and start the server again, all the disks are detected in the right order, and there are no errors. The disks look line this, with a blue square, so I was a bit worried as this is how they look when they are new and need formatting:
2. Boot with the pool or after recreating it.JPG
2. Boot with the pool or after recreating it.JPG (230.48 KiB) Viewed 18744 times
However, if you start the array, all is good / all the drives are green and the data is back, as expected, exactly as in the screenshot below.

So as long as you don't mind the big red errors when you reboot the server, that's not an issue. I use stanby so I would only see these errors once in a while.

However, I wanted to go one server further, so I shutdown again, took all the disks out, rebooted, and and then I selected "none" for the pool disks, hence deleting the pool. I know, it's scary, but that way there is no pool so no errors. I wanted to see if I could get the pool data back if I recreated exactly the same pool.

So I shutdown again (not necessary, but I wanted to make sure nothing was in memory), put all the drives back in a random order, recreated a pool with the same name, and with the drives in the exact same order (in the pool). And bingo, the pool is recreated without formatting anything (it looks exactly as in the screenshot above), with all the backed up data back:
3. After recreation of the pool.JPG
3. After recreation of the pool.JPG (227.3 KiB) Viewed 18744 times
So after your backup, you can safely delete the pool if you don't plan to use the backup for a while and are annoyed by the big red errors everytime you reboot. If you recreate the pool with the same name, same number of disks and all the disks in the same order, you'll get all the data back where it was.

Now I'm going to try a larger copy again, I'll report back if it crashes again.

EDIT: one thing to be aware off, if you copy a share to the backup pool in the root, you're going to get an error like this from the fix common errors plugin (if you use it, and I highly recommend it):
Files detected on backup_pool.JPG
Files detected on backup_pool.JPG (141.28 KiB) Viewed 18737 times
This is because the backup pool is supposed to be a cache, and the share I try to backup is supposed to use another cache, so I guess the plugin is confused. Maybe that's what caused the crash.

So for my second attempt, I've created a folder on the pool backup called backup_the_share_I_want_to_backup, and I copy the content of the share I want to backup into that folder to avoid this issue. I guess you could also avoid it by creating a single folder called backup and copying the shares you want to copy into it. I like the idea of a different folder, because I can select the files I want to copy and MC does it in alphabetical order (that's OCD me).

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sat Apr 09, 2022 4:08 pm

Wow, so much not to like. I'm glad I'm using Unassigned Devices for this, and you may want to explore this further. Sorry, I don't have any spare drives to connect to the server to play around with this, so I'm not much help in what steps to take, but I think it's very similar to what you're doing but using Unassigned Devices along with Unassigned Devices Plus (and enable the Destructive Mode).

I worry what happens if you mistakenly add them back in the wrong order... I'd imagine at the very least that would corrupt any files that were split across two drives.

With UD, I remove it and it doesn't complain. I add it back in and it doesn't make me recreate it from scratch. I don't have to remember the drive order. And I've never gotten any alerts from FCE that my data is in two places, because I don't think it checks outside the array/pools.

At this point, I'm extremely disappointed that they removed all the documentation for doing this in UD, because while you can easily create a permanent pool in Unraid, it seems you can't easily create an unpluggable pool, at least not without the missing drives error mess. There is still a functionality gap that only UD solves.

One other thing I like is that with UD, I can hot-plug in my backup pool drives without even stopping the array. Which brings us to this:
Manni wrote: Sat Apr 09, 2022 2:59 pm If you then shutdown, put the backup disks back (I put them all in at the same time, just to make sure that the disk order doesn't matter), and start the server again,
When I was new to Unraid, I always thought that I had to power down to swap drives. Maybe you already know better, but it literally took me a couple years before I learned that I can hot-plug drives at any time. Note, you would have to stop the array before unplugging a drive that is in the running array, else you get horrible errors. I've learned it's so easy to simply stop the array, pull out a drive, pop in a replacement, assign the new drive, start the array, and voila, in under a minute I'm done. So long as your controllers support hot-plug (they do), this is so much nicer than powering down.

By the way, I did test hot-swapping on my motherboards SATA ports (since I enabled Hot-Plug in the BIOS), and it works perfectly.

If you already knew all that, cool, but from reading your write-up it seemed possible that just like me, you didn't realize how sophisticated the hardware actually is.

Which brings me back to that point about using UD for this backup pool - I love that I don't even have to stop the array, I just plug in my drives, hit mount, run my backup, hit unmount, then unplug them. Same as you would on a Windows PC if you were plugging in a USB backup drive.
Manni wrote: Sat Apr 09, 2022 2:59 pm First observation, I used screen/MC to do the initial copy, and it looks like MC starts randomly (it started at letter "R") and also writes randomly (it wasn't writing in the first drive of the pool, but in the last one, no idea w
I know you're using MC for convenience, but I wanted to point out that rsync already works alpha-sorted. I can see in my log files that it processes all the shares alphabetically, and by monitoring the log files I can see approximately where it is in the backup process.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Post Reply