Decorum for the Forum:
  • Be nice. If you want to be mean, try Reddit.
  • No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
  • No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
  • No Spamming. If you want to make a buck, work smarter... somewhere else.
  • No Adult Content. Half the internet is dedicated to adult content. This half isn't.

Privacy Policy: Click Here to Review (updated September 30, 2020)

My Unraid NAS Backup Solution

Show off your HTPC builds, NAS Servers, and any other hardware. Great place to ask for hardware help too.
Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: My Unraid NAS Backup Solution

Post by Jamie » Sat Apr 09, 2022 8:38 pm

Manni wrote: Sat Apr 09, 2022 7:11 pm Following the above posts, I checked the content of the UD mounted deleted pool, and if I go to "disks" in MC, the backup share is present in any of the disks part of the pool, but the pool doesn't appear as a single pool.

Also if you unmount the first disk, all the other disk remain with "pool" greyed out, and there is no way to unmount them.

So using UD to use a pool created by Unraid doesn't seem to be a good idea...
Thanks Manni, I was going to try your method of creating an unraid pool and putting in unassigned devices but I guess that it is not a good idea.

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sat Apr 09, 2022 8:49 pm

Hi Jamie,

Great, please post your feedback when you do so.

Personally, I would not try the procedure suggested by Paul without having a backup of the array. Too many things can go wrong with the command line...

EDIT: just saw your second post: no, not a good idea if you want to mount the pool using unassigned devices. It works fine if you follow the procedure I suggeted, as long as you don't mind getting an error for the missing disks if you take them out of the server for offline and off location storage.

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 10:53 am

Johnnie came through. The information wasn't deleted, merely hidden. He has made it visible again, yay!

https://forums.unraid.net/topic/46802-f ... ent-462135

I was pretty close in my command line. I was also right that I created the partitions first, and that's where they got their partition names.

For reference (in case the info is ever lost again) I'm going to paste here the steps relevant for creating a single, non-RAIDed pool of drives, combined with some mildly revised steps and extra detail from me.

First Time Setup Instructions to create a Removable Unassigned Devices BTRFS JBOD Drive Pool
  1. Install Unassigned Devices
  2. Install Unassigned Devices Plus
  3. Settings > Unassigned Devices > Destructive Mode: Enable
  4. Plugin/insert your drives for the pool, verify they are visible in UD
  5. With UD, Delete any partitions on those drives - click the X next to the partition name
    Note: If you don't see any partitions, click drive's Settings (gears icon) and make sure Show Partitions is Enabled
  6. With UD, Format the 1st drive using btrfs, choose a mount point name (i.e. BackupPool), enable Share
  7. With UD, Mount the 1st drive. It will be mounted at /mnt/disks/<mountpointname>, i.e. /mnt/disks/BackupPool
    -----------------------------------------------------------------------------------------------------------------------------------------------------------
  8. With UD, Format the 2nd drive with btrfs. Optionally change the mount point name for reference only (i.e. BackupPool_D2)
  9. Note the sdX (devid) in UD, then open a Console/SSH session, add the 2nd drive to the pool by typing:

    Code: Select all

    btrfs dev add -f /dev/sdX1 /mnt/disks/BackupPool
    Replace X with the correct identifier. Note the 1 at the end to specify the partition - for NVMe devs use p1, e.g. /dev/nvme0n1p1
    -----------------------------------------------------------------------------------------------------------------------------------------------------------
  10. Repeat 8 & 9 for any additional drives to add to the pool

NOTE: If you are adding additional drives to an existing JBOD pool at a later date, mount the pool and just follow steps 8-10.

NOTE: If you want instructions for creating a RAIDed pool, please refer to the link above, and follow the instructions for running a balance start.

Notes on Using UD Pools
  • Only mount the first drive with UD, all other drives will automount with it. Same to unmount, only unmount 1st drive.
  • Used/Free space reporting is most accurate if you use the 1st drive for mounting
  • Used/Free space reporting may be incorrect in complex scenarios. To double-check, use command line:

    Code: Select all

    btrfs fi usage /mnt/disks/BackupPool
  • You can have as many UD pools as you want

Remove a drive:
-to remove a device from a pool type (assuming there's enough free space):

Code: Select all

btrfs dev del /dev/sdX1 /mnt/disks/BackupPool
Replace X with correct identifier, note the 1 in the end, and replace BackupPool with the pool mount name


Replace a device:
To replace a device from a pool (if you have enough ports to have both old and new devices connected simultaneously):
  • You need to partition the new device, to do that format it using the UD plugin, you can use any filesystem, then type:

Code: Select all

btrfs replace start -f /dev/sdX1 /dev/sdY1 /mnt/disks/yourpoolpath
Replace X with source, Y with target, note the 1 in the end of both, you can check replacement progress with:

Code: Select all

btrfs replace status /mnt/disks/yourpoolpath
If the new device is larger you need to resize it to use all available capacity, you can do that with:

Code: Select all

btrfs fi resize X:max /mnt/disks/yourpoolpath
Replace X with the correct devid, you can find that with:

Code: Select all

btrfs fi show /mnt/disks/yourpoolpath
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 10:58 am

Manni wrote: Sat Apr 09, 2022 7:11 pm Following the above posts, I checked the content of the UD mounted deleted pool, and if I go to "disks" in MC, the backup share is present in any of the disks part of the pool, but the pool doesn't appear as a single pool.

Also if you unmount the first disk, all the other disk remain with "pool" greyed out, and there is no way to unmount them.

So using UD to use a pool created by Unraid doesn't seem to be a good idea...
Your first screenshot is the correct result (the one you thought wrong), except for best results you should have done the mount using the 1st device (maybe you did?). When you do this, you see the correct pool size on that device, all of the other drives show no space.

Your second screenshot is where you messed up. When you mounted the other drives, you essentially loaded multiple copies of the pool. That's wrong, and probably why you got the weird result above.

EDIT: For reference, here is my Frankenstore backup pool screenshot (from the 1st post) - note the drives show no space and the MOUNT button is usable but should be ignored on the extra drives:

image.png
image.png (217.32 KiB) Viewed 15508 times
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 11:05 am

Manni wrote: Sat Apr 09, 2022 8:49 pm Personally, I would not try the procedure suggested by Paul without having a backup of the array. Too many things can go wrong with the command line...
Ironically, these command line steps are required to create the backup pool to make a backup of the array, so you've defined a no-win, catch-22 restriction. ;)

Yes, there is always a risk at the command line, but take your time, double and triple-check your commands and values, and you'll be fine. The most important thing to check is the device ID's - make sure you are using ID's listed on your drives in Unassigned Devices, and not any drives listed in your array or elsewhere.

Most of the steps outlined above are done in the GUI. For the most part, there's only a single command line needed, to add a device to an existing pool (Step 9). That should be manageable.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sun Apr 10, 2022 11:08 am

As reported, I did mount the first drive. But the other drives, as per the first screenshot, were not mounted automatically, so didn't show any size or any indication that they were part of a pool. Hence it didn't look correct to me. When you described the correct behaviour, you said that mounting the first drive would mount all the others automatically. Did I get that wrong? This is still what you describe in your updated step-by-step. Hence why I tried mounting the other drives manually. Mounting the first drive also doesn't give access to a pool, only to the first drive. How do you get access to the pool name, as happens when the pool is mounted correctly?

Anyway, this is moot, I'm backing up with my external NASes and I won't try your command lines, that's too involved and risky to my taste when the array isn't backed up yet, but if someone is happy to live dangerously and report back, I'm all ears. :)

There is no catch-22 on no-win on my side, I'm just backing up first to my external NASes, that's a win-win (Safe and easy). Then I'll experiment with back-up pools if I need to, but that has already taken too much of my time time so my experimentation has ended. I'll revisit when everything else is sorted (still having issues to wake up a server from S3 state, plus I have a PC that fails launching CMC when the CCC VM is no online, which shouldn't be the case).

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 11:35 am

I just updated my post above with the screenshot. I agree, it looks confusing in the GUI, as the extra drives just sit there looking unused. What matters is that the mounted partition shows the full amount of pool space. The Unraid Pools feature does a better job of displaying this info.

While this may be a moot point for you, I think it is important for others to understand how this works. You mount the first drive, and the others automount with it, but the GUI makes it look like nothing happened. This confused me too in the beginning years ago. You have to just trust that it's working, with the only indicator being the mounted partition's size being larger than the drive.

Because of the way btrfs works, the pool info is replicated on each drive, so technically you can mount any one of the drives to mount the pool. The recommendation to only mount the 1st drive is for more accurate size reporting. The stipulation to only mount a single drive is to prevent from mounting multiple copies of the same pool, since that pool definition exists on all drives. It would be nice if UD handled this a bit better, but since technically you can have multiple partitions/pools each drive, I can't fault the developer too much for the confusing way this works.

I'm happy you got your backup solution sorted, sounds ideal. Do you know what file system it is using? I wonder that if your NAS gives up the ghost, will you be able to access each drive individually to retrieve the data.

Manni wrote: Sun Apr 10, 2022 11:08 am plus I have a PC that fails launching CMC when the CCC VM is no online, which shouldn't be the case.
Feel free to enable CMC Debug Logging on that PC, then send me the log files for review.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sun Apr 10, 2022 11:47 am

If that's the case then it's much easier to create the pool using unraid, delete the pool after creation, and mount the first disk using UD. Feels much safer and easier that way. I agree it's important for others to understand how it works, but if it was working correctly after I mounted the first disk with UD, then I don't see the point of all the multi-steps in your method, using the command line, etc. You can easily add/remove disks using the GUI by re-creating the pool if you need to change anything. So the steps I would recommend are:

- Create the pool using unraid GUI
- Change the balance status in the pool settings to get rid of RAID 1 if you don't need it and reclaim the space
- Stop the array and delete the pool (set the number of disks to "none")
- When you want to use it, mount the first disk using UD.
- If you want to change anything, re-create the pool using the Unraid GUI, with the disks in the same order, and make any addition/deletion you need to the pool, then delete the pool again.

As mentioned, I used the Synology Hybrid RAID or SHR (https://kb.synology.com/en-uk/DSM/tutor ... d_RAID_SHR) to create my backup volume:

"In the end I decided to go for a single SHR volume over the 24 disks, that gives me protection for 1 disk loss and 72TB effective storage in total. It uses all the storage on all disks, only losing one disk of the largest capacity (so 4 TB in my case) for redundancy/parity. I wouldn’t do this for my main storage, but for a backup NAS I think that’s fine."

I definitely can't recover anything if the Synology dies or if I lose more than one disk, but I really don't care about this because it's a backup. I have all the data on my Unraid Servers, each protected by two parity disks. So I really wanted to maximum storage capacity on the Synology backup, with minimal protection. I would never use SHR for a primary data volume, I'd use RAID 6, but that costs a lot of storage (I'd lose 14TB instead of just 4TB).

For the QNAP with 8x2TB, I went for RAID5, which gives me a bit more than 12TB of usable storage, with 1 disk protection.

I'll email you the logs for the PC that doesn't work when the Unraid VM isn't on, thanks. [EDIT: done!]

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 12:24 pm

Manni wrote: Sun Apr 10, 2022 11:47 am If that's the case then it's much easier to create the pool using unraid, delete the pool after creation, and mount the first disk using UD.
I had a similar thought. Either way is good. Which way is easier depends upon the person doing the work.

For me, when I add a 6th drive, I'll just create the partition in UD and use the one command line to add it to the pool, which will be quicker than defining the pool in Unraid, modifying it, and then removing the pool to go back to using UD. I have a high comfort level at the command line, though, so I agree with you that using Unraid's built-in tools will provide an easier path via the GUI, even if it takes marginally more steps.

My biggest concern when adding the pool to Unraid is that I define it correctly, as I don't want to accidentally change the drive order and corrupt any split files. So in my mind I find it less risky to use the one command line to add a drive, versus redefining the whole pool from scratch. At least we have options, and can do what suits our personal preferences.

Manni wrote: Sun Apr 10, 2022 11:47 am I definitely can't recover anything if the Synology dies or if I lose more than one disk, but I really don't care about this because it's a backup
Man that sends shivers down my spine. Yeah, it's only a disposable backup until you actually need it. We are obviously two unique individuals with differing perspectives on risks. I personally have lost too much data over the years due to RAIDed solutions failing in unexpected ways, and I've since bought whole-heartedly into the Unraid approach of worst case only losing the failed drive's data and not the whole array. While I have zero redundancy in my backup, I also have a virtually zero risk of loosing everything. While I trust enterprise RAID solutions, I've lost the ability to trust consumer RAID solutions. But that's just me.

Regardless, we both deserve a pat on the back, as we have (or will soon have) actual backups of a very large storage server. I know very few people that even have a storage server at home like this, and even fewer that have a backup of it. Kudos.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Sun Apr 10, 2022 12:39 pm

I know about the shiver, that's why I moved to Unraid in the first place, but I don't have any other option for my backup if I want to be able to use the HDDs I already have, which are fairly small, hence need a large number of bays/connection. This backup will be convenient to use, gives me an additional layer of protection should disaster strike, but yeah, it's at the mercy of something going wrong with the SHR/RAID volume. Still, better than no backup at all :)

The only other alternative for me would be to build a third Unraid server with 24 bays, and that's a no-no. I have already built two Unraid servers to move ALL my data out of RAID servers, I think it's fine to use these old servers as backup. The available bays on my server A only allowed me to have 7 6TB and that gave me 40TB, which wasn't enough anyway and killed most of my expansion room, compared to the 72TB I get with the Synology.

As some point, if I run out of space on the Unraid servers and don't want to purchase new discs, I might even reformat the Synology as a couple of RAID6 volumes and use them as primary servers for my less essential data (such as bluray) to make more room for 4K discs.

I'm very familiar with the command line (remember, I was born before DOS and I used to program in assembler), so it's not like it intimidates me. I use it all the time, but as soon as we're talking about adding/deleting partitions, I get really cautious. I've done it (using diskpart to correct issues or reclaim space that I couldn't reclaim otherwise), but it's always nerver-wracking because you get one letter wrong and you can delete all your data. I would do it if I had no other choices, but given that there is an alternative, I'd rather use that.

Anyway, as you say, we have options, it's great to know that we can create and maintain backup pools (in two different ways) on our Unraid servers, and definitely a pat in the back for both of us. Hopefully, we'll soon be joined by Jamie. I'm curious to see which path he'll choose, and whether it works or not.

Post Reply