Decorum for the Forum:
  • Be nice. If you want to be mean, try Reddit.
  • No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
  • No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
  • No Spamming. If you want to make a buck, work smarter... somewhere else.
  • No Adult Content. Half the internet is dedicated to adult content. This half isn't.

Privacy Policy: Click Here to Review (updated September 30, 2020)

My Unraid NAS Backup Solution

Show off your HTPC builds, NAS Servers, and any other hardware. Great place to ask for hardware help too.
User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 1:09 pm

I definitely agree that for such a high number of smaller drives, your approach is really the only viable solution. My USB drive stack approach really only works if you're buying the biggest possible drives, to keep the total stack small. Even then, your total storage is so large that you'd need 8 of those 18tb drives, so quite a large drive stack. I only had old 3TB drives on hand, and not nearly enough to come close to backing up my array, which is why I bought new drives for the backup pool. I'm excited for 20TB external USB drives to hit the market at a decent price, but if I had to splurge now, I'd go with an 18TB unit.

Regarding your comment on creating partitions at the command line, for the steps above the creating of partitions is done with UD in the GUI. The command line only adds an existing partition to an existing pool. I definitely agree with trepidation on using diskpart and similar tools, I always hated that and I don't miss those days at all.

The guy who maintains UD seems pretty active on support - I see updates on that plugin near constantly. Perhaps I can encourage him to add some pool management into the GUI. Would be nice to not have to go to the command line to add a disk to the pool, and nicer yet if the GUI reflected that those drives were already in-use when you mount the pool. I'm sure he's focused on other aspects of UD now that Unraid supports multiple pools, but maybe he'd be willing considering UD is better at removable pools.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 1:35 pm

I just messaged dlandon to see if he will consider those enhancements to UD. I'm doubtful, but maybe we'll get lucky.

EDIT: For the past six months, on a daily basis I've been working with an off-shore development team to create a custom .NET application that I designed for an enterprise client (this is one of the main reasons CMC progress is on-hold). I've gotten a little spoilt on being able to tell a developer when, what and how to do what whatever it is that I want. I'm finding it a little frustrating and humbling to be reduced to begging for features that I will likely never get. I wish I could just bark an order and get the desired UD enhancements...

The good news is that we got client sign-off on the delivered solution this past week, so my time is starting to free up again. I've got some catchup to do on some personal matters, but hopefully I can get started on CMC v5 again soon.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: My Unraid NAS Backup Solution

Post by Jamie » Sun Apr 10, 2022 3:41 pm

I have been working with command line apps since the 80s. I usually put everything in a script and check things over 3 - 4 times. I then do the commands one at a time. GUIs have potential problems too. I brought down a company web server by clicking the wrong button. People weren't too happy with me for weeks. I lived through it though. :oops:

I am wondering. Can you bring down the whole array and still work in unassigned devices? That would solve the problem with accidently touching a drive in the array.

Thanks for the CMC update, Paul. I thought that you were working on another project. After all CMC doesn't really pay the bills.

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 4:32 pm

Jamie wrote: Sun Apr 10, 2022 3:41 pm Can you bring down the whole array and still work in unassigned devices? That would solve the problem with accidently touching a drive in the array.
To the contrary, that might make it worse. The drives would still exist on the system, but because the array isn't running then you don't have them actively associated with another process, making them free (or at least free-er) to be accidentally used by the command line statements.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Sun Apr 10, 2022 5:18 pm

dlandon responded regarding my UD queries.

For the issue where the mount buttons remain enabled, here's his response:
dlandon wrote:All devices in the pool must have the same mount point. Click on the mount point when the devices are unmounted and make them all the same.
I guess my strategy of naming them all different with a _D# suffix backfired. I'll give this a try and see how it works.

For my question on adding pool management features to UD, he responded:
dlandon wrote: UD will mount an existing pool, but pool management is beyond UD's scope.
Disappointing, but understandable. So your options remain to either use Unraid's pool management, or the command line, whatever you prefer.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Mon Apr 11, 2022 5:03 am

I had not changed the mounting point, yet I still had the mount button enabled after mounting the first disk, so the issue is not related to you changing the mounting point. UD doesn’t support properly pools created by Unraid, either for mounting or unmounting, at least from a GUI point of view, because Unraid also changes the name of each disk (see my screenshots). He should treat as a pool disks that have the same root, and ignore the last 2 or 3 characters (anything after the underscore).

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Mon Apr 11, 2022 9:26 am

WARNING - DON'T DO THIS

Yeah. He mentioned that you have to "trick" UD into seeing them as a single pool, by giving them all the same mount point, but then went on to say that it was required for the disk id to match the mount point for a pool. The GUI doesn't allow you to make such a change with all drives inserted, so you have to remove all the drives, insert them one at a time, and rename to get around the naming conflict restrictions. And you have to clear the device history too, so then when you insert all drives with the same name & mountpoint, they will appear as one. It's a bit confusing at the moment, but I've been conversing with him for further guidance to make sure I do it right. I'm a little worried I could lose my backup, and it will take me over a week to recreate it, so I'm treading cautiously and waiting for confirmations before testing this.

Of course, I've been using this backup solution for years as-is, so this is a cosmetic enhancement only in my mind, as long as you follow the instructions the current solution works 100%. I'm a little concerned that if I do this change then I will lose the ability to see each drive individually, which lets me monitor drive temps.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Manni
Posts: 593
Joined: Wed May 22, 2019 5:27 am

Re: My Unraid NAS Backup Solution

Post by Manni » Mon Apr 11, 2022 10:32 am

I think the issue is on his side. He should support at the very least a pool created by unraid, and display the other disks the way they are (with pool greyed out) when you mount them manually as well as the first one, without us having to do so. I don't want the pool drives to disappear, I just want them to be shown as members of the pool as soon as you mount the first drive, and for them to appear as normal as soon as you unmount the first drive. Anything else is just scary/unnecessary workarounds.

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Fri Apr 15, 2022 11:41 am

A few days back, I had asked for clarification on the steps to trick UD into seeing the array as a pool. dlandon never responded, and I figured he wasn't going to. I decided I had a clear enough idea on what to do, and that I could undo it if it didn't work as expected. BIG MISTAKE.

So first I powered off all 5 backup drives. I powered on #2, and then was able to rename it from Frankenstore_D2 to Frankenstore. I then powered on #1, and noticed that #1's button was ØARRAY instead of MOUNT, and they both displayed as Frankenstore. The MOUNT button was on #2, which I wasn't crazy about, because I consider #1 the main drive for mounting, but I figured it had something to do with startup order and wasn't too worried.

Since I liked where this was going, I powered off both #1 and #2, and powered on #3 so I could rename it. But when it powered up, it was already showing Frankenstore as the name, not Frankenstore_D3. Somehow renaming #2 had also changed #3... or had it? I decided this was why dlandon told me to clear the drive history, so I powered off the drive, and cleared UD's historical drives of #2-#5, but didn't remove the history for #1.

I then powered on #3 again, and somehow it still showed Frankenstore as the mount name. In addition to the name having already changed, the option to edit the name was now gone.

Curious what the entire backup pool looked like, I powered off #3, and then powered on in sequence #1 through #5. Sure enough, they all showed as mount point Frankenstore, and they all had POOL beside them except for #2, which had MOUNT. So I clicked MOUNT, and it looked like the entire array came up, and the other drives had POOL for the button name, and the POOL button is always grayed out & disabled, essentially exactly like Manni was describing.

At this point, it seemed my goal of having all drives belong to the pool was already complete, after having only renamed a single drive's mount point. But it was still bothering me that drive #2 was the parent MOUNT drive. Since dlandon did say to clear the drive history of all drives, I figured that was necessary to fix this condition. I powered off all 5 drives, removed them from UD's Historical Devices, and then powered them back on, in sequence, from #1 to #5. I figured them time it would be perfect.

Uh-oh:

image.png
image.png (99.14 KiB) Viewed 11887 times

Now all drives have the ØARRAY value in the button, which I presume means they are a member of an incomplete array. I can't edit any of them, and I can no longer mount them. I can't revert them. Even the gears Settings button is blacked out/disabled and can't be clicked.

This is exactly what I was worried about, and why I had reached out to UD's developer dlandon for confirmation on the steps.

I still have some ideas on how to recover. I might be able to remove and re-add the drives to the btrfs pool using the commands documented earlier. I might be lucky and have the data recovered since it's just a simple single pool, but worst case here I may have to spend a week running a new backup.

I should have stayed content with the solution I had in place. It worked perfectly, and I knew not to click the Mount button on any drives other than #1.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: My Unraid NAS Backup Solution

Post by Pauven » Fri Apr 15, 2022 1:17 pm

While troubleshooting the issue, I discovered several error messages in the Unraid log complaining about duplicate devices. I was pretty sure that this was caused by renaming disk label Frankenstore_D2 to Frankenstore. I also noticed that none of the 5 drives had a "sdx" device label. I figured that at this point, Unraid/Linux was confused, and I needed a fresh boot.

I powered off all 5 drives, and then removed them from UD's Historical Devices, and rebooted the server.

After startup, I powered on backup drive #2, and was able to rename it back to Frankenstore_D2. I then powered it off, cleared the Historical Devices again, and powered them all back on in sequence.

This time they all got device ID's, have the button enabled to mount again, and are configurable. Interestingly, they all show mount point or drive label Frankenstore.

image.png
image.png (100.19 KiB) Viewed 11886 times

I then proceeded to mount drive #1, and amazingly I think I've achieved the originally desired result:

image.png
image.png (139.74 KiB) Viewed 11886 times

The data still displays slightly weird in UD, but now they all have the same Frankenstore mount point, the secondary drives show POOL instead of MOUNT. And oddly they all show the exact same pool size and free/used amounts.

It's a bit of a guess at this point as to exactly how I achieved this result, but I think the actual secret was simply removing these 5 drives from the Unassigned Devices Historical Drives. The individual drive labels still have the drive suffix on them, i.e. Frankenstore_D2, Frankenstore_D3, but now they finally share the same mount point. So it's the exact same steps I outlined to create this pool earlier in the thread, with one additional step to unplug them and then clear them from Historical Devices. A lot easier than I expected, and a lot scarier when I tried to follow the directions dlandon shared.

Even more odd, I keep trying to find a way to show that the individual drives still have the suffix, but I'm failing. Somehow they are actually showing as the same disk label ID, Frankenstore. I don't really understand, but I suppose this is the btrfs pool now overriding the label. But I'm not really incentivized to keep troubleshooting this, as it's now working exactly as I hoped, shomehow...

I did make another tweak. I noticed the option to disable the Mount button if I edit the individual drive settings, so I went ahead and disabled it for drives 2-5:

image.png
image.png (79.28 KiB) Viewed 11886 times

Now I don't even have to think about which drive is #1, there's only a single mount option. After mounting, drives 2-5 still change to POOL as shown above. My earlier concern that I wouldn't see individual drive activity or temps was unfounded, I still see everything so all is good.

I've checked and all my backup data seems intact. I'm running an rsync backup now to freshen it up with a few recent additions.

Edit: 35 minutes later and the differential backup is done. It's a relief to have the backup pool functional again. But it's disappointing that I had to reboot the server, as this resets my timer on validating the drive controller changes in my server, as my issues were only showing up after a couple weeks. Now I won't know until May if the issue got fixed. But that's a different topic, irrelevant to this backup thread.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Post Reply