Decorum for the Forum:
Privacy Policy: Click Here to Review (updated September 30, 2020)
- Be nice. If you want to be mean, try Reddit.
- No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
- No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
- No Spamming. If you want to make a buck, work smarter... somewhere else.
- No Adult Content. Half the internet is dedicated to adult content. This half isn't.
Privacy Policy: Click Here to Review (updated September 30, 2020)
My Unraid NAS Backup Solution
Re: My Unraid NAS Backup Solution
I definitely agree that for such a high number of smaller drives, your approach is really the only viable solution. My USB drive stack approach really only works if you're buying the biggest possible drives, to keep the total stack small. Even then, your total storage is so large that you'd need 8 of those 18tb drives, so quite a large drive stack. I only had old 3TB drives on hand, and not nearly enough to come close to backing up my array, which is why I bought new drives for the backup pool. I'm excited for 20TB external USB drives to hit the market at a decent price, but if I had to splurge now, I'd go with an 18TB unit.
Regarding your comment on creating partitions at the command line, for the steps above the creating of partitions is done with UD in the GUI. The command line only adds an existing partition to an existing pool. I definitely agree with trepidation on using diskpart and similar tools, I always hated that and I don't miss those days at all.
The guy who maintains UD seems pretty active on support - I see updates on that plugin near constantly. Perhaps I can encourage him to add some pool management into the GUI. Would be nice to not have to go to the command line to add a disk to the pool, and nicer yet if the GUI reflected that those drives were already in-use when you mount the pool. I'm sure he's focused on other aspects of UD now that Unraid supports multiple pools, but maybe he'd be willing considering UD is better at removable pools.
Regarding your comment on creating partitions at the command line, for the steps above the creating of partitions is done with UD in the GUI. The command line only adds an existing partition to an existing pool. I definitely agree with trepidation on using diskpart and similar tools, I always hated that and I don't miss those days at all.
The guy who maintains UD seems pretty active on support - I see updates on that plugin near constantly. Perhaps I can encourage him to add some pool management into the GUI. Would be nice to not have to go to the command line to add a disk to the pool, and nicer yet if the GUI reflected that those drives were already in-use when you mount the pool. I'm sure he's focused on other aspects of UD now that Unraid supports multiple pools, but maybe he'd be willing considering UD is better at removable pools.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
I just messaged dlandon to see if he will consider those enhancements to UD. I'm doubtful, but maybe we'll get lucky.
EDIT: For the past six months, on a daily basis I've been working with an off-shore development team to create a custom .NET application that I designed for an enterprise client (this is one of the main reasons CMC progress is on-hold). I've gotten a little spoilt on being able to tell a developer when, what and how to do what whatever it is that I want. I'm finding it a little frustrating and humbling to be reduced to begging for features that I will likely never get. I wish I could just bark an order and get the desired UD enhancements...
The good news is that we got client sign-off on the delivered solution this past week, so my time is starting to free up again. I've got some catchup to do on some personal matters, but hopefully I can get started on CMC v5 again soon.
EDIT: For the past six months, on a daily basis I've been working with an off-shore development team to create a custom .NET application that I designed for an enterprise client (this is one of the main reasons CMC progress is on-hold). I've gotten a little spoilt on being able to tell a developer when, what and how to do what whatever it is that I want. I'm finding it a little frustrating and humbling to be reduced to begging for features that I will likely never get. I wish I could just bark an order and get the desired UD enhancements...
The good news is that we got client sign-off on the delivered solution this past week, so my time is starting to free up again. I've got some catchup to do on some personal matters, but hopefully I can get started on CMC v5 again soon.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
I have been working with command line apps since the 80s. I usually put everything in a script and check things over 3 - 4 times. I then do the commands one at a time. GUIs have potential problems too. I brought down a company web server by clicking the wrong button. People weren't too happy with me for weeks. I lived through it though.
I am wondering. Can you bring down the whole array and still work in unassigned devices? That would solve the problem with accidently touching a drive in the array.
Thanks for the CMC update, Paul. I thought that you were working on another project. After all CMC doesn't really pay the bills.
I am wondering. Can you bring down the whole array and still work in unassigned devices? That would solve the problem with accidently touching a drive in the array.
Thanks for the CMC update, Paul. I thought that you were working on another project. After all CMC doesn't really pay the bills.
Re: My Unraid NAS Backup Solution
To the contrary, that might make it worse. The drives would still exist on the system, but because the array isn't running then you don't have them actively associated with another process, making them free (or at least free-er) to be accidentally used by the command line statements.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
dlandon responded regarding my UD queries.
For the issue where the mount buttons remain enabled, here's his response:
For my question on adding pool management features to UD, he responded:
For the issue where the mount buttons remain enabled, here's his response:
I guess my strategy of naming them all different with a _D# suffix backfired. I'll give this a try and see how it works.dlandon wrote:All devices in the pool must have the same mount point. Click on the mount point when the devices are unmounted and make them all the same.
For my question on adding pool management features to UD, he responded:
Disappointing, but understandable. So your options remain to either use Unraid's pool management, or the command line, whatever you prefer.dlandon wrote: UD will mount an existing pool, but pool management is beyond UD's scope.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
I had not changed the mounting point, yet I still had the mount button enabled after mounting the first disk, so the issue is not related to you changing the mounting point. UD doesn’t support properly pools created by Unraid, either for mounting or unmounting, at least from a GUI point of view, because Unraid also changes the name of each disk (see my screenshots). He should treat as a pool disks that have the same root, and ignore the last 2 or 3 characters (anything after the underscore).
Re: My Unraid NAS Backup Solution
WARNING - DON'T DO THIS
Yeah. He mentioned that you have to "trick" UD into seeing them as a single pool, by giving them all the same mount point, but then went on to say that it was required for the disk id to match the mount point for a pool. The GUI doesn't allow you to make such a change with all drives inserted, so you have to remove all the drives, insert them one at a time, and rename to get around the naming conflict restrictions. And you have to clear the device history too, so then when you insert all drives with the same name & mountpoint, they will appear as one. It's a bit confusing at the moment, but I've been conversing with him for further guidance to make sure I do it right. I'm a little worried I could lose my backup, and it will take me over a week to recreate it, so I'm treading cautiously and waiting for confirmations before testing this.
Of course, I've been using this backup solution for years as-is, so this is a cosmetic enhancement only in my mind, as long as you follow the instructions the current solution works 100%. I'm a little concerned that if I do this change then I will lose the ability to see each drive individually, which lets me monitor drive temps.
Yeah. He mentioned that you have to "trick" UD into seeing them as a single pool, by giving them all the same mount point, but then went on to say that it was required for the disk id to match the mount point for a pool. The GUI doesn't allow you to make such a change with all drives inserted, so you have to remove all the drives, insert them one at a time, and rename to get around the naming conflict restrictions. And you have to clear the device history too, so then when you insert all drives with the same name & mountpoint, they will appear as one. It's a bit confusing at the moment, but I've been conversing with him for further guidance to make sure I do it right. I'm a little worried I could lose my backup, and it will take me over a week to recreate it, so I'm treading cautiously and waiting for confirmations before testing this.
Of course, I've been using this backup solution for years as-is, so this is a cosmetic enhancement only in my mind, as long as you follow the instructions the current solution works 100%. I'm a little concerned that if I do this change then I will lose the ability to see each drive individually, which lets me monitor drive temps.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
I think the issue is on his side. He should support at the very least a pool created by unraid, and display the other disks the way they are (with pool greyed out) when you mount them manually as well as the first one, without us having to do so. I don't want the pool drives to disappear, I just want them to be shown as members of the pool as soon as you mount the first drive, and for them to appear as normal as soon as you unmount the first drive. Anything else is just scary/unnecessary workarounds.
Re: My Unraid NAS Backup Solution
A few days back, I had asked for clarification on the steps to trick UD into seeing the array as a pool. dlandon never responded, and I figured he wasn't going to. I decided I had a clear enough idea on what to do, and that I could undo it if it didn't work as expected. BIG MISTAKE.
So first I powered off all 5 backup drives. I powered on #2, and then was able to rename it from Frankenstore_D2 to Frankenstore. I then powered on #1, and noticed that #1's button was ØARRAY instead of MOUNT, and they both displayed as Frankenstore. The MOUNT button was on #2, which I wasn't crazy about, because I consider #1 the main drive for mounting, but I figured it had something to do with startup order and wasn't too worried.
Since I liked where this was going, I powered off both #1 and #2, and powered on #3 so I could rename it. But when it powered up, it was already showing Frankenstore as the name, not Frankenstore_D3. Somehow renaming #2 had also changed #3... or had it? I decided this was why dlandon told me to clear the drive history, so I powered off the drive, and cleared UD's historical drives of #2-#5, but didn't remove the history for #1.
I then powered on #3 again, and somehow it still showed Frankenstore as the mount name. In addition to the name having already changed, the option to edit the name was now gone.
Curious what the entire backup pool looked like, I powered off #3, and then powered on in sequence #1 through #5. Sure enough, they all showed as mount point Frankenstore, and they all had POOL beside them except for #2, which had MOUNT. So I clicked MOUNT, and it looked like the entire array came up, and the other drives had POOL for the button name, and the POOL button is always grayed out & disabled, essentially exactly like Manni was describing.
At this point, it seemed my goal of having all drives belong to the pool was already complete, after having only renamed a single drive's mount point. But it was still bothering me that drive #2 was the parent MOUNT drive. Since dlandon did say to clear the drive history of all drives, I figured that was necessary to fix this condition. I powered off all 5 drives, removed them from UD's Historical Devices, and then powered them back on, in sequence, from #1 to #5. I figured them time it would be perfect.
Uh-oh:
Now all drives have the ØARRAY value in the button, which I presume means they are a member of an incomplete array. I can't edit any of them, and I can no longer mount them. I can't revert them. Even the gears Settings button is blacked out/disabled and can't be clicked.
This is exactly what I was worried about, and why I had reached out to UD's developer dlandon for confirmation on the steps.
I still have some ideas on how to recover. I might be able to remove and re-add the drives to the btrfs pool using the commands documented earlier. I might be lucky and have the data recovered since it's just a simple single pool, but worst case here I may have to spend a week running a new backup.
I should have stayed content with the solution I had in place. It worked perfectly, and I knew not to click the Mount button on any drives other than #1.
So first I powered off all 5 backup drives. I powered on #2, and then was able to rename it from Frankenstore_D2 to Frankenstore. I then powered on #1, and noticed that #1's button was ØARRAY instead of MOUNT, and they both displayed as Frankenstore. The MOUNT button was on #2, which I wasn't crazy about, because I consider #1 the main drive for mounting, but I figured it had something to do with startup order and wasn't too worried.
Since I liked where this was going, I powered off both #1 and #2, and powered on #3 so I could rename it. But when it powered up, it was already showing Frankenstore as the name, not Frankenstore_D3. Somehow renaming #2 had also changed #3... or had it? I decided this was why dlandon told me to clear the drive history, so I powered off the drive, and cleared UD's historical drives of #2-#5, but didn't remove the history for #1.
I then powered on #3 again, and somehow it still showed Frankenstore as the mount name. In addition to the name having already changed, the option to edit the name was now gone.
Curious what the entire backup pool looked like, I powered off #3, and then powered on in sequence #1 through #5. Sure enough, they all showed as mount point Frankenstore, and they all had POOL beside them except for #2, which had MOUNT. So I clicked MOUNT, and it looked like the entire array came up, and the other drives had POOL for the button name, and the POOL button is always grayed out & disabled, essentially exactly like Manni was describing.
At this point, it seemed my goal of having all drives belong to the pool was already complete, after having only renamed a single drive's mount point. But it was still bothering me that drive #2 was the parent MOUNT drive. Since dlandon did say to clear the drive history of all drives, I figured that was necessary to fix this condition. I powered off all 5 drives, removed them from UD's Historical Devices, and then powered them back on, in sequence, from #1 to #5. I figured them time it would be perfect.
Uh-oh:
Now all drives have the ØARRAY value in the button, which I presume means they are a member of an incomplete array. I can't edit any of them, and I can no longer mount them. I can't revert them. Even the gears Settings button is blacked out/disabled and can't be clicked.
This is exactly what I was worried about, and why I had reached out to UD's developer dlandon for confirmation on the steps.
I still have some ideas on how to recover. I might be able to remove and re-add the drives to the btrfs pool using the commands documented earlier. I might be lucky and have the data recovered since it's just a simple single pool, but worst case here I may have to spend a week running a new backup.
I should have stayed content with the solution I had in place. It worked perfectly, and I knew not to click the Mount button on any drives other than #1.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter
Re: My Unraid NAS Backup Solution
While troubleshooting the issue, I discovered several error messages in the Unraid log complaining about duplicate devices. I was pretty sure that this was caused by renaming disk label Frankenstore_D2 to Frankenstore. I also noticed that none of the 5 drives had a "sdx" device label. I figured that at this point, Unraid/Linux was confused, and I needed a fresh boot.
I powered off all 5 drives, and then removed them from UD's Historical Devices, and rebooted the server.
After startup, I powered on backup drive #2, and was able to rename it back to Frankenstore_D2. I then powered it off, cleared the Historical Devices again, and powered them all back on in sequence.
This time they all got device ID's, have the button enabled to mount again, and are configurable. Interestingly, they all show mount point or drive label Frankenstore.
I then proceeded to mount drive #1, and amazingly I think I've achieved the originally desired result:
The data still displays slightly weird in UD, but now they all have the same Frankenstore mount point, the secondary drives show POOL instead of MOUNT. And oddly they all show the exact same pool size and free/used amounts.
It's a bit of a guess at this point as to exactly how I achieved this result, but I think the actual secret was simply removing these 5 drives from the Unassigned Devices Historical Drives. The individual drive labels still have the drive suffix on them, i.e. Frankenstore_D2, Frankenstore_D3, but now they finally share the same mount point. So it's the exact same steps I outlined to create this pool earlier in the thread, with one additional step to unplug them and then clear them from Historical Devices. A lot easier than I expected, and a lot scarier when I tried to follow the directions dlandon shared.
Even more odd, I keep trying to find a way to show that the individual drives still have the suffix, but I'm failing. Somehow they are actually showing as the same disk label ID, Frankenstore. I don't really understand, but I suppose this is the btrfs pool now overriding the label. But I'm not really incentivized to keep troubleshooting this, as it's now working exactly as I hoped, shomehow...
I did make another tweak. I noticed the option to disable the Mount button if I edit the individual drive settings, so I went ahead and disabled it for drives 2-5:
Now I don't even have to think about which drive is #1, there's only a single mount option. After mounting, drives 2-5 still change to POOL as shown above. My earlier concern that I wouldn't see individual drive activity or temps was unfounded, I still see everything so all is good.
I've checked and all my backup data seems intact. I'm running an rsync backup now to freshen it up with a few recent additions.
Edit: 35 minutes later and the differential backup is done. It's a relief to have the backup pool functional again. But it's disappointing that I had to reboot the server, as this resets my timer on validating the drive controller changes in my server, as my issues were only showing up after a couple weeks. Now I won't know until May if the issue got fixed. But that's a different topic, irrelevant to this backup thread.
I powered off all 5 drives, and then removed them from UD's Historical Devices, and rebooted the server.
After startup, I powered on backup drive #2, and was able to rename it back to Frankenstore_D2. I then powered it off, cleared the Historical Devices again, and powered them all back on in sequence.
This time they all got device ID's, have the button enabled to mount again, and are configurable. Interestingly, they all show mount point or drive label Frankenstore.
I then proceeded to mount drive #1, and amazingly I think I've achieved the originally desired result:
The data still displays slightly weird in UD, but now they all have the same Frankenstore mount point, the secondary drives show POOL instead of MOUNT. And oddly they all show the exact same pool size and free/used amounts.
It's a bit of a guess at this point as to exactly how I achieved this result, but I think the actual secret was simply removing these 5 drives from the Unassigned Devices Historical Drives. The individual drive labels still have the drive suffix on them, i.e. Frankenstore_D2, Frankenstore_D3, but now they finally share the same mount point. So it's the exact same steps I outlined to create this pool earlier in the thread, with one additional step to unplug them and then clear them from Historical Devices. A lot easier than I expected, and a lot scarier when I tried to follow the directions dlandon shared.
Even more odd, I keep trying to find a way to show that the individual drives still have the suffix, but I'm failing. Somehow they are actually showing as the same disk label ID, Frankenstore. I don't really understand, but I suppose this is the btrfs pool now overriding the label. But I'm not really incentivized to keep troubleshooting this, as it's now working exactly as I hoped, shomehow...
I did make another tweak. I noticed the option to disable the Mount button if I edit the individual drive settings, so I went ahead and disabled it for drives 2-5:
Now I don't even have to think about which drive is #1, there's only a single mount option. After mounting, drives 2-5 still change to POOL as shown above. My earlier concern that I wouldn't see individual drive activity or temps was unfounded, I still see everything so all is good.
I've checked and all my backup data seems intact. I'm running an rsync backup now to freshen it up with a few recent additions.
Edit: 35 minutes later and the differential backup is done. It's a relief to have the backup pool functional again. But it's disappointing that I had to reboot the server, as this resets my timer on validating the drive controller changes in my server, as my issues were only showing up after a couple weeks. Now I won't know until May if the issue got fixed. But that's a different topic, irrelevant to this backup thread.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter
Author, Chameleon MediaCenter