Re: My Unraid NAS Backup Solution
Posted: Sun Nov 19, 2023 1:55 pm
Whoohoo! It's working!
It was quite a bit more effort to modify that mirror.sh bash script, I had tons of typos and mistakes. But I finally figured out the correct syntax.
Additionally, I had to add one more IF clause to evaluate how deep the directory is before overriding where it is created. If the directory depth is 5, then it will be overridden from my Frankenstore MergerFS mount point to one of the FS# mount points. I only do this for depth of 5 as my Unraid share name is depth 4, i.e.
/DEPTH1/DEPTH2/DEPTH3/DEPTH4/DEPTH5
/mnt/disks/Frankenstore/Blu-rays/DEPTH5
So by overriding only depth 5 directories, I override where the movie folder is written, but not the subfolders. This was important for folder rips, which have a ton of subfolders. What I was finding is that the parent movie folder might get created on FS1, but as data was written then future subdirectories got split over to FS2 because FS1 was too low on free space.
But since I already had plenty of buffer free space, I didn't want those subdirectories to split off like that. So by not overriding the subdirectories, they get written to the Frankenstore mount point, which lets MergerFS decide where to write them, and it will keep them in the original parent's directory.
For example on Bullet Train, I override the main folder but not the subfolders. Here's what that looks like:
mkdir /mnt/disks/FS1/Blu-Rays/Bullet Train
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/META
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/PLAYLIST
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/STREAM
So all subfolders were created on Frankenstore, which wrote to the existing path inside Bullet Train which was created on FS1.
EDIT: Unfortunately, I just had a minor failure on this new logic. I had a folder inside an otherwise empty folder, so instead of being 5 layers deep it was 6 directories deep:
/mnt/disks/Frankenstore/Blu-Rays/Butch Cassidy and the Sundance Kid/47030325_BUTCH_SUNDANCE
That 6th folder is the folder rip. So in this case, the logic thought it was creating the 6th folder, but the 5th folder didn't exist yet and it didn't know this, so it allowed the 6th folder to be created in Frankenstore, which put it on FS1, bypassing my new logic to put it on the next drive.
So I just added a new check, to see if the parent folder exists when the directory level is 6 deep. If the parent folder is missing, then it will still trigger the new override logic to create the folder on one of the FS# drives.
EDIT 2: The new-new logic appears to be working as desired. My Butch Cassidy folder was correctly created on FS2 instead of FS1. I updated the revised mirror.sh code in the previous post to reflect this change.
At this point, my backup is running on it's own. I'll continue to monitor, though it looks like it really is correctly filling up each backup drive, one-by-one, and not erroring/crashing on drive transitioning.
It was quite a bit more effort to modify that mirror.sh bash script, I had tons of typos and mistakes. But I finally figured out the correct syntax.
Additionally, I had to add one more IF clause to evaluate how deep the directory is before overriding where it is created. If the directory depth is 5, then it will be overridden from my Frankenstore MergerFS mount point to one of the FS# mount points. I only do this for depth of 5 as my Unraid share name is depth 4, i.e.
/DEPTH1/DEPTH2/DEPTH3/DEPTH4/DEPTH5
/mnt/disks/Frankenstore/Blu-rays/DEPTH5
So by overriding only depth 5 directories, I override where the movie folder is written, but not the subfolders. This was important for folder rips, which have a ton of subfolders. What I was finding is that the parent movie folder might get created on FS1, but as data was written then future subdirectories got split over to FS2 because FS1 was too low on free space.
But since I already had plenty of buffer free space, I didn't want those subdirectories to split off like that. So by not overriding the subdirectories, they get written to the Frankenstore mount point, which lets MergerFS decide where to write them, and it will keep them in the original parent's directory.
For example on Bullet Train, I override the main folder but not the subfolders. Here's what that looks like:
mkdir /mnt/disks/FS1/Blu-Rays/Bullet Train
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/META
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/PLAYLIST
mkdir /mnt/disks/Frankenstore/Blu-Rays/Bullet Train/BDMV/STREAM
So all subfolders were created on Frankenstore, which wrote to the existing path inside Bullet Train which was created on FS1.
EDIT: Unfortunately, I just had a minor failure on this new logic. I had a folder inside an otherwise empty folder, so instead of being 5 layers deep it was 6 directories deep:
/mnt/disks/Frankenstore/Blu-Rays/Butch Cassidy and the Sundance Kid/47030325_BUTCH_SUNDANCE
That 6th folder is the folder rip. So in this case, the logic thought it was creating the 6th folder, but the 5th folder didn't exist yet and it didn't know this, so it allowed the 6th folder to be created in Frankenstore, which put it on FS1, bypassing my new logic to put it on the next drive.
So I just added a new check, to see if the parent folder exists when the directory level is 6 deep. If the parent folder is missing, then it will still trigger the new override logic to create the folder on one of the FS# drives.
EDIT 2: The new-new logic appears to be working as desired. My Butch Cassidy folder was correctly created on FS2 instead of FS1. I updated the revised mirror.sh code in the previous post to reflect this change.
At this point, my backup is running on it's own. I'll continue to monitor, though it looks like it really is correctly filling up each backup drive, one-by-one, and not erroring/crashing on drive transitioning.