CMC Forums

just an FYI, Krusader and Dolphin propagates all the top level folder names first, then propagates the underlying folders and files. Some people have specified that this issue prevented the first level folder to propagate to the second drive when space runs low. I think that this maybe the issue here because I deleted a title on my share that would not copy due to disk space issues and copied the title back to the share and it went to the second drive with no issue.

My folder structure is or will be this. I will probably work on moving the TV series to a separate share after everything is migrated, but my desire is to get everything off the Drobos before I start that process. I will probably use the unbalance plugin for that job because from my research it is well suited for doing that type of migration.

Here is my share setup. Also can you change a share name after there are files in the share? I might want to rename this share later.


I am using a share mymovies here.

I think that for now I just want to have the files/folders split at the top level movie/boxset/tvshow folder. Like this.

Tower - mymovies - 2001 a space odyssey
Tower - mymovies - Game of thrones
Tower - mymovies - Harry potter

Here is my current structure.

ISO Movies structure

for regular movies
Tower - mymovies - 2001 a space odyssey - iso with metadata

for tv shows
tower - mymovies - Game of thrones - GOT Season 1 - GOT season 1 Disk 1 - iso with metadata.
tower - mymovies - Game of thrones - GOT Season 1 - GOT season 1 Disk 2 - iso with metadata.
tower - mymovies - Game of thrones - GOT Season 2 - GOT season 2 Disk 1 - iso with metadata.

for boxsets
tower - mymovies - Harry Potter - Harry Potter and the sorcerers stone - iso with metadata
tower - mymovies - Harry Potter - Prisoner of Askaban - iso with metadata

VIDEO_TS structure
Tower - mymovies - 2001 a space odyssey - VIDEO_TS

tower - mymovies - Game of thrones - GOT Season 1 - GOT season 1 Disk 1 - VIDEO_TS.
tower - mymovies - Game of thrones - GOT Season 1 - GOT season 1 Disk 2 - VIDEO_TS.
tower - mymovies - Game of thrones - GOT Season 2 - GOT season 2 Disk 1 - VIDEO_TS.

tower - mymovies - Harry Potter - Harry Potter and the sorcerers stone - VIDEO_TS
tower - mymovies - Harry Potter - Prisoner of Askaban - VIDEO_TS

Jamie wrote: Thu Jul 25, 2019 10:47 am just an FYI, Krusader and Dolphin propagates all the top level folder names first, then propagates the underlying folders and files. Some people have specified that this issue prevented the first level folder to propagate to the second drive when space runs low. I think that this maybe the issue here because I deleted a title on my share that would not copy due to disk space issues and copied the title back to the share and it went to the second drive with no issue.

That is a huge issue - this behavior is not compatible with the way shares span multiple drives and need to split. When the folders are created, there is sufficient space, but then when the data is copied in much later, the data is forcible written to the drive where the folder exists, and space runs out. This is similar to how I first discovered the share-split issue, as I had created multiple directories on a share, then later copied data into them and ran into the same problem.

Note that I don't think the 'Unbalance' plugin will help here, as Krusader/Dolphin propogating the folder names first is going to lead to overfilling disks. Once you solve that issue, then the need for the 'Unbalance' plugin is gone (unless you have other goals).

Regarding your folder structures:

\\tower\mymovies\2001 A Space Odyssey\ <--Split at top one directory level
\\tower\mymovies\Game of Thrones\GOT Season 1\GOT Season 1 Disk 1\ <--Split at top two directory levels
\\tower\mymovies\Harry Potter\Harry Potter and the Sorcerers Stone\ <--Your choice, split at top two directory levels for carefree handling, or split at top one level if you really want to keep all movies on the same drive (but be prepared for manual effort to make this happen)

There is no difference in split for ISO vs. Folder - the directory levels count from the left, not the right.

I'm not sure if you can rename a share later - I think the answer is yes. You should try now before you get too far into it. Worst case, you could take a few manual steps to rename a share later, though I think the GUI will handle this scenario okay.

One idea - not sure if this will help or not - The way you select the source data to copy might be a factor.

For example, let's say you have a Drobo like this:

\\Drobo1\mymovies\movieA
\\Drobo1\mymovies\movieB
\\Drobo1\mymovies\movieC

If you go to the Drobo1 root and select to copy "mymovies", which includes all of the sub-folders, it might be that the behavior is different than if you go inside the mymovies folder and individually select movieA, movieB and movieC and then do the copy.

My guess is that you are already doing the latter, which is what I would recommend, so this won't help, but thought I should mention this in case you are somehow doing the former.

Another option may be to use Midnight Commander instead of Krusader or Dolphin. In my experience, MC when used with the latter option above, it doesn't create the target parent folder until right before it copies the data inside.

Tip on using Midnight Commander - use a Windows Putty session to connect, and even though it will look like a legacy DOS/console screen, you can actually use your mouse to navigate and right-click to select.

Thanks for your advice Paul. I will give midnight commander a shot. If that doesn't do what I want I'll use windows though it may cut my time down a half or third.

Right now all my drobos are structured this way.

Drobo1/volume_1/Drobo_X/mymovies/titles
Drobo2/volume_1/Drobo_y/mymovies/titles.
Drobo3/Volume_1/Drobo_V/Mymovies/titles.

If I copy the whole Drobo_X folder then dolphin, or Krusader, might work for me. Giving me

Tower/Drobo_X/mymovies/GOT
Tower/Drobo_X/mymovies/2001 a space odyssey
Tower/Drobo_X/mymovies/harry_potter
Tower/Drobo_Y/mymovies/GOT
Tower/Drobo_Y/mymovies/2001 a space odyssey

Would I split on the 2nd or third folder here? I am thinking third. I am only concerned with breaking on the titles.

Jamie wrote: Thu Jul 25, 2019 12:05 pm Would I split on the 2nd or third folder here?

2nd.

Also, Midnight Commander is included with Unraid. To run it, from the console prompt you enter mc.

Another idea might be to do the copies in smaller groups. In Windows, it's easy to select a bunch of folders and do a properties on them to see the total file size. Select a group of folders that is 10% smaller than a drive size, then copy them. Then copy one movie at a time until Unraid advances to the next disk, then go back to copying a big group of folders 10% smaller than the next drive.

You gave me a few good solutions, Thanks.

I will use your method if I try windows. I will use windows as a last resort because I think that it will increase the migration time by a factor of 2.

Right now I'm getting 1 TB every 6 hours using unraid to do the migration.

Jamie

Jamie wrote: Thu Jul 25, 2019 1:51 pm Right now I'm getting 1 TB every 6 hours using unraid to do the migration.

It was taking me around 18 hours to migrate 3TB drives, so that sounds about right.

okay last message and I will not bug you again today.

I tried to copy the Drobo_X folder as discussed before using dolphin and krusader. Looking at the disk1 and the the share through Midnight commander while the copy was happening, maybe 10 minutes, I noticed that all the sub folder's names down to the bottom most level were propagated. Just no files. I had cleaned the share and disk 1 completely before kicking off the copy.

I have decided to use windows on one of my NUCs to do the migration. I will do 2 TB per copy which probably will take 18 - 24 hours I believe. I will know more after a few tests. I have to get a KVM switch and a dvi cable first. That should come from amazon on Sunday.

Thanks for convincing me not to try to not do this all in one copy. The first drobo is just under 10 TB.

By the way how do you and how often do you backup your flash drive??

Jamie

On a rare occasion I've made a backup copy of my flash drive, sometimes to a Windows PC (\\Tower\flash), sometimes to a Share in my array (but that's not so great, as if you have flash drive issues, you probably have array access issues too).

Here's some guidance on recovering from a failed flash drive: https://forums.unraid.net/topic/58685-b ... usb-drive/

In general, you risk corrupting your array if you try to recover your flash drive from an out-of-date backup that doesn't have your current drive assignments. It's actually better to reassign your drives from scratch (I've had to do that before). And if you come up with a drive organization system like I have (bottom left is Parity 2, bottom right is Parity 1, all data drives are 1..20 left to right, top to bottom, and two bottom middle drives are not in the array), then if I had to assign my drives from scratch it would be very easy.

I still think you can use Krusader/Dolphi/MC successfully without going through your Windows PC as an intermediary.

You would use your Windows PC just to determine the size of a group directories, like this:

In the screenshot above, I've selected 10 directories, and doing a right-click Properties I see that is 366 GB. If you want to copy 2 TB, then select a bigger group of directories until you get to the size you want. Aim for about 100 GB less than full target disk size, so if your target disk is 2 TB, then try to get 1.90 TB worth of movie directories.

Then copy those same directories to the Unraid mymovie Share. Check the directory group size in Windows because it is super easy, but do the copy using Krusader/Dolphin/MC.

That will take the 12+ hours as you expect. Come back when it's done, and if the disk free space is now below the share's min free space (you set to 200 GB, so the strategy above should put you below that threshold) then you are ready for the next batch. Follow the same steps as above, but obviously with a different group of movie folders that you haven't copied yet.

Unraid should automatically send each batch to the next drive, as you've filled up the previous drive beyond the min space threshold, and you won't run out of space because you are doing smaller batches that are less than the next drive's freespace.

Size each batch to essentially fill up the next drive in your array (still aiming for 100 GB less than each disk's size).

One good thing about this approach is that you are doing smaller batches, so if there's a problem (i.e. power outage, network hiccup, etc.), I find it easier to manage recovery with a smaller group of directories versus a larger group.