Decorum for the Forum:
  • Be nice. If you want to be mean, try Reddit.
  • No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
  • No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
  • No Spamming. If you want to make a buck, work smarter... somewhere else.
  • No Adult Content. Half the internet is dedicated to adult content. This half isn't.

Privacy Policy: Click Here to Review (updated September 30, 2020)

What is Unraid and how to build an Unraid media server

Show off your HTPC builds, NAS Servers, and any other hardware. Great place to ask for hardware help too.
User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: What is Unraid and how to build an Unraid media server

Post by Pauven » Wed Jan 29, 2020 2:09 pm

Litlgi74 wrote: Wed Jan 29, 2020 1:16 pm If adding a cache drive to the system will automatically populate the drives according to the share settings... I'm going to go that route.... But will the cache drive start moving data to the array as soon as it reaches it's capacity?

Uhhhmmmm, I think you missed this in my post above:

Pauven wrote: Wed Jan 29, 2020 1:16 pm you do NOT want to use a cache drive. It will get in the way.

A cache drive just acts like a small buffer. For example, I have a 2TB cache drive, but I keep a lot of data on it, so I only have about 600 GB available for caching. If I am copying to my Movies share which has 5 TB free, I can only copy 600 GB before I get a disc full error, because the copy is to the cache drive, not the array.

If I want to copy more than 600 GB, then I would have to copy in multiple passes, i.e. copy 500-580 GB, stop, run the Mover, wait for the Mover to finish copying/moving to the array, then copy my next batch of data. This completely defeats the purpose of the cache drive (faster immediate copying of smaller amounts of data, overnight moving to the array when you're asleep), and only makes your job harder, so don't use a cache drive for large scale data migrations.


Jamie wrote: Wed Jan 29, 2020 12:51 pm You can use Dolphin or Krusader on a share if the size of all the combined data you are moving is smaller then the smallest drive.

What Jamie is saying is that you can use Dolphin or Krusader to copy up to 3TB of data at once, since your smallest Unraid drive is 3TB. This assumes you are copying 3 TB of data directly to an empty 3TB drive.

If you are copying to the shares, then the math gets harder. You have to see which Unraid drive has the least amount of free space that is also more than the Minimum Free Space you set for that share. So for example, if you have 1 TB free on one of your disk1 3 TB, and your min free space is set to 150 GB, then using Krusader/Dolphin to copy to that Unraid share it will pick disk1 (since it has more than 150 GB free), but because you only have 1 TB free if you are trying to copy more than 1 TB you will get out of space errors.


You seem to have some preconceived notions about how Unraid works, probably based upon your experience with FreeNAS and other NAS products. Unraid is different and unique. While the benefits of Unraid are great, the design does make this particular data migration task harder than you would expect.

Long story short, you keep looking for a shortcut to copy a complete share from your FreeNAS to a single share in Unraid, and unless you use the right tool, this WILL NOT WORK. And to be clear, Krusader and Dolphin are NOT THE RIGHT TOOL for this. If you are willing to copy smaller amounts of data, directly to each disk, where you manually manage the amount of data copied to prevent from running out of space, then Krusader and Dolphin are fine.

The ONLY way to accomplish your goal of a FreeNAS share to Unraid Share copy in one pass is to use a tool that copies 1 folder at a time, just like Jamie stated, and doesn't create each folder until it copies it. To be clear, I don't know a tool that does this, other than that excerpted line of code I provided above from the Unraid Mover script, which processes each directory one by one.

And it is at this point I would recommend you turn to the larger Unraid forums community for more help on this. I've never done a data migration like this with Unraid. Jamie has, so some of what I know came from his own experience with his migration, plus my experience with having shares run out of disc space when I did not use them correctly. Many of the users on the Unraid forum are 10x smarter than I am when it comes to this stuff. They might be able to point you to a tool that will work correctly to copy the data the way you want. Or they might be able to help you customize that line of code I shared above, so that it will correctly copy the way you want.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: What is Unraid and how to build an Unraid media server

Post by Jamie » Wed Jan 29, 2020 2:18 pm

Litlgi74 wrote: Wed Jan 29, 2020 1:16 pm
.You can use Dolphin or Krusader on a share if the size of all the combined data you are moving is smaller then the smallest drive.
I really don't have the time to sit and calculate which files are going to go where and on which drive. If adding a cache drive to the system will automatically populate the drives according to the share settings... I'm going to go that route.... But will the cache drive start moving data to the array as soon as it reaches it's capacity?

Thanks Again
John
I don't know if adding a cache to the equation is going to work either if you use Krusader and Dolphin. You're trying to trick the share to do it's job properly but the problem is not with the share. The problem is with the tool that your using to do the migration. Paul will know more on what to expect by adding the cache. I have not explored getting a cache since I manually rip my discs to the share.

Litlgi74
Posts: 12
Joined: Mon Jan 27, 2020 5:08 pm

Re: What is Unraid and how to build an Unraid media server

Post by Litlgi74 » Wed Jan 29, 2020 4:13 pm

Thanks again to the both of you for all your help...

I have one final question.... So when do the automatic settings of the share (fill-up and split level) actually happen/work? This is the part I am struggling with.

John

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: What is Unraid and how to build an Unraid media server

Post by Pauven » Wed Jan 29, 2020 5:33 pm

Fill-up & Min Space Free is evaluated at folder creation, but not data copy. <-- Read this a few times and make sure you understand it.

This may help. This is how I first discovered the limitation. I was ripping a bunch of movies, and went ahead and created the folders ahead of time (mistake!). So I had something like this:
  • \\Tower\Movies\Movie_1
  • \\Tower\Movies\Movie_2
  • \\Tower\Movies\Movie_3
  • \\Tower\Movies\Movie_4
  • \\Tower\Movies\Movie_5
Keep in mind that my Movies share has a split level of 1. All of these directories got created on disk3, which had plenty of space. So from a disk perspective, the folders looked like this:
  • \\Tower\disk3\Movies\Movie_1
  • \\Tower\disk3\Movies\Movie_2
  • \\Tower\disk3\Movies\Movie_3
  • \\Tower\disk3\Movies\Movie_4
  • \\Tower\disk3\Movies\Movie_5
I then proceeded with ripping ISO's for all 5 movies. Movie_4 failed (bad disc), so I deleted the contents of the Movie_4 folder, but left the directory.

I got a replacement for Movie_4, but it took a few weeks. By that time, I'd ripped a lot of movies, and filled up a disk and Unraid was now writing to disk4. I re-ripped Movie_4, putting it in the original folder I had created weeks earlier, and quickly got an "Out of Space" error. I really thought this was an Unraid problem, and promptly opened a support ticket, since it was obviously writing to disk3, not disk4.

But the real issue was my assumption was that Unraid would somehow move the old directory to a new disk with available space. Because I had set the split level to 1, Unraid was keeping the directory intact, and since it already existed on disk3, then all future writes to that directory had to be on disk3. I had to delete the Movie_4 directory and re-create it, and this time Unraid created the directory on disk4.

So Unraid decides where to put a directory at the time the directory is created, based upon the split level and the allocation method, but does not re-evaluate this decision once you start copying data. This is why tools like Krusader and Dolphin don't work well, because they make two passes: pass 1 creates the directories, which all end up on the same disk because empty directories don't take any space, then pass 2 fills the directories with data, which if you're copying too much data causes that disk to fill up and not roll to the next disk.

Now, if you had a tool that both creates & copies a single directory at a time, then the problem goes away, because once the disk is full, the next directory copied still has to be created, and it will be evaluated and placed on the next disk.

This is actually one of the benefits of the cache drive. Say that I'm currently filling up disk15, and it has 320 GB free, and the next disk is disk16 with 3TB free. I can rip 10 Blu-rays, each 45 GB in size for a total of 450 GB, and while I'm ripping to my Blu-rays share, behinds the scenes Unraid is actually writing to the cache drive. Then in the middle of the night the Mover kicks in, and starts copying one directory at a time. Since my Blu-ray's Min Free Space is set to 180 GB, the following occurs:
  • Mover moves Movie 1, disk15 now has 320 GB free so it is used
  • Mover moves Movie 2, disk15 now has 275 GB free, so it is used
  • Mover moves Movie 3, disk15 now has 230 GB free, so it is used
  • Mover moves Movie 4, disk15 now has 185 GB free, so it is used
  • Mover moves Movie 5, disk15 now has 140 GB free, which is below 180 GB min, disk16 is used
  • Mover moves Movie 6, disk16 now has 2955 GB free, so it is used
Note that the above is greatly simplified. In reality, Unraid evaluates every disk, from disk1 to disk20, to find the lowest # disk that has more than the min free space setting for that share. But the main take-aways here are that Unraid evaluates which disk to write to when the folder is created, if the folder "Level" is the configured Split Level, and also that even though I had the Min Free Space set to 180GB, that's before the copy, so after the copy is complete that disk was 40 GB below the min free space level.


Now, regarding the Split Level, here's the overall layout plus some examples:
  • \\Server\Share\Level1\Level2\Level3\Leve4\Level5\etc.
  • \\Server\Movies\Artic\Disc1\VIDEO_TS
  • \\Server\Movies\Dunkirk\Dunkirk.ISO
  • \\Server\TVShows\Rome\Season1\Disc1\VIDEO_TS
  • \\Server\TVShows\Rome\Season1\Disc2\VIDEO_TS
  • \\Server\TVShows\Rome\Season2\Disc1\VIDEO_TS
  • \\Server\TVShows\Rome\Season2\Disc2\VIDEO_TS
The best way to think of the Split Level is probably this: it is the Level at which Unraid re-evaluates which disk to write to when creating a directory at that level.

For a "Movies" Split Level of 1, Unraid would evaluate where to create each Level 1 directory (Artic, Dunkirk), but everything inside of those directories will always be written to that same disk. So as soon as you create a new folder named "Frozen" in Movies, it evaluates where to put that folder, then all writes will be to that folder on the same disk.

For a "TVShows" Split Level of 2, Unraid would evaluate where to create each Level 2 directory (Season1, Season2), and everything inside of those directories will always be written to that same disk, but the parent Level 1 folders (Rome) could exist on multiple disks. For example, say HBO decides to make a Season 3 of Rome and I want to add it to my collection. Perhaps Seasons 1 and 2 were written to disk5 years ago, which is now full. Like this:
  • \\Server\disk5\TVShows\Rome\Season1\...
  • \\Server\disk5\TVShows\Rome\Season2\...
The Rome directory already exists (on disk5), so I don't have to recreate it. Instead, I just create a new subfolder named Season3. When I create this folder, Unraid knows it is at Level2, and because TVShows is configured as a Level 2 split, it evaluates where to create the folder - in this case perhaps it gets created on disk14. Everything I rip to Season3 will stay in that same folder on disk14. At the same time, if I add some new metadata to Seasons 1 & 2, those new files will still be written to disk5, since that's the disk those folders were created on:
  • \\Server\disk5\TVShows\Rome\Season1\...
  • \\Server\disk5\TVShows\Rome\Season2\…
  • \\Server\disk14\TVShows\Rome\Season3\...
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: What is Unraid and how to build an Unraid media server

Post by Jamie » Wed Jan 29, 2020 5:35 pm

On what I call a sequential file transfer,it happens when the minimal space allotted for the share kicks in. When that kicks in it knows that it must create the proper folders and data to continue on to the next drive. With the Krusader and Dolphin tools which populate all the folders first and then populates the data, the share doesn't realize that it is time to move on to the next drive. The leveling trigger just never kicks in.

Maybe if you just used the top level for a break, the share may give you what you wanted. The problem is that it may populate over a lot more drives.

I still think your trying to force Krusader or Dolphin to do something that they weren't designed to do. Your trying to fix the problem by fixing the share when the root of the problem is the tool.

I think the best thing to do is to hookup your NAS, or whatever device you are transferring to the Unraid server and run a linux script to populate your data. I am no longer good with the greps and all the other commands, so I decided to work with what I knew best which was the windows explorer.

Jamie
Last edited by Jamie on Wed Jan 29, 2020 5:51 pm, edited 2 times in total.

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: What is Unraid and how to build an Unraid media server

Post by Jamie » Wed Jan 29, 2020 5:47 pm

After reading Pauls message, I don't think designating a level 1 share will help.

I still think of Unraid as magic instead of getting down to it's inner workings.

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: What is Unraid and how to build an Unraid media server

Post by Pauven » Wed Jan 29, 2020 6:09 pm

Jamie wrote: Wed Jan 29, 2020 5:47 pm I still think of Unraid as magic instead of getting down to it's inner workings.

I started using Unraid in 2009, so I've had a good decade to learn a thing or two. Jamie's barely dipped a toe in the water, though he's doing great.


Jamie wrote: Wed Jan 29, 2020 5:47 pm After reading Pauls message, I don't think designating a level 1 share will help.

Agreed. For a big migration like this, the split level isn't the issue. The issue is that the directory should not be created until it is needed.

And if you create a directory and end up not being able to use it (bad disc or something), then remember to delete the directory. That directory is a placeholder on a disk, and at some point in the future if you write to that directory, it goes to that disk, even if that disk doesn't have enough free space anymore.

This is why I always try to leave 500 MB to 2 GB free on every disk, so that any movie metadata written to those same directories on the disk don't run out of space.

I keep my Min Free Space limits high enough so I don't have to think about it, but that also means that eventually if I want to fill up a disk I'll have to do it manually. So once I get down to around the 50-60 GB level (which is where my DVD share will get me), I'll grab a stack of discs to rip, and use a spreadsheet and log how big each disc is before I rip them. Then I will rip a combination of movies that gets me into that 500-2000 MB range of space remaining. Sure, it's manual and a bit of a pain, but it's only something I bother with on rare occasions. As you can see, my extra effort has allowed me to get the most out of my disk space.

image.png
image.png (52.6 KiB) Viewed 8193 times
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Litlgi74
Posts: 12
Joined: Mon Jan 27, 2020 5:08 pm

Re: What is Unraid and how to build an Unraid media server

Post by Litlgi74 » Wed Jan 29, 2020 6:22 pm

Wow... Thanks Paul!

I almost feel guilty not responding in such detail. Lol.

I had a thought... Since my original FlexRAID drives are NTFS... Maybe this might work.
  • Create a Windows VM
  • Map the FlexRAID drives in Windows via Unraid unassigned drives plug-in
  • Transfer the data from the mapped FlexRAID drive to the appropriate Unraid share via Windows explorer
  • If using a cache drive will ensure population of the drives according to my share settings... I'll order one now
I almost hate ask... But what do you think?

Jamie... All my FlexRAID and Unraid drives are in one 45 bay JBOD enclosure connected to an HBA card on a single PC. So having two systems other than a VM and Unraid on the same PC isn't really possible.

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: What is Unraid and how to build an Unraid media server

Post by Pauven » Wed Jan 29, 2020 6:39 pm

Litlgi74 wrote: Wed Jan 29, 2020 6:22 pm If using a cache drive will populate the drives according to my share settings... I'll order one now

You would need a cache drive bigger than your array to use it for a big data migration like this. I don't know how many times I can say this: DO NOT USE A CACHE DRIVE FOR A BIG DATA MIGRATION. Sorry for yelling, but this is the 3rd time I've written this, and you keep suggesting a cache drive will somehow work for this - it won't. A cache drive is copying smaller amounts of data that can fit on the cache drive, not bigger amounts like you need to copy. And seriously, if you had a cache drive that large, you wouldn't even need Unraid...


Litlgi74 wrote: Wed Jan 29, 2020 6:22 pm I almost hate ask... But what do you think?

I don't like this idea at all, sorry. It doesn't make sense to me. If you are using Windows to manage the copy, then you can copy via Windows (no matter where it lives or whether it is a VM or real) from your FlexRAID to Unraid.

You would only need to use the Unassigned Devices plugin to mount an external NAS as a local drive if you wanted to use Unraid to manage the copy, but if you are using Windows, then this doesn't make sense at all.

Regarding Windows Explorer, I don't know how it does the copy - I've never tested to see if it creates all the directories and then copies data, like Krusader/Dolphin, or if it creates directories & copies one-by-one. Even though Jamie used a Windows PC and Windows Explorer to migrate his data from his Drobo's to Unraid, I think he still had to make sure he didn't copy too much data at once, so my guess would be that Windows Explorer acts like Krusader/Dolphin and has the same issue. But I really don't know, so I guess you could try.

And if you are willing to let a Windows PC/VM manage your copy for you, that really does open up a whole bunch of copy programs you can try. I used to use TeraCopy, which is great for copying large amounts of data. But no matter what tool you use to copy, you'll need one that will only copy 1 directory at a time, before creating and copying the next one. It should be easy to test. Just copy two Blu-ray movie folders, monitoring the target Unraid share when you start the copy. If you see both movie folders show up before the first movie has copied, that tool is no good.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Jamie
Posts: 942
Joined: Wed Dec 27, 2017 11:26 pm

Re: What is Unraid and how to build an Unraid media server

Post by Jamie » Wed Jan 29, 2020 6:49 pm

You might be able to use midnight Commander. Paul mentioned it as a possibility somewhere in this thread. I didn't try it because I had the Windows option and didn't feel like working with it at the time.

Post Reply