What is Unraid and how to build an Unraid media server

[Pauven]

Hi Litlgi74, welcome. Glad to hear you are interested in Unraid, it's a great solution. You've asked some very good questions, and I'll do my best to provide good answers. Sorry in advance, this is going to be a long one...

I primarily use ISO rips, except for TV Series where I primarily use Folder rips. But as the developer of CMC, I have to maintain examples of all media types for development and testing, so I have both movies and TV Series as ISO, Folder, and encoded like MKV.

In general, I do not find that the media type affects my shares so much. Rather, it is the file size that I worry about most, and that is what I factor into my share settings.

First, I find several benefits of splitting my media across multiple shares:

• I find it quicker to navigate to what I'm looking for when browsing the directories
• It helps identify My Movies folder monitoring mistakes (i.e. Movie 'XYZ' identified as Blu-ray, but stored in \\Tower\DVDs\XYZ)
• I can use a different split level for TV Series (Automatically split top 2 levels, vs. top 1 level for 4K/Blu-rays/DVDs)
• I can configure different minimum free-space for each disc type (except TV Series), making it easier to fill up without issues

The reason I have the split level at 2 for TV Series is because I store them as \\Tower\TV_Series\Show\Season, so setting it to 2 keeps entire Seasons together on the same disk (where-as a split level of 1 would keep the entire Show with all seasons together on the same disk).

You can think of split levels like this \\Server\Share\Level1\Level2\Level3\Level4. So with a split level of 2, you could have

• \\Server\disk1\Elementary\Season 1
• \\Server\disk2\Elementary\Season 2

So level 2 split, which each folder can be moved to a different disk. If I used split level 1 for my TV series, then all of 'Elementary' would have to be on the same disk. Imagine trying to find room for 12 seasons of The Big Bang Theory on Blu-ray (30 + discs, over 1TB of data) on a single array disk. Doubly hard if you've been collecting each season one-by-one as they come out, and you've ripped them years apart. It just won't be possible to use a split level of 1 for TV Series organized like that - in fact if you do, then at some point you will rip a new season, and it will try to write it to the previous disk that is now already full, and you will get errors.

Which brings us to the minimum free space settings.

• For DVD's, I have minimum free space set to 60GB

• For Blu-rays, I have minimum free space set to 180GB

• For UHD, I have minimum free space set to 100GB

• For TV Series, I have minimum free space set to 100GB

As you can see, I'm a little all over the map. But there is a method to my madness. It helps to understand how fill-up works with the split level, and also that I use a cache drive.

Let's say I have a Blu-ray movie box set with 4 discs (crazy, but possible), and I rip all 4 discs to the same folder in my Blu-rays share, i.e. \\Tower\Blu-rays\4Disc_BR_Movie. As I rip them, they all go to my cache drive for the time being, so they actually exist at \\Tower\cache\Blu-rays\4Disc_BR_Movie.

I also rip another 1 disc Blu-ray movie, and that ISO is a large 47 GB rip.

Then in the middle of the night (or earlier if I invoke the Mover manually) the Mover moves the folder 4Disc_BR_Movie from cache to the main array share. Before moving that folder, it will look for the first HDD that has at least 180 GB free. Let's say it finds disk 12 has 195 GB, and moves it there.

Now, each of these 4 Blu-rays happened to be fairly large, around 42GB each. 42 GB x 4 = 168GB. After the move, my disk 12 now has 27 GB free. While that 180 GB min free space seemed really high a few minutes ago, now it seemed like it was barely enough. At this point, the next Blu-ray movie folder to copy, that second 47 GB rip, has to find a new disk, because disk 12 no longer has at least 180 GB free.

Now let's think about this another way - what if I had set my Blu-rays min free space to 50GB - after all, that should be large enough for any single rip, right? But my 4Disc_BR_Movie has 4 discs inside of it, and the split level is set to 1 (which means that the folder will NOT split). The Move kicks in to move that folder from the cache to the array, but this time it finds 86 GB free on disk 4, so it copies there instead. The first two ISO's, being a collective 84 GB, move fine, but then the third one fails when disk space runs out, because I'm trying to movie a 168 GB folder to a disk with only 86 GB free.

So my sizing logic is this: for DVD's, Blu-rays, and 4K, how many discs do I think might be in a single folder, and how much free space do I want left over as a safety net? For Blu-rays, I typically don't expect more than 3 or 4 ever in a single folder (for a box set, perhaps something like Indiana Jones, Lethal Weapon, etc.), so I take an above average size (i.e. 45 GB) x 4, or around 180 GB.

For 4K, I have so few of these (expensive), and typically not box sets, so I have it set to 100GB, which is the max possible size of any 1 4K disc (very rare, most are 75GB or less).

For DVD's, I went higher than 4x 8.5 GB (34GB) as that was getting uncomfortably low on space to me. So I just rounded up to 60 GB (or around 7 discs).

For TV Series, I have DVD, Blu-ray and 4K all mixed together, so I don't really have a 1 size fits all. Instead, what I do is occasionally adjust the min free space setting before doing a rip, especially if I am doing a multi-disc 4K TV series. That's one of the nice things about Unraid, you can change the min free space anytime, and you don't even have to stop the array.

I also might adjust the min free space setting on the other shares too, as needed. For example, if I am ripping something like the 8-disc Harry Potter series on Blu-ray, I'd probably crank up the min free space to 360 GB, then drop it back down to 180GB after the next mover run.

My ultimate goal with the min free space is to be able to rip a disc and not have to worry about the mover failing because a HDD ran out of space during a move. I'd rather have 179 GB of free-space go un-used than to have a folder move fail.

Every so often, when I see a disk gets below the lowest threshold (i.e. 60GB for DVD's), then I start manually ripping straight to a disk, bypassing the share and the cache. For example, say I have 20 GB free on disk 3, I might rip 3 or 4 DVD's, carefully chosen to use up around 18 GB, to \\Tower\disk3\DVD's\. That allows me to fill up and use every last nook and cranny.

Hope that helps.

Paul

[Litlgi74]

Wow... Thanks for the reply!

I think I'll take your advice on shares and minimum free space.

Can individual shares be created after the migration of data to the Unraid array? I plan on using Krusader to do the transfer of one drive at a time. Most of my drives have Blu-ray, DVD, 3D, UHD, TV_Shows folders on each drive... Since I have many disks to transfer...I'd rather do the migration as a whole disk vs. copying the individual folders Blu-ray, DVD, 3D, etc.

I'm still a little fuzzy on the split level... But I think I have the jist of things.... My main concern was with BD and DVD folder structures. I think I would be fine with split level one for each as long as the folders where actually "movies"... But when it comes to TV Shows... I have DVD, BDs, and UHDs to contend with. Do you keep all formats of TV Shows on one share?

Thanks again for the quick and thorough response.
John

[Pauven]

Hey John,

Yes, you can add/modify/remove shares at anytime. Yes, I keep all TV together on one share, and adjust the min free space based upon what I'm ripping to get the desired result.

What you're describing sounds very similar to the path that @Jamie just took. Perhaps he can share his perspective now that he's completed this migration, and created shares afterwards.

I too have struggled to understand split levels, and to be honest I still do a bit. To me, the biggest problem with split level is that it is misnamed. It's more of a "Don't Split Level", though I prefer to think of it as a "target disk evaluation folder level", not that that helps much. Here's what I mean by that:

Here's some real world examples of how a split level would work, using a Lethal Weapon box set of all four movies on dual-sided DVD, so there's actually 8 DVD's ripped. Let's say I rip them as follows:

• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon\Side A
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon\Side B
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side A
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side B
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side A
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side B
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side A
• \\Tower\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side B

Behind the scenes, the DVDs share is allowed to use any of my discs. Now let's say I have a brand new Unraid server with zero data, and 10 x 3TB drives, and I'm using the "Most-free" allocation method which means every time the target disk is determined it will go with whichever drive has the most free space. In the beginning, all drives have 3TB free, so it defaults to the lowest drive number, disk1.

With "Automatically split only the top level..." I get this result:

• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon\Side A
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon\Side B
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side A
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side B
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side A
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side B
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side A
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side B

The target disk was only evaluated at the top level, for folder "Lethal Weapon Collection", and went to disk1, after which all writes to "Lethal Weapon Collection" went to the same disk.

With "Automatically split only the top two levels..." I get this result:

• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon\Side A
• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon\Side B
• \\Tower\disk2\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side A
• \\Tower\disk2\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side B
• \\Tower\disk3\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side A
• \\Tower\disk3\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side B
• \\Tower\disk4\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side A
• \\Tower\disk4\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side B

The target disk was evaluated at every 2nd level, so once for each movie folder like "Lethal Weapon 2", "Lethal Weapon 3", etc. Each time a 2nd level folder was evaluated, it went to the next drive with the most free space, so it split to disk2, disk3, and disk4, but kept the entire movie on each drive. This is why I use split level 2 for my TV Series, as it keeps seasons together on the same drive, but allows for a Series to spread across multiple drives.

With "Automatically split only the top three levels..." I get this result:

• \\Tower\disk1\DVDs\Lethal Weapon Collection\Lethal Weapon\Side A
• \\Tower\disk2\DVDs\Lethal Weapon Collection\Lethal Weapon\Side B
• \\Tower\disk3\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side A
• \\Tower\disk4\DVDs\Lethal Weapon Collection\Lethal Weapon 2\Side B
• \\Tower\disk5\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side A
• \\Tower\disk6\DVDs\Lethal Weapon Collection\Lethal Weapon 3\Side B
• \\Tower\disk7\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side A
• \\Tower\disk8\DVDs\Lethal Weapon Collection\Lethal Weapon 4\Side B

The target disk was evaluated at every 3rd directory level, so once for each disc folder like "Side A" and "Side B". Each time a 3rd level folder was evaluated, it went to the next drive with the most free space, so it split to disk2, disk3, all the way to disk8, but kept the entire disc on each drive.

If I did something crazy like use a split level of 4, then it would start to evaluate for each DVD Folder like VIDEO_TS, so a DVD Folder could be split across multiple drives. Yikes.

The big difference with the "Most-free" vs. "Fill-up" is when the evaluation occurs. With Most-free, the evaluation happens everytime a new folder is created at that split level, to determine which disk it should write to. With Fill-up, all writes continue to the lowest disk # that has more than the "min free space" amount, and only after it hits that amount on a disk will it then evaluate the next target disk. It still does that evaluation at that folder level specified by the split, same as Most-free, but only after a disk is "full" according to the min free space limit.

Now, here's the rub - there's no real right answer as to what split level you should use, it's up to you, and there are pro's and cons to both. Going with a higher split level allows Unraid to chop and dice subfolders into smaller bits and spread them out over more drives, making it easier to fill up space without worry.

But I don't like that if I had multiple drive failures beyond what my parity protects, that I could lose a small piece of a whole bunch of movies. I'd rather lose 80 entire Blu-ray Folders, rather than a small part of 800 Blu-ray folders, so I want to try and keep each Title together (or Season for TV) so that recovery, if necessary, is a straightforward 1:1 and not piecemeal. Also, when you play back a disc, every drive on which that disc resides must be spun up, so if you use a high split level, then you might have to spin up multiple drives just to play back one movie. That defeats the main benefit of Unraid to me, as I only want 1 drive to spin up to play 1 movie.

So my advice is to always go with the lowest split level that doesn't cause problems.

Paul

[Jamie]

Hey John,

It is great to see others using this topic on unraid. I started wondering whether anyone other than myself would benefit from my struggles of getting the equipment together and also migrating my Drobos to unraid. So far no problems.

If you use Krusader or Dolphin copy your files directly to a disc, not a share. The share will never default to a new drive and you will run out of disc space on the drive where the share is populating the data. I tried this several times and in my research people on the Unraid forum said that you should only use these two tools to write directly to the drives.

I created one share Drobo-Movies and added a folder "MyMovies" under the share. I then populated the Drobo folders and files to the Drobo-Movies share using my NUC PC. My setup is as follows.

I have an unraid server, A NUC PC, and a Drobo. All of these were attached the an ethernet router. I was fortunate to have a spare router where I could setup up a private network for these devices. I then mapped a network drive to my drobos and a network drive to the unraid share on my unraid server. The PC in effect was a traffic cop for the drobo and unraid server.

Yes the speed wasn't as optimal as connecting a drobo directly to the unraid server but I wanted the ability to walk away and let it run. I would monitor it from time to time. I would just swap in a drobo when each drobo was done copying.

I also set up a spread sheet with all the MyMovie Drobo folders and compared that to my new unraid folders to make sure that they matched up.

This took a little less than a month to complete. I started with 6 Drobos with a combined disk space = 60 - 65 TB. It took me about 3 days to copy the files from one Drobo. I did not continuously run the Drobos. I would copy half the files on the drobo and paste them to the unraid server.

It would take me at most 4 hours to verify the files and folders.

Jamie

[Litlgi74]

Hello Jamie and Paul...

Thank you both for taking the time to give very thorough and informative responses.

If you use Krusader or Dolphin copy your files directly to a disc, not a share. The share will never default to a new drive and you will run out of disc space on the drive where the share is populating the data. I tried this several times and in my research people on the Unraid forum said that you should only use these two tools to write directly to the drives

I had wonder(ed) if I am/was doing something wrong... My first try was Krusader failed... But I was writing directly to the drives at that point. So I deleted all the information from the first drive, and rebooted the system. I thought I read on the unraid forum somewhere that it was best to transfer directly to a share versus individual drives. So I tried transferring the data to share (at that point I only had one) using highwater and the top level split... Everything seemed to work fine until I added more drives to the array... Which apparently is a No-No for highwater. Once the new drives were added the first drive started to fill up to 100%... So I deleted all the data from the array again. That's when I found this thread.

So today I tried using Paul's method a fill-up and minimum space requirements.... Still transferring directly to the share via krusader, everything seem to be working properly. That was until I got to a TV show that was near the end of my second hard drive to be transferred. As I said earlier, most of my Rip's are in Blu-ray folder structure format... Which as you know, has many levels. I don't think I'd be experiencing any problems if my files were ISOs or MKVs... But that cannot be changed now. Anyway... Once Krusader got to the Twin Peaks Blu-ray rip on my TV shows share... The minimum space was ignored forcing (I think) the top two levels to be written to the first drive of the share... Tower/TV_Shows/Twin Peaks/TWIN_PEAKS_D8/BDMV/Stream/0000.m2ts... followed by all the other m2ts files as well.... Like I said, I think this is where the problem started.

It appears as though I'm going to have to convert all my movies and TV show episodes to MKV or ISO... just wish there was a simpler way to get all of these files and folders to the array before another one of my flexRAID drives fail...

If I were to continue to use fill up... And if I use the default split to any directory as required... Would that get all of my data to the array without over-running the drive minimum space limits?

John

[Pauven]

If you use Krusader or Dolphin copy your files directly to a disc, not a share. The share will never default to a new drive and you will run out of disc space on the drive where the share is populating the data. I tried this several times and in my research people on the Unraid forum said that you should only use these two tools to write directly to the drives.

Thanks so much for sharing your experience, Jamie. I had forgotten about that issue, and I'm pretty sure this is what John is struggling with.

I had wonder(ed) if I am/was doing something wrong... My first try was Krusader failed... But I was writing directly to the drives at that point. So I deleted all the information from the first drive, and rebooted the system. I thought I read on the unraid forum somewhere that it was best to transfer directly to a share versus individual drives. So I tried transferring the data to share (at that point I only had one) using highwater and the top level split... Everything seemed to work fine until I added more drives to the array... Which apparently is a No-No for highwater. Once the new drives were added the first drive started to fill up to 100%... So I deleted all the data from the array again. That's when I found this thread.

I think I see what is going on now.


First, what you read about using shares instead of drives for Krusader (or Dolphin) is either wrong or you read it backwards. You do not want to write directly to a share using these tools, here's why:

Let's say on your old server you have a 40 TB share, and on your new server you also have a 40 TB share made up of 5 x 8TB drives. Using Krusader/Dolphin, you try to copy your entire Media share from your old server which has 35 TB of data. The first thing Krusader/Dolphin does is create the directories - ALL the directories. Now, directories don't take up any space, so as you are writing to your new 40 TB Unraid Media share, there is always sufficient space, so all directories get written to disk1. The next step is that Krusader/Dolphin begin to copy files into the folders it already created. But because of how Unraid works, especially with a split level of 1, it will not split these folders across other drives, and instead will attempt to keep the folders whole, on disk1. So now you are trying to copy 35 TB of data to an 8 TB drive. Not gonna work.


So writing straight to a disk bypasses any share logic, and is recommended for a migration like this.

Next, the thing to remember when writing to a disk is that, even if you are writing to a share folder on that disk, the share settings DO NOT APPLY. It's the same as on a Windows machine, if you are trying to copy 35 TB of data to an 8 TB drive, it's going to run out of space. The share settings will not magically kick in and stop the copy before space runs out.

Which means that to successfully execute this copy, you will have to manually select directories to copy to the new Unraid disks, making sure not to copy too much data to any one disk.


One last thought on this, if you really are looking for a more automatic way of doing this. The Unraid Mover tool, which copies data to/from the cache drive and the array, does not have the Krusader/Dolphin problem of creating all the directories first and then copying files. Instead, it copies one directory at a time. I honestly don't know exactly how it does this (my guess is the Linux RSync command), but the point here is that there is a way to automate this copy if you use the right tools. In this case, you would want to use the 'Unassigned Devices' plugin, which would allow you to mount your old NAS as a local disk, and then use the correct Linux commands to copy from the mounted network share to the local Unraid share - and yes with this approach you would want to copy to the share and not the disk, so the share can manage where things go. Since I haven't looked at Mover, I don't know how it accomplishes this, but it might be internal logic in the Mover program to only process one directory at a time, and not actually a Linux command that is doing this magic.

With this last migration approach, you do NOT want to use a cache drive. It will get in the way. So make sure that you either don't have a cache drive installed, or it is disabled in the Share settings. Cache drives don't apply to copying straight to disk, though, so you don't have to worry about it in that approach.

[Pauven]

A google searched found the Unraid Mover script on GitHub:

https://gist.github.com/fabioyamate/4087999

I reviewed the script, and there is a bit of programming magic. The actual move occurs near the end, in this line:

find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
\( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete \) -o \( -type f -exec rm -f /mnt/user0/{} \; \)

The magic is that the results of the "find" command are passed, one by one, to the "rsync" command. This is how it is processing one directory at a time.

It should be possible to use this as a baseline for a command to copy from a "Remote SMB/NFS Share" (mounted via the Unassigned Devices plugin). Keep in mind that this command above performs a "move", not a "copy", and you would want to do a copy not a move.

[Jamie]

What Paul stated about why Krusader and Dolphin doesn't work well with a mamoth file migration is correct. When you are copying files from another device to a share, Krusader and dolphin creates all the folders onto the share all at once including sub folders. They then populate the data into each folder on the share and you run out of disc space because the share is not triggered to do the leveling and minimum drive space logic.

You can use Dolphin or Krusader on a share if the size of all the combined data you are moving is smaller then the smallest drive.

Paul probably described the process better than I. We had this same disussion in the thread.

It has nothing to do with file size or data type. I've migrated extremely large ISO's along with small xml's and jpgs.

I hit the wall several times until Paul stated that I needed to do the copying sequentially so at his suggestion, I used the PC with mapped drives and then just told windows explorer to copy the files from the Drobo to the unraid share. I chose this way because I am more acquainted with Windows and I am not well acquinted with linux. A fast migration was no longer a goal as long as I got all the files migrated properly. Also if the process should stop somewhere I wanted an easy way to start over from the file and folder where the process broke.

[Litlgi74]

.You can use Dolphin or Krusader on a share if the size of all the combined data you are moving is smaller then the smallest drive.

Can you elaborate on this fact further?... The majority of my FlexRAID drives are 2TBs. My new Unraid drives are all 3TB except for one 4TB.

I may have said earlier... But I was somewhat successful in transferring my data with Krusader from the old drives to the new while using Highwater and top level split. Each disk would fill up to the 60% mark before moving on to the next disk in the array... All was going fine for about six 3TB disks untill I added several more blank disk to the array... The the first drive started filling to completion and the copy failed.

I really don't have the time to sit and calculate which files are going to go where and on which drive. If adding a cache drive to the system will automatically populate the drives according to the share settings... I'm going to go that route.... But will the cache drive start moving data to the array as soon as it reaches it's capacity?

Thanks Again
John

[Jamie]

What is the total amount of disk space that you are moving at one time? If all the drives are 2TB each and your trying to move more than 2 TB in one copy then your going to run out of disc space. Krusader and Dolphin just doesn't work the right way to get the share to kick over to the next drive. You need something that will copy each file and folder sequentially as though you were ripping a DVD, Bluray, or 4k bluray to the unraid share.

The only way you can copy the data with Krusader, or Dolphin, is to calaculate the amount of data you are moving and make sure it is less then the smallest drive. So don't use a share method. Copy to disc.

You need to move away from Krusader and Dolphin and use a better tool like the script option that Paul suggested. Believe me, I was pounding my head against the wall until my my research told me that Krusader, or Dolphin, were not the right tools to use in this type of migration. You will never get the share to kick over to the next drive using these tools unless your data is less than 10 TB in size and you have 10 TB drives in your unraid server.

Jamie