What is Unraid and how to build an Unraid media server

[Pauven]

At some point, you may want to try adding a drive without doing a preclear first, just for perspective. Unraid will see that the drive needs clearing, and will begin that process, which will take about 6.5 hours for a 3TB drive, or 17 hours for an 8TB drive (just for examples). During this time, your array will be in a reduced state of performance. The clear writes zeroes to the drive, but doesn't test or verify anything. After the clear is done, you'll still have to format it. It's incredibly inconvenient.

The preclear allows you to clear a drive before adding it to the array, so array performance isn't affected, plus you can do extra passes and validate the results and monitor drive health. Instead of it taking 17 hours to add an 8TB drive, it's added in just a couple minutes. The format process is the same, and also just a couple minutes.

To me, the preclear is so essential, it really should be part of unraid, and not a 3rd party plugin, but at least it exists.

[Jamie]

Hi Paul,

Just an FYI. I wasn't questioning the value of the preclear. Everything I have read says that it is the best way to go. Especially testing the drive before putting the drive into the array. I was just concerned that if I formatted the drive after the preclear, that the format would erase the drive signature that the preclear created leaving me to have to clear the drive again. I guess from my research the format will not erase the preclear signature.

Jamie

[Jamie]

Hi Paul,

I successfully pre-cleared my first two drives and added them to my array. Now that I have gone through the process of pre-clearing two drives, assigning the two drives and adding them to the array, I see that I was unnecessarily concerned about the format. My confusion probably confused you on why I was so concerned about formatting the drive before or after the pre-clear.

While I was in the middle of the pre-clear cycle, I noticed the format label beside the unassigned disc names and I was worried that I had missed a step of formatting the drive before pre-clearing. Hence the format questions and confusion.

My drives passed the pre-clear, but I could not figure out how to add the drives to the array and format them. I stumbled on this video and it cleared up all my questions. The guy in the video probably stated some things in the pre-clear portion of the video which you probably do not agree with, just know that I followed your preclear suggestions. During my research on the subject most people agree with you regarding which script to use during the preclear.

Here's the video.
https://www.youtube.com/watch?v=byv4kP4UkE0

[Pauven]

I think I understand now. You'll have to forgive me, as a long-time Unraid user from the days before plugins, I subconsciously block out all of the non-core functionality when thinking about Unraid. I realize now that you must have the Unassigned Devices plugin installed, which gives you extra options for working with drives outside your array, including formatting drives. With core Unraid, you never even see the format prompt until after you've added a drive to your array, and that would be after you've pre-cleared a drive.

Unassigned Devices is a great plugin, I use it too, but I can see how it would be easy to get confused by these options. UD allows you to format a drive to use it outside the array (something users had been asking for a long time), because stock Unraid only let's you use drive inside the array.

I've never seen any of Mike Fury Tech's videos. Seems very helpful. I can't be upset that he recommends using gfjardim's script - it's better than nothing and better than stock Unraid. I think it is good that their are multiple scripts that provide choice to the user, so you can pick what you think is best.

Recently I precleared two 8TB drives at the same time. The first one I used Joe L.'s script. On the second one, I forgot to set that parameter, and it defaulted back to gfjardim's script (really wish that was a sticky option). This gave me an opportunity to compare them both side by side. I still find Joe L.'s script to be superior, though it seems much of my performance concerns regarding gfjardim's script (from years ago now) are no longer an issue, as least not on my current server with tons of memory.

So definitely double-check the pre-clear script each time you run a pre-clear if you don't want to use gfjardim's.

Congrats on finally getting your array up and going!!!

[Pauven]

Hah! I just got to the end of Mike's video where he starts discussing errors. He's completely wrong in his interpretation.

He points to a "Runtime_Bad_Block" error as something that occurred on the first pass, showing that an error was generated DURING the pre-clear. He is wrong.

The "INITIAL" is a snapshot of the SMART values from BEFORE the pre-clear, and the "CYCLE 1" are the new values from AFTER the first pass. Runtime_Bad_Block was already a 1 before he even began pre-clearing, and it didn't change. In the video, you can see that the drive power on hours is over 31-thousand hours, and in all that time there's only ever been 1 bad block. This is actually a healthy drive.

What you want to look for is any changes from the INITIAL to the later passes. If you see bad blocks, errors, or uncorrectable counts increasing during a pre-clear, then that is a sign of a bad drive. But having values stay the same as the initial values is quite okay, as it shows the drive is not getting worse, regardless of what those initial values were.

That's not to say to completely disregard the initial values. If you had thousands of errors in the INITIAL state, that would still be reason for concern, even if they didn't increase during the pre-clear.

[Jamie]

Thanks Paul for all your help and advice that got me this far. Without your help this would have been an insurmountable task.

Yes, I have unassigned devices installed. I installed that in preparation of the drobo migration.

I just installed binhex-Krusader and setup my host paths. This video is a big help.

https://www.youtube.com/watch?v=I0XCFPAsWZE

I will have to go through it a few more times and play around with all the file tasks to get the hang of things.

I am just about ready to migrate my first drobo to the unraid server. I will have to wait until Monday or Tuesday, to start that process. I have to wait until then to do it because of our heat wave that we are experiencing and the possibility of a power outage. I need three days of nice weather so that I can get through the first Drobo box migration without a shutdown.

Things are looking good

Jamie

[Jamie]

I started the migration of my first Drobo to unraid about 10 hours ago and Krusader has processed 1.7 TB of data. The first Drobo has about 10 TB of data.

I am really liking Krusader. You can create a smb "Network Drive" inside of Krusader and compare and sync two folders through it's sync function under tools. I compared my unraid mymovies share with the drobo's mymovies folder and told Krusader to sync the Drobo's movie files to the unraid share.

I think this is great because if the sync is interrupted by a power outage I will just have to go back into Krusader and do another compare and sync which should just continue on where it left off.

Thank you for bringing this excellent piece of software to my attention!!!!!.

Jamie

[Pauven]

That sounds awesome. I've never used Krusader myself, but it sounded like a good solution. That's one of the great things about Unraid, the plugin system has solutions for nearly everything these days, plus you can do Dockers and full blown VM's. Seems like there is literally nothing Unraid can't do.

[Jamie]

Hi Paul,

Krusader did not work for me.

I ran out of disk space even though I was copying the files from my Drobo to my unraid share which had more then enough disc space. It turns out that Krusader and Dolphin both populate the top level folders of drive 1 immediately and then populate the the files to each folder. Because the top level folders are all populated the share allocation never kicks over to the second drive when space runs low. If I told my share to split on the top 2 folders instead of only the top level folder Krusader and dolphin would populate the share properly and the share allocation would work and kick over to drive 2 when space runs low. I have the feeling that if I went down two levels, I would still have issues because many of my non boxset and TV show discs only have one folder.

I think I only have two options here.

Try the Unbalance plugin or copy from my Drobo to unraid using one of my NUCs windows PCs to initiate and play "traffic cop" between the Drobo and unraid machines. At least with windows I know that the top level folders will all be populated synchronously instead of all at once.

What do you think? Any ideas?

Yesterday, I got two HGST HE 8 drives which I want to run a pre-clear on during the next couple of days. I will not put them in the array until the migration is complete but I want to make sure that the drives are good, During this time that the pre-clear is taking place, I will do some research.

Any suggestions will be helpful.

Jamie

[Pauven]

Good job deducing the problem. I remember many years ago having a similar struggle, and Unraid does require a change to your mindset to deal with these types of issues. You are absolutely correct in your understanding, but you might be misinterpreting the folder levels.

First, I think it would be helpful for you to share the basic folder structure you are copying. A top directory split setting may not be appropriate for your incoming folder structure. For example, in the structure \\Tower\DVDs\Backdraft, the "Automatically split only the top ONE directory level" treats the Share "DVDs" as the top level, not "Backdraft", which is actually level 2. Another example, \\Tower\TV_Series\Elementary\Season 1\Disc 1\, "TV_Series" is level one, "Elementary" is level two, "Season 1" is level three, and "Disc 1" is level four. If you aren't factoring that the Share itself is the top directory, you may be setting your split level to a lower value than is appropriate.

So while my basic folder structure is \\Tower\Blu-rays\Thor, if your basic folder structure is \\Tower\Movies\Blu-rays\Marvel\Thor, you would need a completely different split level. I don't want to make assumptions on what you're doing, so share your folder structure.

Second, I should point out that another reason I split my discs into 4 different shares (4K, Blu-rays, DVDs, TV_Series) is that I can have different share settings for each. For TV_Series, I split at the top two levels, for the others the top level:

image.png (28.57 KiB) Viewed 12780 times

The reason I split TV_Series at the top two directory levels is because I group all the seasons into a parent folder named for the show, which allows me to keep seasons together, but the series itself may be spread across multiple drives. It you weren't planning on keeping your TV Series on a separate share, you probably want to reconsider for this reason alone. The big problem with TV Series is that I'm often adding seasons for on-going shows (i.e. Veep, Homeland), so it might be years apart between adding seasons, and it's unlikely there is still room on the drive where the previous season was saved. While I want all the discs for that season to remain together, I fully expect that different seasons will end up on different drives. A split level of two, for my particular folder structure, works perfectly.

All my movies split at the top level directory, which basically means that once the movie folder is created (i.e. \\Tower\Blu-rays\Shawshank Redemption\, then everything written to that folder will always stay together on the same drive - which works great for the metadata files that get written sometimes months/years after the ISO. Note that this isn't an Unraid requirement - Shares transparently merge the contents from multiple drives into a single Share directory - but rather my own OCD that drives me to use this configuration.

Third, you can change the split level at any time, it doesn't have to remain static. It doesn't affect what was already copied, only newly written files. For the type of copy you are doing, from an external source much larger than your drives into a Share that spans multiple drives, you may need to crank up the split level to three, four, or five, or even that catch-all option of "Automatically split any directory as required". Once your copy is done, you can go back to your target split level for future additions.

It might be that you need to temporarily use a split level of three for your TV Series, and a split level of two for your movies, and that you would need to copy them separately after first setting the split level in prep for each copy session.