Your first HDDs and Migrating Data to Unraid
Once you get your system built, I'm sure you will be eager to start copying your data over to it. But before you jump in, here's some things to think about that should make the transition easier.
User Shares
Before you copy data over to Unraid, you will want to define one or more User Shares. A share is basically a virtual directory that can span one, multiple or even all drives. I have a lot of shares, and here's an example of some of my shares:
- Backups
- Blu-Rays
- DVDs
- TV_Series
- Music
Obviously, you can set up your shares any way you want. You might want to do a single 'Movies' share instead of breaking out Blu-rays and DVD's as I have. The important thing is that you should copy data into shares (and not directly to drives) and let Unraid manage where the data goes. You can always move data around later, but it's easier to get this right the first time. I use dedicates shares for DVDs and Blu-rays as I set different minimum free space limits for each individually, which helps me automatically use more of my available disc space without issues.
There are also quite a few parameters you can define for shares. Things like assigning or excluding drives from a share, minimum free-space, and the fill-up method.
For my Blu-Rays share, I include All disks, but exclude Disk 20, which I have reserved for my PC backups. I have the minimum free-space set to 180GB, which ensures that any Blu-ray discs copied to it will have enough room before the copy begins (180 GB is approximately 4 full-size BR ISO files) . The allocation method I use is "Fill-Up", which sequentially fills up each disk, one by one, until it hits the 180GB minimum free-space limit I set, then it moves to the next disk. I have the split level set to "Automatically split only the top level directory as required", which is the default and works well, though you may want to read up on this to determine if you like this behavior. I also have the "Use cache disk" set to yes, so that means any new data I copy to this share will copy first to the fast SSD cache disk, then overnight move into onto the protected HDD array.
- image.png (498.5 KiB) Viewed 17336 times
For comparison, I have my DVDs share set with nearly identical parameters, though I have the minimum free space set to 100GB. Yes, this is much larger than a single DVD ISO, though from experience I have found that there are scenarios for which having the min free-space set higher works better. For example, if I copy an entire TV show season's worth of DVD's all at one, that might be 5 or 6 DVD's, or 40+ GB of data. Because they are for a single show, they would all reside in the same directory, so by setting the minimum free space to 100GB I ensure that the entire season gets copied together to the same physical drive (a preference of mine), while also preventing accidentally running out of space on a drive when copying a lot of data to a single directory.
Something else to know about shares and the split level, once a directory gets written to a drive, any further contributions to that directory get written to the same drive. For example, let's say you have \Blu-Rays\Elementary\Season 1 that gets created on drive 5. Perhaps when you create it you only add the very first Blu-ray disc, then come back a few weeks later and finish copying those ISOs. Unraid will naturally want to add all the additional discs to the same directory (Season 1) on the same drive (drive 5), and if you didn't keep the minimum free-space high enough, then your free space buffer may not be enough for these additional ISO rips.
Years ago I kept my minimum free space values aggressively low, and would occasionally have errors when drive space ran out. That's why I keep my values quite a bit higher now, and I haven't had problems in years. On occasion I will manually copy files directly to a disc to fill up more free space, bypassing the configured User Share limits, but it is best to keep your shares with higher free space minimums as those are automatic behind the scenes copies you don't want to fail. When a Share's minimum free space limit is hit, any NEW directories automatically get written to the next drive with sufficient free-space, but any further contributions to the existing directory get written to the same drive.
The other thing to know about the minimum free space parameter is that this is the minimum free-space pre-copying, not post-copying. Since my minimum free space for Blu-rays is set to 180GB, let's say I have a drive with 200GB free so it meets this criteria. I then copy a 45GB BR ISO to the share, and it goes to that drive, decreasing the available free space to 155GB. Because it is now below 180GB, it is excluded from any further Blu-ray contributions (at least for new titles/directories), but because it is above 100GB it is still good for the limit I set for DVD ISOs. So writing to a share can and will take space below the minimum free-space limit you set, as this parameter controls how much space must be available before a copy is performed to that drive, not after the copy.
When to enable Parity
Obviously one of the main reasons to use NAS devices like Drobos and Unraid is for data protection from drive failures, so you're probably expecting to enable Parity protection before copying any data into the array.
There's actually a good reason to wait on Parity. When you copy data to a HDD, you can copy at the full speed of that drive. But once you apply Parity, you will copy at less than 1/2 the full speed of the slowest drive. This is because both the data drive and the parity drive have to first be read, then new parity calculated while the new data is written, and parity data written to the Parity drive too.
As long as your data still exists on your Drobos, I would recommend not enabling Parity during the initial copies while you're first populating the array. But before you start to decommission your Drobos, and pull their drives for re-use in your Unraid server, you will want to enable Parity.
What size Parity Drives
The requirement for Unraid is that your parity drives have to be equal to the largest data drives in the system. If your largest data drive is a 4TB HDD, then your parity drives should also be at least 4TB (if they were smaller, then Unraid will artificially restrict your data drives to the size of your parity drives).
You may be thinking that in the future you want to use larger data drives, so go ahead and install larger parity drives now. And you can do this, as I have. For years I used 3TB drives only for data and parity, then I had a couple spare 4TB drives from a decommissioned PC I wanted to add. Before I could use them, I needed to upsize my parity drive, and I decided that 8TB would be my new drive size choice. So I now have two 8TB parity drives, and a couple 4TB data drives, and about 16 3TB data drives.
Perfect, right? Well, mostly, but there is one caveat that you need to know and understand about Unraid - the Parity Check and drive Rebuilds.
It is a best practice to do a monthly Parity Check (you can do it more/less often if you desire), and the Parity Check spins up all drives, and reads every last bit on every last drive and makes sure your parity data is valid (and can correct it if it is wrong). The same will happen with a drive rebuild (i.e. upgrading a drive to a larger size, or replacing a failed drive).
When I had only 3TB drives, my monthly Parity Check took about 7.5 hours. Then, when I added the two 8TB parity disks to my 3TB data drives, my monthly parity check increased to about 15 hours (because it had to read every last bit of 8TB now, not just 3TB).
Then when I added my two 4TB data drives, my Parity Check increased to 18.5 hours. I'll give you a second to think about that... do you know why?
It's because HDD's read faster at the beginning of the platter (outside edge), and slower at the end (inside edge). Your parity check might start off at 120 MB/s, then slow down to 60 MB/s by the end of the disc. But that slowdown is now happening muliple times, at the end of the 3TB drives, at the end of the 4TB drives, and again at the end of the 8TB drives. And here's the real gotcha - the parity check will always proceed at the speed of the currently slowest drive. So at the end of 3TB, all of your drives might be reading at 60 MB/s, then as soon as the 3TB drives are done the speed might jump up to 100 MB/s, slowing down again to 70 MB/s at the end of the 4TB drives, then jumping up to 150MB/s once those are done, and gradually slowing down to 90 MB/s at the end of the 8TB drives.
If I only had 8TB drives, then they would all slow down at the same rate, and my parity check would proceed at the speed of my 8TB drives - my entire parity check might only take 12 hours instead of over 18.
So while the ability to mix and match drives of any size is a huge benefit to the Unraid solution, if you mix a lot of different drive sizes then your monthly parity check could take longer. For some users, their parity check takes 1-2 days. And during the course of the parity check, all your drives are spinning (well, at least at the beginning), generating noise and heat and vibration - this is wear and tear on your drives.
Your goal should be to minimize how long your parity checks/drive rebuilds take, and the best way to do this is to use drives of all the same size. So you might be using a mix of 3 and 4TB drives today, but as you plan your next size upgrade, I recommend you do as I have, and pick a single size and plan to use that for all drives. I picked 8TB, and will gradually upgrade all my drives to 8TB over the coming 1-2 years. I will end up with a 160TB server, which I think is as far as I want to push this thing, so this will be my final size upgrade.
The nice thing is that by gradually upgrading to 8TB drives, I'm minimizing my expense as these drives become cheaper over time. But during this transition period, I have to accept the longer parity check and drive rebuild times that come with having a mix of 3, 4 and 8TB drives. This would be even worse if I had more drive sizes in my mix.
So long story short, use parity drives that are equal in size to your largest data drives, until such time as you are actively upgrading to a larger size. This will keep your parity checks shorter, minimizing unwanted noise/vibration/heat on your system. And also try to standardize on a single drive size for best performance overall.
What rotational speed for your HDD's
Also, a brief note of drive RPM speeds: In general, Unraid performs at the speed of your slowest drives. So having a few 7200RPM drives mixed in with 5400 RPM drives will not normally give you a speed boost. True, if you are just reading data from a single 7200 RPM drive, that will be a bit faster, but during the parity check or drive rebuilds, it's the slowest drive that sets the pace, not the fastest. And the same on writing data to the array: you could have 7200 RPM parity drives, but if you are writing data to a 5400 RPM drive, then the 7200 RPM drives will operate no faster than the slower 5400 RPM data drives. The might be spinning faster, generating more heat and vibration, but they won't be working faster. So for this reason you should try to have all the same speed drives in your system. Either all 5400 RPM, or all 7200 RPM, as a mixture of speeds will drive up expenses, but not drive up performance.
Minimizing HDD Expense while Migrating from your existing Drobos
It makes sense to me that you will want to pull your drives from your Drobos and re-use them in your Unraid server. But you can't do that until you copy the data off of them first. When you put a drive in your Unraid system, Unraid will prep it by writing all zeros to it and formatting it for use in Unraid - this will destroy any data on that disk.
So you will need some starter discs in your Unraid system so you have a place to copy your Drobo data.
I don't know exactly what size Drobos you have, so I'll just lay out a hypothetical example that you can adjust to your needs.
It makes sense to start with your smallest, 5-bay Drobos first. Let's say you have a 3TB drives installed in it, so you have 12TB of total available storage on that Drobo.
For your Unraid server, you could start off with 4 new 4TB drives. Three of those 4TB drives would be for data, giving you an equivalent of 12TB of space in Unraid. You would copy your data over at full speed, then assign the 4th drive as the 4TB parity drive.
Now that your data is copied and protected, you can then decommission your 1st Drobo and move those 5 3TB drives into your Unraid array. That will add 15TB of space (more than the 12 TB of space the Drobo gave you), so now you can copy a 5-drive Drobo that has 4TB drives in it. This copy from the 2nd Drobo will go slower, since you now have parity enabled on Unraid.
Once the 2nd Drobo is copied, you repeat this process: decommission the Drobo, move the five 4TB drives into Unraid, and you will gain 20TB of space, and you'll be ready to copy your next Drobo. After your 2nd Drobo's discs are added, you will have about 14 drives in your Unraid system, so it's probably a good idea to go ahead an enable the 2nd Parity drive at this time. You can do it earlier - it won't affect performance any more than adding that first parity drive, but since parity drives are not available for data, it might increase your migration costs if you have to buy an extra HDD to enable the 2nd parity earlier.
I think you currently have 31 drives across all your Drobos, so there is no way you'll be able to re-use all those drives in your Unraid server. Since your Unraid server only has 24 bays, and you'll be using 2 drives for Parity and possibly keeping 2 drive bays empty for maintenance tasks, that gives you only 20 drives for data. Assuming an even split of 3TB and 4TB drives, that would give you 70 TB of storage. That may not be enough to copy all your Drobos, so you may need to start off with bigger than 4TB drives for your initial drives. You might need to do 6TB or even 8TB drives as your initial drives in order to have enough total space to complete the Drobo data migration.
Keep in mind that you can upgrade a drive to a larger size at any time. So you can temporarily pop in 3TB drives as you migrate your smaller Drobos, then when a 4TB drive get's free up from decommissioning a larger Drobo, you can replace a 3TB Unraid drive with the 4TB drive, and after the rebuild you will gain that extra 1TB of space.
If you buy 4 4TB drives, and re-use 18 4TB drives from your current Drobos (assuming you have that many), then you will have all 4TB drives (nice) and 80 TB of storage. You wrote in your original post that you currently have 75TB of storage space across all your Drobos, so this would allow for a slight expansion in storage space, and you could migrate off of all your Drobos. When you get to your last 8-drive Drobo (using all 4TB drives, I assume), you will have already filled out your Unraid server with 22 drives, hopefully all 4TB at that point, so you should in theory have sufficient space to copy that last Drobo even without using those drives. It's possible that if you don't have enough 4TB drives in your collection, you might still have some 3TB drives in your Unraid server at this point, so you might be a little short on space (really depends on just how full your Drobos are), so you may have to upgrade a few 3TB drives to 4TB out of pocket instead of re-using Drobo drives to get to the finish line.
Hopefully you can minimize that expense by following the example I laid out above. If you don't think you can get there with purchasing four 4TB drives as your starter drives, you may want to go up a size on those initial drives (perhaps all 6TB drives, or 8TB like I'm doing). But since you went large on the # of drives in your server, I don't really recommend going bigger than 8TB for any drives at this time. Bigger drives are a lot more expensive, and with that many bays you get a better bang for your $ by using more smaller drives.
Hopefully all of the above made sense. I know some of these Unraid concepts are new for a Drobo user, and I might have over-generalized a few areas. Just let me know if you need any clarity on any topics. I can also help you determine what size and how many starter drives you need to get through your full data migration, I would just need more specifics on your existing Drobos (I would need # of drives and drive size for each Drobo, plus how much space used in TB on each Drobo).
Paul