CMC Forums

Hi Paul,

In what way do you use the cache drives? I think you said that you rip your titles to cache and then have a nightly process to copy the newly ripped titles to the other unraid discs. Did I get that wrong? How did you set this up and what type of gotchas might there be involved with setting this up and implementing it? Also you mentioned that you have to defrag the drives nightly. Do you have some script that you run?

Jamie

Using a Cache drive sounds complex, which is what kept me away from using it for many years. In reality, it is so simple to use, and Unraid does all the hard work for you.

Essentially, a Cache drive should be an SSD (or NVMe if you prefer), so that you get the benefits of fast speed and without spinning up a HDD.

Cache drives are used in relation to a Share. For each Share, you can specify one of the following options for "Use cache disk":

1. NO - Never use the Cache
2. YES - Temporarily use the Cache, space permitting
3. ONLY - Only use the Cache (data stays on Cache, never copies to array)
4. PREFER - Permanenent use the Cache, space permitting

If you set it to NO, then any writes to the Share (i.e. DVDs) will always go straight to the HDD array.

If you set it to YES, then any writes to the Share will automatically be written to the Cache disk if there is enough room (it adheres to the same min free space parameter defined for that share, i.e. 100GB that I set for my DVDs share is validated against the Cache disk, and if insufficient room then the writes are direct to the HDD array). The next time the MOVER is run (typically runs automatically each night), these files will be copied from the Cache disk to an array HDD.

If you set it to ONLY, then all data for that Share will be written to the Cache disk, even if there isn't sufficient space. A good example of how you could use this is for your Music Share. If it is small enough to fit on your Cache, you can set it to ONLY, and it will then reside on the SSD drive. That way you can play music from any PC in the house, getting really rapid data access from the SSD, and never spin up any HDD in your array.

If you set it to PREFER, then you get the same behavior as ONLY, unless there is insufficient space on the Cache disk, then new writes to that Share will go to the HDD array.

One of the nice things about the Cache disk and the MOVER is that you can change the setting above for a Share, and the MOVER will automatically move the data where you want it for you the next time it runs. For example, let's say you had your Music Share set to NO for the Cache disk, and the data was sitting on a HDD in your array. You then change it from NO to ONLY, and the next time the MOVER runs (overnight, or on demand if you don't want to wait) your entire Music Share will be relocated from your HDD array to your Cache disk.

Assigning a Cache disk is super simple, same as you assign any other disk to your array. On the array setup screen where you assign your Parity and Data disks for your array, you also assign your Cache disk.

Using a Cache disk is transparent - you really won't even know it is happening unless you look for it. For example, let's say I'm copying a new Blu-ray ISO to my array for the movie Batman Begins. I go to my Share named Blu-Rays and create a new folder named Batman Begins. Since my server name is Tower, the full UNC path looks like this: \\Tower\Blu-Rays\Batman Begins\. I then place my Batman.iso file in that folder.

Behind the scenes, Unraid created \\Tower\cache\Blu-Rays\Batman Begins\Batman.iso (so it's on the Cache disk). Right after copying the ISO out there, I can't wait to watch it, so I go sync with CMC and play the ISO. It exists at \\Tower\Blu-Rays\Batman Begins\Batman.iso.

Then overnight the MOVER kicks in, and since I have the Blu-Rays Cache disk set to YES, it knows to move the new directory into my protected HDD array. It checks my disks, and finds that disk14 has enough free space (more than 180 GB per my Blu-Rays min free space setting), and moves it from \\Tower\cache\Blu-Rays\Batman Begins\Batman.iso to \\Tower\disk14\Blu-Rays\Batman Begins\Batman.iso.

Since Batman Begins is so awesome, I decide to watch it again the next day. I launch it in CMC and play the ISO that is still at \\Tower\Blu-Rays\Batman Begins\Batman.iso.

That's how Shares work, they are a virtual directory that spans multiple disks (including the Cache disk) and presents them as a single directory. Behinds the scenes, Unraid can choose on which disks to store the data (Cache, disk1, disk14, whatever), and makes those decisions based upon the parameters you configured for that Share. But when you use that Share, you don't have to know any of that to get at your data, you simply access your Share.

Now there is one problem with Cache disks - they are OUTSIDE your protected array. So for my DVDs and Blu-Rays Shares that I have set to YES, for the few hours they sit on the Cache disk they are unprotected. If the Cache disk fails, I loose that data. Then after the MOVER relocates them to the HDD array, they will be protected.

For my Music Share, if I set it to ONLY use the Cache disk, then that data is never protected in the array (at least not automatically). I can still make an extra copy of my Music and place it in another Share (i.e. I keep an extra copy in my Share named Backups, which is set to NO for using the Cache).

For years people complained that the Cache drive was too vulnerable, and they wanted to have it protected too. So recently LimeTech added a Cache Pool option, which allows you to use multiple Cache disks and to pool them together for protection (or striped for more performance but even less protection). This is one of the main reasons I specified that particular motherboard in the build list, as with 10 SATA ports, you have 2 left over for hooking up 2 SSDs and creating a RAID 1 mirrored Cache pool.

Spaceinvader One also has a nice video that covers most of what I discussed above, and also explains a few other nice details:

https://www.youtube.com/watch?v=ij8AOEF1pTU

Paul

Paul, This is great information that you are providing! Thank you.

Jamie

The main reason people use Cache disks is for speed (after all, Cache is in the name, right).

When I write to my array, writes that go to the Cache typically go at full speed, only limited by my network speed (gigabit, or 100-120 MB/s) and by my PC that is doing the writing (copying from a SSD will get me that full speed, but a direct rip from a DVD drive will be much slower).

But when I write to my array without a Cache, the speed get's limited to how fast data can be written to the protected array. This is typically about half the speed of a HDD. For example, let's say I could normally write to a HDD at 120 MB/s, due to the Parity protection overhead my actual write speeds will be less than half (< 60 MB/s). Obviously the speed of your array is determined by your particular server hardware and drives, but this generalization holds true.

So by writing first to the Cache drive, your writes complete faster, plus you avoid to momentary HDD spin-up delay as the Cache drive is ready to go.

But there is another HUGE benefit that I gain by using a Cache drive.

I have 4 Blu-ray drives, and I can rip from all four simultaneously. But if this data is written to the same place at the same time, it becomes fragmented. The last thing I want is for my ISO files to become fragmented, as that can affect playback performance. Especially ISO's that are fragmented on a spinning HDD.

By having my ISO rips first hit the Cache drive, they are fragmented only on the Cache drive, and SSD's are fast enough that you can play back a fragmented file without any issues - that's why we don't defragment SSD's, they don't need it.

Then overnight when the MOVER relocates the files from the Cache drive to a protected array HDD, it does so one file at a time, effectively defragmenting the files. So my ISO's end up perfectly defragmented inside my protected HDD array.

Nice, eh?

The only challenge I have had with this approach is that the Cache drive (when configured as YES for the Share), adheres to the same min free space requirements as the other disks in that Share (i.e. 180 GB min free space for my Blu-Rays Share). While I have a 1TB Cache drive, I use it for a lot of data, including several Windows Server VM images. So right now my 1TB Cache disk is using 618GB, and only has 382 GB free for caching my new ISO rips.

If I'm busily ripping lots of Blu-ray ISOs, 4 at a time, for a few hours, I can easily drop below the min free space on the Cache disk after only 5 or 6 ISO rips. And because Unraid so transparently implements the caching service, it will automatically start writing to the protected HDD array once my Cache disk drops below the min free space for that Share. If that happens when I'm ripping multiple ISOs at the same time, I will then be writing fragmented ISO images to my array.

Since I so heavily use my Cache disk space for things like my gigantic VM images, I really need to upgrade my cache to 2TB. I'd also like to implement a RAID 1 Cache Pool when I do it. And I like to use NVMe drives for my Cache disks (faster VM startup). Buying 2 x 2TB NVMe drives is going to be super expensive, which is why I'm only using a single 1TB cache right now.

But for your use (no VMs, light Docker use) I think you will find a 1TB Cache to be sufficient, and the regular SSD's will be more than fast enough, maxing out your gigabit network speeds. And if you want your cache to be protected, a couple 1TB SSDs in a RAID 1 Cache Pool should be fairly affordable and a perfect fit for your motherboard.

But if you think you might be heavily using your Cache disk too, you might want to consider a larger size. Assuming you use 100 GB of your cache for other things (Music, Dockers), you would have about 900 GB free for caching ISO rips. That would be enough for at least 20 Blu-Ray rips (more if they are small ISOs). If you had 4 drives going at the same time, you can rip about 4 per hour, and you could do 20 in just 5 hours. At that point, you can either take a break and let the MOVER do it's job, or you will need a larger Cache disk.

Only you know how heavily you will use the Cache disk, so think carefully about it and determine what size Cache disk will best suite you.

Paul

The copying of files brings up another question. I had dreams of just copying the complete mymovies folder from each separate drobo all at one time.

I was going to limit the copying to one drobo at a time but all the folders on that drobo.

1. Should I turn off the cache while I'm doing this process?

2. Should I do each drobo copy in batches?

A limited number of folders at a time? I have a feeling that this process may saturate my network. So maybe I should do the copying during the night. What do you think? I do some streaming during the evening and my sister does a lot of streaming during the afternoon and evening.

3. Even though the routers, switches, and NICs can do a gigabit for each second, I have learned that your mileage may vary due to various factors.

How many tb, or gb do you think that I can move in one 10 hour period? Do you know of any calculators, methods that I can use to figure this out?

The first 2 drobos have 12 tb of data each.

Jamie

The cache drive, even a large 2TB drive, would be too small for use while migrating data from your Drobos, so yes you should disable the cache drive while you're migrating your data.

The parity may also slow down the copies to 50% speed, so a lot of users (myself included) wait to turn on parity until after the copies are done. You can always do a quick test, copy an ISO with parity enabled and without, and see if there is a speed difference. There's no harm in waiting to enable parity, as you still have your original data on the Drobo after you copy it - though definitely enable Parity before you decommission (pull drives) from any Drobo. It'll probably take about 18 hours to build the parity data once you turn it on (regardless of how much data you've copied).

If I had to guess, I'd imagine the slowest component will be your Drobos, not your network or your Unraid server, so the Drobos will most likely dictate how long the copy will take. You could try copying some data down from your Drobos to figure out how long it takes, and multiply it out to estimate full Drobo copy duration.

As a test, I just copied a 3.4GB MP4 directly to my array, bypassing the cache. The was a 15 second delay during which the copy was 'Calculating', then it took off at 120 MB/s and ran strong for a while, then dropped all the way down to 20 MB/s, then climbed back up into the 70-90 MB/s range for the rest of the copy.


Total duration was around 60-70 seconds.

Including that original 15 second delay, average copy speed was about 50-55 MB/s. Without that delay, the sustained throughput was 62-76 MB/s. This copy was to one of my older WD Red 3TB 5400 RPM drives. Copying to a 7200 RPM 8TB drive would be quicker as those drives are a lot faster.

Assuming your Drobo can sustain 50 MB/s, then copying 12 TB of data would take 240,000 seconds, or 4,000 minutes, or 67 hours.

This is one of the main reasons why I recommend to delay enabling parity, as instead of 50-75 MB/s, I would be maximizing my gigabit ethernet at 100-120 MB/s. Assuming your Drobo can go that fast, this would cut your copy time in half, from 67 hours to 33 hours.

One thing you also need to do is figure out how to go straight from your Drobo to the Unraid server. If you initiate the copy from your PC, then technically you are copying Drobo -> PC -> Unraid, and with your PC playing middle man you may hurt your throughput. Though I've never done it myself, I'm sure it is possible to initiate the copy directly from the Unraid server, eliminating the PC in the middle.

You may also need to think about how to validate your copy. I know there are some copy tools that can also validate the copy is good, and while that might double the copy time, it might be good insurance to make sure your data didn't get corrupted during the copy.

I'll poke around and see if there are any good tools to use from Unraid to copy and validate data directly from another NAS.

Paul

Jamie wrote: Sat Mar 02, 2019 3:50 pm How many tb, or gb do you think that I can move in one 10 hour period? Do you know of any calculators, methods that I can use to figure this out?

The absolute maximum that you will get on a gigabit network is about 120 MB/s, so in a 10 hour period you could copy up to 4.3 TB. These are network limitations.

Having jumbo frames enabled may help, though to be honest I've never used jumbo frames, as it seems hardware support (NICs/Switches) is inconsistent.

To calculate I just use Excel and plug in the numbers. Gigabit = 1,000,000 bits/s, divide by 8 = 125,000 Bytes/s aka 125 MB/s, but minus overhead you'll max out at closer to 120MB/s best case. Then I just multiplied by 60 to get 7.2 GB/min, 60 again to get 432 GB/hr, and by 10 to get 4.3 TB/10 hours.

But like I said, most likely your Drobos are slower, so just copy a big file off of one and time how long it takes, then we can calculate their throughput.

So I already figured out how to connect the Drobo directly to the Unraid server - surprisingly easy!

There's a plugin called "Unassigned Devices", and I already had it installed. The only reason I've ever used it is so I can see any extra HDD's I have in the server that I did not include in my array. Here's a screenshot, and in the middle you will see Unassigned Devices and I have one drive, Device sdg which is my WD Red 8TB drive that I have precleared (that post is coming...).

Right below that line you see SMB Shares | NFS Shares | ISO File Shares, and the button I highlighted for "ADD REMOTE SMB/NFS SHARE". That buttons been there for a long time, I've just never paid any attention to it. It's part of the Unassigned Devices plugin. Apparently you can add a direct connection to either an SMB or an NFS network share. Here's a screenshot showing the parameters you can provide for SMB:



Since I don't have a network enabled Drobo, I'm stuck at this point. After you connect it, presumable you should be able to directly copy from your Drobo to your Unraid server. Not sure "how" you would do that copy just yet, still looking at options for initiating and controlling the copy, though the leading candidate is the Krusader Docker. Command line is always an option, though I find that a pain for copies. Midnight Commander (MC at the command line) is included with Unraid, and might prove to be a good solution too.

After further review, it looks like the binhex-krusader Docker plugin is perhaps the best option for controlling the copies. Here's their description:

Krusader is an advanced orthodox file manager for KDE and other desktops in the Unix world. It is similar to the console-based GNU Midnight Commander, GNOME Commander for the GNOME desktop environment, or Total Commander for Windows, all of which can trace their paradigmatic features to the original Norton Commander for DOS. It supports extensive archive handling, mounted filesystem support, FTP, advanced search, viewer/editor, directory synchronisation, file content comparisons, batch renaming, etc.

Essentially, you would first install the Unassigned Devices plugin, then add the remote SMB share to a Drobo, then install and log into the Krusader Docker, which will give you a drag and drop style GUI to move files from point A to point B. Your files would go straight from the Drobo to the Unraid server.

You might be able to optimize your network path between them so that the rest of your home network isn't saturated while these two devices talk to each other. As long as you can have a single switch with both the Drobo and the Unraid server connected directly into it, their traffic shouldn't affect the rest of your network.

Here's the webpage on Krusader so you can read more about it: https://krusader.org/

It seems like it would have some options to do file comparisons to ensure you get a good copy.

Paul

Thanks for your research Paul. It's greatly appreciated.

Fortunately I have a netgear GB router sitting around doing nothing so I could take each drobo and hook it up to that router in the laundry room to the unraid server. So I could just dedicate that router to this task without affecting the rest of the network. The unraid server would have it's own dedicated LAN.

I can also copy all the folders in one try for each drobo, if you think that it would serve me well to do it all in one fell swoop.

My concern would be the lack of parity drives, but I think that I am willing to take the risk just to get the drobos offline as quick as possible. Not having the cache and parity drives would mean that I can invest my time and money in replacing the drobos until all are decommissioned.

You can run an unraid server without a parity drive in a drive bay can't you?

Jamie