Unraid uses an unusual parity scheme: All the bits from the same position across all drives are added up to calculate a parity value, and that value is written to the same position on the parity disk. If a drive is replaced, that simple math formula can be reversed to rebuild the contents on the replaced drive. With dual parity, a different calculation is performed and written to the 2nd parity disk, so that two drives can be rebuilt at the same time.
Unraid also has a great feature in that you can add additional drives at any time, expanding your storage. When you first add a drive, Unraid does not make it immediately available for use. Instead, it first "Clears" the drive by writing zeros to the entire drive, then writes an Unraid signature to the drive, then finally adds the drive to the available storage. When this last step occurs, because the drive has all zeroes, it does not alter the parity values that were previously calculated, as adding zero to a value does not change a value.
During this drive clearing process, the Unraid system is in a reduced state of performance (Note: it might actually be that Unraid makes the array unusable during the drive zeroing - I've seen that mentioned, but it has been so long since I've allowed Unraid to clear a disk that I truly don't remember). Considering that simply Clearing a disk (writing zeroes) takes around 2 hours per TB, you can see that this is incredibly inconvenient especially when adding larger drives.
Pre-Clearing Drives
The wonderful Unraid user Joe L. realized that he could write a script to do this Clearing routine before you even add the drive to your array. By doing the Clearing outside the array, array performance would not be impacted during the Clearing. This also opened up the possibility of Clearing drives on a different machine, or simply clearing a stack of drives that you then store offline, waiting until you need to use them.
Joe L. also recognized the anecdotal wisdom that HDDs most commonly fail either when brand new (manufacturing defects) or very old (wear and tear), and that he could turn his Pre-Clearing script into a stress test to validate that drives are healthy, and hopefully to cause new drives with manufacturing defects to fail early, before you add them to your array, and when returning them to the seller is likely easier.
Joe L. made his Pre-Clear script configurable to suit the goals of different users: you can do a simple one pass, writing the zeros and the signature the same as Unraid does, or you can go with multiple passes, as many as you desire, to really give a drive a workout. A standard pass includes a pre-read, a write, then a post-read to validate the data (zeros) were written correctly. The script will even give the before and after SMART health stats, so you can see if a drive is tending towards a failure profile.
Typically this default Pre-Clear script's config takes 3x's longer to process than it does to simply read the drive once - i.e. if it would take 17 hours to read a full 8TB drive, then it will take 50+ hours to read + write + read the drive. Adding more passes increases the processing time.
Running a Pre-Clear on all drives, even previously used drives from other devices, has become a best practice for Unraid. Knowing that a drive is healthy before you add it to your array is a huge benefit that, as far as I am aware, is unique to Unraid. Another benefit is that when adding a Pre-Cleared drive to an Unraid array, due to the presence of the Cleared signature on the disk, Unraid adds the drive instantly, without the long period of array downtime experienced when you allow Unraid to do this Clearing.
Unfortunately LimeTech has not seen fit to integrate a Pre-Clear function into Unraid, so you are still forced to use 3rd party solutions.
Is there a Pre-Clear Plugin?
Yes, there is a plugin named "Preclear Disk", by gfjardim. This plugin ships by default with gfjardim's own pre-clearing script, and it needs to be noted that this is NOT the Joe L. script. gfjardim had different goals for his script (quickest possible clearing, less stress on disks), that many users (myself included) found objectionable - we actually want to stress test the disks, not take it easy on them.
I did some testing with an older version of the Preclear Disk plugin back in August of 2016, and I had very poor results with gfjardim's script: https://forums.unraid.net/topic/54648-p ... ent=485163
I discovered some high memory consumption issues with the Preclear Disk's built-in script. Worse, I discovered that the Preclear Disk script wasn't properly flagging two known bad drives as bad. I documented these issues, and the lack of response from the developer left a bad impression on me, so the next time I did a Pre-Clear I went back to Joe L.'s script, which works perfectly.
Unfortunately, Joe L. has expressly denied consent for his script to be built into a plugin, and has also refused to do so himself.
Running Joe L.'s preclear_disk.sh Script
The challenge to running Joe L.'s script is that it is command-line based, and if you want to run more than one at a time then you need to use Screen to initiate multiple console sessions that you can toggle amongst. Actually, the Screen console session manager is smart to use with even just a single console session, so that if you accidentally get disconnected from the remote console session, you can get back to it by resuming your console session via Screen.
You can install Screen via the Nerd Pack plugin (which is an awesome plugin for installing tons of Linux utils). And you can read more about Joe L.'s preclear_disk.sh script here: https://forums.unraid.net/topic/2732-pr ... quick-add/
It does appear that Joe L. has stopped support of this script, with his last update being Apr 05, 2014.
But I really want a Plugin!
Me too. Luckily, gfjardim has made his Preclear Disk plugin compatible with Joe L.'s preclear_disk.sh script. You'll need to use a patched version of Joe L'.s script and copy it to the right location, but once you've done that then you can use the Preclear Disk plugin as a front-end for Joe L.'s most excellent preclear_disk.sh script.
The patched version of the script simply changes a few things to make it compatible with the plugin and with newer versions of Unraid. You can find it here: https://raw.githubusercontent.com/gfjar ... isk_15b.sh
The Preclear Disk plugin also uses Screen to manage multiple instances of the preclear_disk.sh script, same as if you were doing it from the command line, so you'll need Screen installed too.
I really wish there was a video on this
Spaceinvader One to the rescue, and he does give some additional details that I omitted from my write-up above: https://youtu.be/csGYrd5G0ik
Summarized Step-by-Step Instructions
- Install the Community Applications plugin (which itself is a front-end for installing all other plugins)
Code: Select all
Plugins > Install Plugin > https://raw.githubusercontent.com/Squidly271/community.applications/master/plugins/community.applications.plg - Install Nerd Pack Plugin: Apps > search for Nerd Pack
- Enable Screen in the Nerd Pack (Settings > Nerd Pack)
- Install Preclear Disk: Apps > search for Preclear Disk
- Download the patched version of Joe L.'s preclear_disk.sh script and save to \\<servername>\flash\config\preclear.disk. Make sure you save it as "preclear_disk.sh".
- With a target disk inserted in the server, run a Preclear: Tools > Preclear Disk
- Click "Start Preclear" for the disk you want to Preclear.
- Change the Script from "gfjardim - 1.0.3" to "Joe L. - 1.15"
- Set the options you want
My Preferences: Operation = Clear, Cycles = 1, Browser Notifications on every 25%, Default on Read/Write, and I don't skip the Pre-read
Hope this helps!
Paul