03 Apr 2013

With the growing use of home media servers and home theater PCs (HTPCs), many of us now turn to larger capacity hard drives.  The traditional trend is to use a server with hardware RAID to take many smaller drives and turn them in to one larger drive for media, but many software based RAID options now exist to help keep the hardware cost down and simplify making large arrays that are protected from hardware failure.  Keep in mind that RAID is never a substitute for backup, however.  In this article we take a look at the software solution called FlexRAID, which can let your computer and take hard drives and act as a NAS (network attached storage).

Pricing and Info

FlexRAID is a software RAID/Storage Pool service that you can install on a workstation or server to let that machine act as a NAS (network attached storage) or storage system provider.   It can run on Linux or Windows.  FlexRAID can turn independent hard drives of various sizes, makes and models into what are called storage pools and also turn storage pools into storage clouds.  Not only that but FlexRAID provides data protection, monitoring and recovery through its own RAID implementation (RAID-F) over any file system (it won't destroy existing data).  In this article we will cover how to get FlexRAID set up and protecting data in a RAID configuration as well as a few benchmarks.

I should mention that you can download a free trial from the FlexRAID website and they have a cost breakdown here.  If you seek only Data Protection (RAID) then the cost is $39.95.  For just Storage Pooling with no protection of data the cost is $29.95.  There is a combo of these two for $59.95.  Here we are basically covering the combo set of features.

Installing FlexRAID

So lets get started installing and configuring FlexRAID 2.0.

Next we find the OpenEgg.org folder in the Start Menu and click on the FlexRAID.html file to open the web configuration window.  This essentially launches http://localhost:8080/?locale=en_US

Configuring and Using FlexRAID

 

In this case I chose Request a Trial License.  After this it brings you to the log in window which since none are set up, you can just click to login (leave blank).  This brings us to the main configuration window.

System Control Panel

System Toolbox

We can set the login and reset activation from this main window, amongst other options that can be done.

In the System Toolbox we can backup the FlexRAID Configuration Database.  When you click to back it up, it backs it up to "C:\Windows\system32\config\systemprofile/FlexRAID-Config.db.bak"  Note that in the case of x64 systems like Windows 7 x64, this path is actually "C:\Windows\syswow64\config\systemprofile/FlexRAID-Config.db.bak" and on Server 2012 this path was the root of the OS drive.

Notifications

In the notifications section you can configure specific types of email and SMS alerts.

Setting up a Snapshot RAID array using Cruise Control

Cruise Control mode is the simplified method for creating an array for protected (RAID) data.  In this section we configure a Snapshot RAID array with Cruise Control.  If you want to see how its done with full options and full control then skip down to the Expert section.  Note that you could also choose Real-Time instead of Snapshot.  I won't cover that in this article, but if you do use Real-Time, data is protected in real-time at the expense of some performance on the array.  In my case my data is mostly static, so Real-Time protection is really not required.

To get started we click "Add New Configuration".

Add New Configuration

We can use this to get the new setup going.  First we give it a name and choose the mode and type.  Snapshot type configurations will update the parity information on a pre-configured schedule whereas Real-Time happens as data changes in real-time.  Snapshot configurations should perform much faster in MB/sec than Real-Time.  For the first test I will choose Cruise Control and Snapshot.  Note that you could import an existing configuration via the PPU location if one existed.

This creates a shortcut on the FlexRAID desktop which if we open takes us to this window.

Now we open Drive Manager to begin the setup of the drives and RAID.  In my case I have three of the Seagate 4TB drives and I've set them up as GPT and formatted their partitions as NTFS with a drive letter for each.  If I hadn't formatted them and created drive letters, they simply wouldn't appear in the list and couldn't be used.

We can click one drive at a time and "Add to Storage Pool", in each case we give it a unique name and choose the "Unit of Risk" or UoR, which can be PPU, DRU or URU types.  A Unit of Risk in traditional RAID is a physical hard drive, but with FlexRAID it can be another type of data source.  A unit of risk is any unit that failure can be recovered from.  This could be a file, folder, LUN, media disc, standard RAID volume, etc.  This would be anything that has data and needs protection. 

A DRU or Data Risk Unit is a UoR that contains user data. It can be a single drive, a partition, a group of drives or folders, a network share, etc., anything that can contain your data.

A PPU is a Parity Protection Unit.  You can have one or more of these and its size is as big as your largest DRU. 

URU stands for undo recovery unit.  A URU is an optional element in Snapshot RAID used to backup the parity data about to be modified prior to executing a RAID update.  The concept is that if the update fails or a drive dies before the update completes, the URU can be used to revert the operation so that the failed drive can be recovered.  When using an URU, the update must succeed entirely or all changes to the parity data must be reverted.  A URU should be the size of the largest data change you anticipate making times the number of PPUs. If you change about 200GB of data max per update cycle, then your URU should have 200GB of available space for each PPU. The URU can be toggled as enabled or disabled on each update run.

So basically in my case I have 3 drives which in traditional RAID 5 means 1 drive is parity (or one of the drives maximum capacity becomes the parity).  This is the same with FlexRAID (except the parity isn't striped across all three drives).  So for one you would choose PPU and the other two will simply be DRU.  That is, the PPU size has to be the size of your largest DRU.  Keep in mind the "DRU" drives can already contain data and won't be erased, however, the drive you choose to be the parity, in this case, the PPU type, will have its data erased in the "Initialize RAID" step after you have added all the drives to the storage pool.

Now we go back to the main view and click on one of the drives in the newly created storage pool.  We click Initialize RAID and it warns you that the PPU drive will be erased.  The progress window will appear.

Next back in the main window for the new configuration, we click "Start Storage Pool".  It warns you to reboot and then click Start.

After rebooting, you can simply go back into the panel and click to start the storage pool.  After you do this, a new drive letter will appear, in this case drive letter V which has 7.2 TB of free space.  You can configure the drive letter in the settings to be a different letter if you choose.

You can also go into the settings on the storage pool and configure it to start automatically:

Note that in order for network sharing of the RAID array to work correctly, it is best to use the FlexRAID Shares section to setup sharing from there, rather than from within Windows itself.

See the section below on scheduling Updates, Validation and Verification when using Snapshot mode, to ensure your data is protected.

Setting up a Storage Pool and RAID array using Expert Mode

Storage Pool Creation

First we need to create a new Storage Pool.

Click "Add New Configuration"

Give it a name (for the storage pool).  Select Expert Mode.

After clicking create, we can double click the storage pool on the FlexRaid desktop and go to the Storage Pool Preferences and set it to start automatically and 1000 ms for the delay.  Be sure you click save.

If you want to change the drive letter, you can go to Storage Pool Merge Configuration and change it there.  Note that its in the Storage Pool Merge Configuration section where we actually would click stop to stop the storage pool if you want to delete it.  You might not think to look there for this button, but that is where it is.  You should leave the setting of Auto-Folder-Priority for the most energy efficient use when copying files.  There is also a setting in Run-Time Properties for the file size reserve.  The default is 50GB.  If you think you will be copying files larger than 50GB to the storage pool/raid array, you may want to increase this value to prevent insufficient space errors on copy. 

NOTE:  We also have to go to Run-Time Properties and type in the paths to each drive we want to include in the Storage Pool.  If we don't input values here, it will just pool everything including the PPU drive which isn't what we want for raid.

UPDATE:  In the above image I have the drive letters one per line.. for some reason in my configuration this actually ended up not working properly when I would recreate the configuration.  I ended up using this format all on the same line:  "G:\;H:\;"

We can also set the server shares as with cruise control mode.  You should always set the server shares in this interface, rather than the Windows default one, to prevent potential issues.

It is necessary to go to Storage Pool Merge Configuration and click the Publish button to get the pool started when you are all set.

Creating the RAID array in Expert mode

Now we can click "Add new configuration" again and select expert mode, giving it a name and leaving the rest at defaults.  However, I chose to set the RAID Engine to TX.  This giving more flexibility on PPU (Parity) down the road.  After clicking create you can double click the Raid array config on the desktop to open its settings.  As with Cruise Control mode, each PPU you use must be at least the same size as your largest DRU.

UPDATE:  Just a note, in the above image, I have G and H both under DRU1.. this is considered DRU spanning.  If you are just intending to have one drive per DRU, the proper way is to have DRU1 with G: and then DRU2 with H:  If you do the configuration as in the above image, parity updating will fail, because the PPU size is smaller than G and H combined in a spanned single DRU.  Also note, that if any of your DRU drives are bigger even by a small margin, than your PPU drive, then the parity update will also complain of lack of space for parity information.  In the case of my one drive it was formatted slightly larger than the PPU drive, so I ended up needing to shrink the DRU drive in Disk Management to get the sizes to line up.

In this window we can simply drag the drives you want to be DRU's to the DRU window and any parity drives to the PPU window.

Next we click "Command Execution Center" and create.

If you had data on your Parity drive from a previous configuration, it will warn you before overwriting the data.

Snapshot RAID - Scheduling and Updates, Validation, Verifications

The commands to Update, Validate, etc can be found in the "Command Execution Center."  They can be run manually or set in a schedule.

Quick Validate

In this area there is a "Quick Validate" task.  It can be run to report the amount of changes since the last update.  Even with big arrays it's very fast.  At the end it will either say success, no changes found or it will say some were found, meaning you need to run an Update.

Validate

Validate is for change detection with corruption detection via checksum validation.  Quick Validate just looks at time stamps where Validate scans content.  After running you can get one of two results.  One result being the array is healthy or the other if changes are detected.  With changes being detected you should run the Update task.  Even with static data, its good practice to validate at least weekly.

Update

Update will update the parity info on the PPU drive.  It will succeed or tell you the PPU has failed (or a DRU has failed).  If it has failed you can use a Restore to fix the failed data.  It is generally a good idea to run an Update daily.

**If the logs are indicating problems then you should NEVER click Update.

Verify

Verify uses bit for bit verification on the entire RAID array.  The process can take some time (longer than a Validate).  You get one of two results at the end.  One is that "Verify Success: all bits match"  which is a good message.  Another is "Verify: bit mismatch" which can mean the array is not in sync or healthy.  In the case of not matching it could be because you modified the data and you need to run an update first.  If you have a failed drive or datarot then you need to do a Restore.  Good practice is to run a Verify at least once a month.

Restoring after Data Loss

As mentioned, if validate or quick validate reveal issues then you can restore the data or replace the hard drive and restore the data (use Hard Disk diagnostics to determine the root cause).  In the "Command Execution Center" you can restore DRU's and PPUs as needed.  After restoration run a "Quick Validate" then run Update if everything is ok.

Scheduler

With Snapshot RAID you can the Scheduler to define when to update, validate and verify data on the array automatically. 

The recommended schedule is to do an update daily, validate weekly and verify monthly as mentioned previously.  One possible schedule might be updating at 1am daily, validate at 2am each Wednesday and do a manual verify at the start of every month at 3am (manual here is to avoid a conflict with the weekly validate on a Wednesday).

We open Scheduler and select the boxes similar to below.  In this case we set the Update for the Raid array.  Here I chose every day at 145 am using Standard Entry style.  Note that you can also specify and end date if you wish.  Click "add schedule" and it gets added.  Repeat the same for validate and verify.

Warning:  You need to make sure the array isn't being accessed during these operations.  If media is being added to your array and modifying metadata, for instance, the update might succeed, but things like verify and validate will fail.  Watch out also for files that have dates in the future.

Generally speaking, when you do an update, it should become quick each day, depending on the modifications done to data, say the 20-40 minute range in time.  You should always run an update before verify/validate to ensure the array is synced before checking integrity of the data.

Benchmarks

Lets take a look at my own benchmarks of a single drive using PerformanceTest and see how the Disk Mark compares with a Raid TX single drive parity setup that we will configured.  Our FlexRAID RAID TX array will consist of 3 drives, in this case 5900 RPM Seagate drives which are the STCA4000100 drives.  In that previous article, I was using a different hardware set for the benchmarks.  So I re-ran the benchmark on my homemade NAS build and came up with the following result for a single one of the drives:

As you can see the Disk Mark was 965.8 and the Sequential Read 125.1 MB/sec, Sequential Write 138.6 MB/sec and Random Seek + RW 3.36 MB/sec.  All of these very good numbers for this 5900 RPM SATA III drive which is on the motherboard SATA III controller.  We can use this as a rough baseline to compare actual FlexRAID performance as we get the setup configured and tested.

I opted to use PerformanceTest to test both a snapshot and real-time FlexRAID configuration and got the following (skewed) results.

As you can see this results simply cannot be correct.  PerformanceTest reported Sequential reads at 792.3 MB/sec, writes at 733.3 MB/sec and random 6.2 MB/sec.  Apparently there is some major caching going on with this benchmark in this situation.  I also got nearly identical numbers when using PerformanceTest on a real-time array as well.

So at this point I opted for some large file Windows transfer tests as a gauge on performance.  Some initial results with both the snapshot and real-time are below.

This was more consistent with what I had expected the FlexRAID performance to be.  In general, since configuring either a snapshot or a real-time array, I'm seeing average numbers in the 90-120 MB/sec range, which is more than acceptable for a media server in my case.  This was transferring a large file from the OS drive which is SATA III to the Snapshot RAID array.  I did not see a large increase in performance by choosing Snapshot, though ultimately I have opted to stick with Snapshot arrays for now.  During parity updates I usually see between 118 MB/sec and 128 MB/sec.

Tech-Stew Take Home

FlexRAID is a very powerful software application that can let you store large amounts of data with protection like traditional hardware RAID setups, but at a fraction of the cost with more flexibility.  Thanks to its RAID engine you can mix and match drives, have multiple (or many) parity drives and choose between real-time and snapshot updates on the parity information.  Once you get a good handle on how the interface works and the various options, you will find that FlexRAID should do the trick for your needs, but don't expect hardware RAID blazing speed as with traditional striped RAID sets.

 


Use this QR code in a QR reader application on mobile to open quickly on a mobile device

blog comments powered by Disqus

Flight-Stew Now Live

Be sure to visit our newest site, Flight-Stew, now active!

Recently Added

Subscribe

Poll

Which format do you prefer for buying games on Xbox One and PS4?


Show Results

[forcestyle][nozoom][nocaption]

Donate

       Support Tech-Stew
   [nozoom]
        Make a Donation

Recent Comments