RAID, or Redundant Array of Independent Disks, is a way to spread data across a set of drives to prevent the loss of one drive causing permanent data loss. RAID works in two different ways: (i) a complete mirror image of the data is stored on a second drive; or (ii) parity blocks are added to the data in order to recover failed blocks.
Various challenges are increasingly coming to light with respect to RAID especially as HDDs grow in size (6 TB HDDs are now available), and new technologies are being built, including erasure coding. Organizations need to take these factors into account when considering how to best store their data.
Both processes involved in RAID increase the amount of storage used. Mirroring doubles data size while parity usually adds one-fifth more data. Performance penalties also occur. Two drive write operations are necessary when writing an updated block, and parity usually requires the reading of blocks from all the drives to allow the generation of the missing blocks.
A dual-parity approach (called RAID 6) has been adopted by industry, which entails the creation of two non-overlapping parities for each drive set. RAID 6 took a hit, however, with the release of 4TB and larger drive sizes as rebuild can take days, plus the risk of failure during the procedure has also grown with increased HDD capacity.
When a disk does fail, the Raid Rebuild process starts, during which time there isn’t protection against a second or third mechanism failure.
Nonetheless, the most common techniques used by object storage systems, including services in the cloud, is data distribution coupled with replication of the dataset. It is essentially an extension of mirroring with three copies being made, one of which is typically remote, which allows for disaster recovery if the other two copies fail or go down.
One alternative to Raid is erasure coding (EC). Erasure coding involves breaking the data into fragments, which are then expanded and encoded with redundant data pieces and stored over a number of different locations or storage media. The combination of complex math and parity-like blocks creates a hardy protection scheme, which can tolerate high levels of failure.
EC is typically used in distributed storage systems whereas RAID is used for redundant storage of entire disks within a server. Erasure coding is normally specified in an N+M format: 10+6, for instance, means that data and erasure codes are distributed across 16 (N+M) drives, and that any 10 of those are able to recover data. A full six drives can therefore fail; a significantly higher number than replication.
Erasure coding is capable of regenerating data on all available disks and from data on various different disks that allows for a huge rebuild bandwidth (sometimes referred to as “declustered rebuild”). This significantly improves the rebuild time as opposed to RAID in which the rebuild is confined to the resources of the disk set.
EC is not efficient for encoding small objects i.e. datasets involving less than 1 million files; or for small clusters because of the large number of devices required to achieve fault-tolerance.
Many industry experts currently recommend the use of replication for active primary and secondary data, and the use of erasure coding for archival (“warm” or “cold” data), in which performance is not an issue.