| dm-raid |
| ======= |
| |
| The device-mapper RAID (dm-raid) target provides a bridge from DM to MD. |
| It allows the MD RAID drivers to be accessed using a device-mapper |
| interface. |
| |
| |
| Mapping Table Interface |
| ----------------------- |
| The target is named "raid" and it accepts the following parameters: |
| |
| <raid_type> <#raid_params> <raid_params> \ |
| <#raid_devs> <metadata_dev0> <dev0> [.. <metadata_devN> <devN>] |
| |
| <raid_type>: |
| raid1 RAID1 mirroring |
| raid4 RAID4 dedicated parity disk |
| raid5_la RAID5 left asymmetric |
| - rotating parity 0 with data continuation |
| raid5_ra RAID5 right asymmetric |
| - rotating parity N with data continuation |
| raid5_ls RAID5 left symmetric |
| - rotating parity 0 with data restart |
| raid5_rs RAID5 right symmetric |
| - rotating parity N with data restart |
| raid6_zr RAID6 zero restart |
| - rotating parity zero (left-to-right) with data restart |
| raid6_nr RAID6 N restart |
| - rotating parity N (right-to-left) with data restart |
| raid6_nc RAID6 N continue |
| - rotating parity N (right-to-left) with data continuation |
| raid10 Various RAID10 inspired algorithms chosen by additional params |
| - RAID10: Striped Mirrors (aka 'Striping on top of mirrors') |
| - RAID1E: Integrated Adjacent Stripe Mirroring |
| - RAID1E: Integrated Offset Stripe Mirroring |
| - and other similar RAID10 variants |
| |
| Reference: Chapter 4 of |
| http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf |
| |
| <#raid_params>: The number of parameters that follow. |
| |
| <raid_params> consists of |
| Mandatory parameters: |
| <chunk_size>: Chunk size in sectors. This parameter is often known as |
| "stripe size". It is the only mandatory parameter and |
| is placed first. |
| |
| followed by optional parameters (in any order): |
| [sync|nosync] Force or prevent RAID initialization. |
| |
| [rebuild <idx>] Rebuild drive number 'idx' (first drive is 0). |
| |
| [daemon_sleep <ms>] |
| Interval between runs of the bitmap daemon that |
| clear bits. A longer interval means less bitmap I/O but |
| resyncing after a failure is likely to take longer. |
| |
| [min_recovery_rate <kB/sec/disk>] Throttle RAID initialization |
| [max_recovery_rate <kB/sec/disk>] Throttle RAID initialization |
| [write_mostly <idx>] Mark drive index 'idx' write-mostly. |
| [max_write_behind <sectors>] See '--write-behind=' (man mdadm) |
| [stripe_cache <sectors>] Stripe cache size (RAID 4/5/6 only) |
| [region_size <sectors>] |
| The region_size multiplied by the number of regions is the |
| logical size of the array. The bitmap records the device |
| synchronisation state for each region. |
| |
| [raid10_copies <# copies>] |
| [raid10_format <near|far|offset>] |
| These two options are used to alter the default layout of |
| a RAID10 configuration. The number of copies is can be |
| specified, but the default is 2. There are also three |
| variations to how the copies are laid down - the default |
| is "near". Near copies are what most people think of with |
| respect to mirroring. If these options are left unspecified, |
| or 'raid10_copies 2' and/or 'raid10_format near' are given, |
| then the layouts for 2, 3 and 4 devices are: |
| 2 drives 3 drives 4 drives |
| -------- ---------- -------------- |
| A1 A1 A1 A1 A2 A1 A1 A2 A2 |
| A2 A2 A2 A3 A3 A3 A3 A4 A4 |
| A3 A3 A4 A4 A5 A5 A5 A6 A6 |
| A4 A4 A5 A6 A6 A7 A7 A8 A8 |
| .. .. .. .. .. .. .. .. .. |
| The 2-device layout is equivalent 2-way RAID1. The 4-device |
| layout is what a traditional RAID10 would look like. The |
| 3-device layout is what might be called a 'RAID1E - Integrated |
| Adjacent Stripe Mirroring'. |
| |
| If 'raid10_copies 2' and 'raid10_format far', then the layouts |
| for 2, 3 and 4 devices are: |
| 2 drives 3 drives 4 drives |
| -------- -------------- -------------------- |
| A1 A2 A1 A2 A3 A1 A2 A3 A4 |
| A3 A4 A4 A5 A6 A5 A6 A7 A8 |
| A5 A6 A7 A8 A9 A9 A10 A11 A12 |
| .. .. .. .. .. .. .. .. .. |
| A2 A1 A3 A1 A2 A2 A1 A4 A3 |
| A4 A3 A6 A4 A5 A6 A5 A8 A7 |
| A6 A5 A9 A7 A8 A10 A9 A12 A11 |
| .. .. .. .. .. .. .. .. .. |
| |
| If 'raid10_copies 2' and 'raid10_format offset', then the |
| layouts for 2, 3 and 4 devices are: |
| 2 drives 3 drives 4 drives |
| -------- ------------ ----------------- |
| A1 A2 A1 A2 A3 A1 A2 A3 A4 |
| A2 A1 A3 A1 A2 A2 A1 A4 A3 |
| A3 A4 A4 A5 A6 A5 A6 A7 A8 |
| A4 A3 A6 A4 A5 A6 A5 A8 A7 |
| A5 A6 A7 A8 A9 A9 A10 A11 A12 |
| A6 A5 A9 A7 A8 A10 A9 A12 A11 |
| .. .. .. .. .. .. .. .. .. |
| Here we see layouts closely akin to 'RAID1E - Integrated |
| Offset Stripe Mirroring'. |
| |
| <#raid_devs>: The number of devices composing the array. |
| Each device consists of two entries. The first is the device |
| containing the metadata (if any); the second is the one containing the |
| data. |
| |
| If a drive has failed or is missing at creation time, a '-' can be |
| given for both the metadata and data drives for a given position. |
| |
| |
| Example Tables |
| -------------- |
| # RAID4 - 4 data drives, 1 parity (no metadata devices) |
| # No metadata devices specified to hold superblock/bitmap info |
| # Chunk size of 1MiB |
| # (Lines separated for easy reading) |
| |
| 0 1960893648 raid \ |
| raid4 1 2048 \ |
| 5 - 8:17 - 8:33 - 8:49 - 8:65 - 8:81 |
| |
| # RAID4 - 4 data drives, 1 parity (with metadata devices) |
| # Chunk size of 1MiB, force RAID initialization, |
| # min recovery rate at 20 kiB/sec/disk |
| |
| 0 1960893648 raid \ |
| raid4 4 2048 sync min_recovery_rate 20 \ |
| 5 8:17 8:18 8:33 8:34 8:49 8:50 8:65 8:66 8:81 8:82 |
| |
| |
| Status Output |
| ------------- |
| 'dmsetup table' displays the table used to construct the mapping. |
| The optional parameters are always printed in the order listed |
| above with "sync" or "nosync" always output ahead of the other |
| arguments, regardless of the order used when originally loading the table. |
| Arguments that can be repeated are ordered by value. |
| |
| |
| 'dmsetup status' yields information on the state and health of the array. |
| The output is as follows (normally a single line, but expanded here for |
| clarity): |
| 1: <s> <l> raid \ |
| 2: <raid_type> <#devices> <health_chars> \ |
| 3: <sync_ratio> <sync_action> <mismatch_cnt> |
| |
| Line 1 is the standard output produced by device-mapper. |
| Line 2 & 3 are produced by the raid target and are best explained by example: |
| 0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0 |
| Here we can see the RAID type is raid4, there are 5 devices - all of |
| which are 'A'live, and the array is 2/490221568 complete with its initial |
| recovery. Here is a fuller description of the individual fields: |
| <raid_type> Same as the <raid_type> used to create the array. |
| <health_chars> One char for each device, indicating: 'A' = alive and |
| in-sync, 'a' = alive but not in-sync, 'D' = dead/failed. |
| <sync_ratio> The ratio indicating how much of the array has undergone |
| the process described by 'sync_action'. If the |
| 'sync_action' is "check" or "repair", then the process |
| of "resync" or "recover" can be considered complete. |
| <sync_action> One of the following possible states: |
| idle - No synchronization action is being performed. |
| frozen - The current action has been halted. |
| resync - Array is undergoing its initial synchronization |
| or is resynchronizing after an unclean shutdown |
| (possibly aided by a bitmap). |
| recover - A device in the array is being rebuilt or |
| replaced. |
| check - A user-initiated full check of the array is |
| being performed. All blocks are read and |
| checked for consistency. The number of |
| discrepancies found are recorded in |
| <mismatch_cnt>. No changes are made to the |
| array by this action. |
| repair - The same as "check", but discrepancies are |
| corrected. |
| reshape - The array is undergoing a reshape. |
| <mismatch_cnt> The number of discrepancies found between mirror copies |
| in RAID1/10 or wrong parity values found in RAID4/5/6. |
| This value is valid only after a "check" of the array |
| is performed. A healthy array has a 'mismatch_cnt' of 0. |
| |
| Message Interface |
| ----------------- |
| The dm-raid target will accept certain actions through the 'message' interface. |
| ('man dmsetup' for more information on the message interface.) These actions |
| include: |
| "idle" - Halt the current sync action. |
| "frozen" - Freeze the current sync action. |
| "resync" - Initiate/continue a resync. |
| "recover"- Initiate/continue a recover process. |
| "check" - Initiate a check (i.e. a "scrub") of the array. |
| "repair" - Initiate a repair of the array. |
| "reshape"- Currently unsupported (-EINVAL). |
| |
| |
| Discard Support |
| --------------- |
| The implementation of discard support among hardware vendors varies. |
| When a block is discarded, some storage devices will return zeroes when |
| the block is read. These devices set the 'discard_zeroes_data' |
| attribute. Other devices will return random data. Confusingly, some |
| devices that advertise 'discard_zeroes_data' will not reliably return |
| zeroes when discarded blocks are read! Since RAID 4/5/6 uses blocks |
| from a number of devices to calculate parity blocks and (for performance |
| reasons) relies on 'discard_zeroes_data' being reliable, it is important |
| that the devices be consistent. Blocks may be discarded in the middle |
| of a RAID 4/5/6 stripe and if subsequent read results are not |
| consistent, the parity blocks may be calculated differently at any time; |
| making the parity blocks useless for redundancy. It is important to |
| understand how your hardware behaves with discards if you are going to |
| enable discards with RAID 4/5/6. |
| |
| Since the behavior of storage devices is unreliable in this respect, |
| even when reporting 'discard_zeroes_data', by default RAID 4/5/6 |
| discard support is disabled -- this ensures data integrity at the |
| expense of losing some performance. |
| |
| Storage devices that properly support 'discard_zeroes_data' are |
| increasingly whitelisted in the kernel and can thus be trusted. |
| |
| For trusted devices, the following dm-raid module parameter can be set |
| to safely enable discard support for RAID 4/5/6: |
| 'devices_handle_discards_safely' |
| |
| |
| Version History |
| --------------- |
| 1.0.0 Initial version. Support for RAID 4/5/6 |
| 1.1.0 Added support for RAID 1 |
| 1.2.0 Handle creation of arrays that contain failed devices. |
| 1.3.0 Added support for RAID 10 |
| 1.3.1 Allow device replacement/rebuild for RAID 10 |
| 1.3.2 Fix/improve redundancy checking for RAID10 |
| 1.4.0 Non-functional change. Removes arg from mapping function. |
| 1.4.1 RAID10 fix redundancy validation checks (commit 55ebbb5). |
| 1.4.2 Add RAID10 "far" and "offset" algorithm support. |
| 1.5.0 Add message interface to allow manipulation of the sync_action. |
| New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt. |
| 1.5.1 Add ability to restore transiently failed devices on resume. |
| 1.5.2 'mismatch_cnt' is zero unless [last_]sync_action is "check". |
| 1.6.0 Add discard support (and devices_handle_discard_safely module param). |
| 1.7.0 Add support for MD RAID0 mappings. |
