Steps to recover when RAID1 fails

Note: The original version of this file is at http://codeine.rocks.net.nz/raidrecover.html.

Overview

1. Replace offending disk
2. Partition it to match removed disk
3. /sbin/raidhotadd /dev/mdX /dev/sdx

More Detail

• If necessary, or prudent, power down the system and remove the offending disk. Replace it, and bring the system up. If the offending disk was the boot disk, try moving the mirror to it's position in the array, and place the new disk in the mirror position.
• CEK: Our system would not boot from the mirror - we had to boot from the boot floppy.
• Use cat /proc/mdstat to examine the state of RAID, it will pretty clearly indicate the absence of the mirror.
• Use fdisk to get details of the operating disk's partition structure, and record. eg.:

  fdisk /dev/sda
  p (prints the partition table)
  q (quits)
  

• Use one of the partitioning tools ("Disk Druid" or fdisk) to recreate the partition structure, matching the disk that is still operating. The partition types of most use are:
  • 83 - (default) Linux
  • 82 - Linux Swap
  • FD Linux Raid Autodetect
  Don't forget to write your changes to the disk.
• CEK: Be sure to set the first partition as bootable
• CEK: We used fdisk to partition the drive. After you write the partition map, fdisk advices you to reboot. This is not necessary.
• CEK - You can use the command sfdisk to copy the partition table - much easier than doing it via fdisk. (if your second hard disk is the same size as your first)
• Using the information in the /etc/raidtab file, issue as many of the following commands as necessary to regenerate the mirrors.

  /sbin/raidhotadd /dev/mdX /dev/sdxN

  (Where X is the raid device number, x is the device and N the partition)

• CEK: It is helpful to note that there is not a one-to-one correspondence between X and N. Read the raidtab.
• Monitor progress of the regeneration by:

  cat /proc/mdstat

  Note: The regeneration is sequential for each mdX device...

• CEK: Here's another good recovery script.