OS Installation

Q
How do I rebuild a software RAID 1 array after a disk failure?
A

* Behavior

  • How do I rebuild a software RAID 1 array after a disk failure?
  • One of my software RAID disks died and my system won't boot.

* Environment

  • Red Hat Enterprise Linux 5 or later.

* Resolution

  1. Boot the machine with only the "good disk" in the system.
  2. Look at /proc/mdstat and fdisk -l output to verify RAID status. Paste output into the ticket if possible.
  3. Add the new disk into the system and boot.
  4. Install grub on both disks so if a falure occurs the system will still have one disk with a mbr:
    [root@host ~]# grub-install /dev/sda
    [root@host ~]# grub-install /dev/sdb
  5. Partition the new disk to make it exactly like the disk that is still there with the same starting and ending cylinders:
    [root@host ~]# fdisk /dev/sda
  6. Make the partition ID read "fd" which is "Linux raid autodetect" as in the following example:
    Disk /dev/sda: 750.1 GB, 750156374016 bytes
    255 heads, 63 sectors/track, 91201 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 38 305203+ fd Linux raid autodetect
    /dev/sda2 39 1058 8193150 fd Linux raid autodetect
    /dev/sda3 1059 91201 724073647+ fd Linux raid autodetect
  7. Tell the kernel to detect the newly created partitions:
    [root@host ~]# partprobe
  8. Rebuild each md device by adding the second disk back in using the /proc/mdstat output with the mdadm command:
    [root@host ~]# mdadm /dev/md0 --add /dev/sda1
    [root@host ~]# mdadm /dev/md1 --add /dev/sda3
    [root@host ~]# mdadm /dev/md2 --add /dev/sda2
  9. Monitor the progress of the rebuild with this command:
    [root@host ~]# watch -n 2 cat /proc/mdstat
Product Model
QuantaGrid D51B-1U QuantaGrid D51BP-1U QuantaGrid D51PS-1U QuantaGrid D51PC-1U QuantaGrid D51B-2U QuantaGrid D51BV-2U QuantaPlex T41S-2U (4-Node) QuantaPlex T41SP-2U (4-Node) STRATOS S810-X52L (4-Node) STRATOS S910-X31E (9-Node) STRATOS S910-X31B (9-Node) STRATOS S900-X31A