Wednesday, June 6, 2018

Software RAID with NVME devices

  1. Assuming there is no data on the devices ( if there is data, DO NOT run these commands, data will be lost! ), lets wipe the existing superblocks
    # mdadm --zero-superblock /dev/nvme1n1
    # mdadm --zero-superblock /dev/nvme2n1

    if that fails with the error below or similar
    mdadm: /dev/nvme1n1 appears to be part of a raid array:
    level=raid0 devices=0 ctime=Fri Jun  1 07:00:00 2018

    clean up the first 1GB from that disk
    # dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1000
    # dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=1000

  2. And revert mdadm.conf back to its defaults
    # echo "DEVICE /no/device" > /etc/mdadm.conf

  3. Ensure the nvme devices are still blacklisted in /etc/multipath.conf if you are using multipath.
    blacklist {
      devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|nbd)[0-9]*"
      devnode "^hd[a-z][0-9]*"
      devnode "^etherd"
      devnode "^nvme.*"
      wwid 3600605b00d9c63204000540309e0e2c9150


    that should prevent device-mapper-multipath from using the nvme disks
  4. Reboot
  5. Assemble the software raid after the reboot
    # mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1

  6. Add the nvme devices to mdadm.conf
    # echo "DEVICE /dev/nvme1n1" > /etc/mdadm.conf
    # echo "DEVICE /dev/nvme2n1" >> /etc/mdadm.conf

  7. And add the md0 config to mdadm.conf
    # mdadm --detail --scan >> /etc/mdadm.conf

  8. Remove the DEVICE entries from /etc/mdadm.conf afterwards, so that it only contains the ARRAY line
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=78d45ac:0434da03:ce4a9b23:c946bb65

  9. Reboot
The UUID displayed for the same disk in blkid command and mdadm.conf file differ of value:

$ blkid |grep -i md5
/dev/md5: UUID="b26359eb-51ce-42b2-9ec4-ed39b4fd9e2f" TYPE="ext3"

$ cat /etc/mdadm.conf |grep -i md5

ARRAY /dev/md5 level=raid1 num-devices=2 uuid=21b778e3:e267c7dd:c96f4304:db5d882f

  UUID mentioned in blkid is an identity for filesystem on the md block device while the one in /etc/mdadm.conf identify the md block device itself.

UUID of blkid is stored in filesystem structure (consistent with stab) and it helps in uniquely identifying the filesystem among the available filesystems on the system, while uuid of mdadm.conf resides in the device metadata and helps the md subsystem identify that particular RAID device uniquely.

In particular, it helps identify all the block devices that belong to the RAID array. Difference of UUID also exists for raid array, device, partition, LVM, PV, VG etc.

No comments:

Post a Comment