Software RAID with NVME devices

1. Assuming there is no data on the devices ( if there is data, DO NOT run these commands, data will be lost! ), lets wipe the existing superblocks
   
   # mdadm --zero-superblock /dev/nvme1n1
   # mdadm --zero-superblock /dev/nvme2n1
   
   if that fails with the error below or similar
   
   mdadm: /dev/nvme1n1 appears to be part of a raid array:
   level=raid0 devices=0 ctime=Fri Jun  1 07:00:00 2018
   
   clean up the first 1GB from that disk
   
   # dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1000
   # dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=1000
   
   
2. And revert mdadm.conf back to its defaults
   
   # echo "DEVICE /no/device" > /etc/mdadm.conf
   
   
3. Ensure the nvme devices are still blacklisted in /etc/multipath.conf if you are using multipath.
   
   blacklist {
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|nbd)[0-9]*"
     devnode "^hd[a-z][0-9]*"
     devnode "^etherd"
     devnode "^nvme.*"
     wwid 3600605b00d9c63204000540309e0e2c9150
   
   }
   
   that should prevent device-mapper-multipath from using the nvme disks
4. Reboot
5. Assemble the software raid after the reboot
   
   # mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1
   
   
6. Add the nvme devices to mdadm.conf
   
   # echo "DEVICE /dev/nvme1n1" > /etc/mdadm.conf
   # echo "DEVICE /dev/nvme2n1" >> /etc/mdadm.conf
   
   
7. And add the md0 config to mdadm.conf
   
   # mdadm --detail --scan >> /etc/mdadm.conf
   
   
8. Remove the DEVICE entries from /etc/mdadm.conf afterwards, so that it only contains the ARRAY line
   
   ARRAY /dev/md0 level=raid1 num-devices=2 UUID=78d45ac:0434da03:ce4a9b23:c946bb65
   
   
9. Reboot

The UUID displayed for the same disk in blkid command and mdadm.conf file differ of value:

$ blkid |grep -i md5
/dev/md5: UUID="b26359eb-51ce-42b2-9ec4-ed39b4fd9e2f" TYPE="ext3"

$ cat /etc/mdadm.conf |grep -i md5
ARRAY /dev/md5 level=raid1 num-devices=2 uuid=21b778e3:e267c7dd:c96f4304:db5d882f

  UUID mentioned in blkid is an identity for filesystem on the md block device while the one in /etc/mdadm.conf identify the md block device itself.

UUID of blkid is stored in filesystem structure (consistent with stab) and it helps in uniquely identifying the filesystem among the available filesystems on the system, while uuid of mdadm.conf resides in the device metadata and helps the md subsystem identify that particular RAID device uniquely.

In particular, it helps identify all the block devices that belong to the RAID array. Difference of UUID also exists for raid array, device, partition, LVM, PV, VG etc.