HTPCv3/Pi-NAS

Move Software RAID from Old NAS

Hard disk order in existing / legacy NAS:

gdisk -l /dev/sdb
Disk identifier (GUID): BC49445F-2559-4B6B-8387-09ACA853DB25

gdisk -l /dev/sdc
Disk identifier (GUID): 7D92F6A3-B46C-474C-8210-A86F5A8212F7

gdisk -l /dev/sdd
Disk identifier (GUID): B83091EF-B6EF-4F44-89A5-18C31E307701

gdisk -l /dev/sde
Disk identifier (GUID): 25EC6A05-8839-4F9D-801C-4860905FADCB

Match UUIDs /dev/sdX names with disk serial numbers:

$ udevadm info /dev/sdb | grep -E "ID_SERIAL|UUID"
E: ID_SERIAL=WDC_WD30EFRX-68EUZN0_WD-WMC4N1475240
E: ID_SERIAL_SHORT=WD-WMC4N1475240
E: ID_PART_TABLE_UUID=bc49445f-2559-4b6b-8387-09aca853db25

$ udevadm info /dev/sdc | grep -E "ID_SERIAL|UUID"
E: ID_SERIAL=WDC_WD30EFRX-68EUZN0_WD-WMC4N1911391
E: ID_SERIAL_SHORT=WD-WMC4N1911391
E: ID_PART_TABLE_UUID=7d92f6a3-b46c-474c-8210-a86f5a8212f7

$ udevadm info /dev/sdd | grep -E "ID_SERIAL|UUID"
E: ID_SERIAL=WDC_WD30EFRX-68EUZN0_WD-WMC4N1954715
E: ID_SERIAL_SHORT=WD-WMC4N1954715
E: ID_PART_TABLE_UUID=b83091ef-b6ef-4f44-89a5-18c31e307701

$ udevadm info /dev/sde | grep -E "ID_SERIAL|UUID"
E: ID_SERIAL=WDC_WD30EFRX-68EUZN0_WD-WCC4N0663527
E: ID_SERIAL_SHORT=WD-WCC4N0663527
E: ID_PART_TABLE_UUID=25ec6a05-8839-4f9d-801c-4860905fadcb

 

On the new machine /dev/sd[b,c,d,e] are /dev/sd[g,h,i,j] respectively.

With the disks in the new machine, and the mdadm.conf file copied over, the following command failed:

$ sudo mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/sdg (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdg
mdadm: No super block found on /dev/sdj (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdj
mdadm: No super block found on /dev/sdi (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdi
mdadm: No super block found on /dev/sdh (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdh
...
mdadm: Merging with already-assembled /dev/md/htpc-ubuntu:0
mdadm: /dev/sdg1 is identified as a member of /dev/md/htpc-ubuntu:0, slot 0.
mdadm: /dev/sdj1 is identified as a member of /dev/md/htpc-ubuntu:0, slot 3.
mdadm: /dev/sdi1 is identified as a member of /dev/md/htpc-ubuntu:0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md/htpc-ubuntu:0, slot 1.
mdadm: failed to add /dev/sdh1 to /dev/md/htpc-ubuntu:0: Device or resource busy
mdadm: failed to add /dev/sdi1 to /dev/md/htpc-ubuntu:0: Device or resource busy
mdadm: failed to add /dev/sdj1 to /dev/md/htpc-ubuntu:0: Device or resource busy
mdadm: failed to add /dev/sdg1 to /dev/md/htpc-ubuntu:0: Device or resource busy
mdadm: failed to RUN_ARRAY /dev/md/htpc-ubuntu:0: Input/output error

The OS had tried to build device md127 automatically. Just needed to stop that device and then the RAID was detected and started just fine:

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sde1[2] sdc1[1] sdd1[0] sdb1[4]
      11720536064 blocks super 1.2

$ sudo mdadm --stop /dev/md127
mdadm: stopped /dev/md127

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
unused devices: <none>

$ sudo mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/sdg (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdg
mdadm: No super block found on /dev/sdj (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdj
mdadm: No super block found on /dev/sdi (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdi
mdadm: No super block found on /dev/sdh (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdh
....
mdadm: /dev/sdg1 is identified as a member of /dev/md/0, slot 0.
mdadm: /dev/sdj1 is identified as a member of /dev/md/0, slot 3.
mdadm: /dev/sdi1 is identified as a member of /dev/md/0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md/0, slot 1.
mdadm: added /dev/sdh1 to /dev/md/0 as 1
mdadm: added /dev/sdi1 to /dev/md/0 as 2
mdadm: added /dev/sdj1 to /dev/md/0 as 3
mdadm: added /dev/sdg1 to /dev/md/0 as 0
mdadm: /dev/md/0 has been started with 4 drives.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid5 sdg1[0] sdj1[4] sdi1[2] sdh1[1]
      8790401640 blocks super 1.2 level 5, 4k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>

$ /usr/share/mdadm/checkarray --status /dev/md0
md0: idle

 

If the old mdadm.conf file is not available we can create a new one. Add the following to /etc/mdadm/mdadm.conf:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

Then run the following (as root) to append the disk array definition to the file:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

 

Add to fstab and mount, this remounts the RAID as read-write (note it was auo-read-only above and one could have used "mdadm --readwrite md0" but that would be temporary). Note that nofail is after auto, so that if the drive isn't present at boot time the system won't stall:

$ echo "/dev/md0 /media/9TB-Array ext4 rw,auto,nofail,errors=remount-ro,nosuid,nodev,uhelper=udisks2 0 0" >> /etc/fstab

$ mount -a

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdg1[0] sdj1[4] sdi1[2] sdh1[1]
      8790401640 blocks super 1.2 level 5, 4k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>


$ sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Mon Mar  3 14:14:29 2014
        Raid Level : raid5
        Array Size : 8790401640 (8383.18 GiB 9001.37 GB)
     Used Dev Size : 2930133880 (2794.39 GiB 3000.46 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Oct 14 12:51:45 2021
             State : clean
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 4K

Consistency Policy : bitmap

              Name : htpc-ubuntu:0
              UUID : 8ea128cd:47130c7a:9fb4a5a5:66750b6b
            Events : 16225

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       4       8       33        3      active sync   /dev/sdc1

 

Speed Test

USB is not a great protocol for disk access. A 4x3TB RAID-5 software array over a single USB3 port on a RasPi4 is not as slow as suspected:

Write:
$ dd status=progress if=/dev/zero of=/media/9TB-Array/zeros bs=1M count=1000
972029952 bytes (972 MB, 927 MiB) copied, 9 s, 108 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 9.61399 s, 109 MB/s

$ dd status=progress if=/dev/urandom of=/media/9TB-Array/urand bs=1M count=1000
1048576000 bytes (1.0 GB, 1000 MiB) copied, 57 s, 18.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 57.051 s, 18.4 MB/s

Read:

$ dd status=progress if=/media/9TB-Array/zeros of=/dev/null
990137856 bytes (990 MB, 944 MiB) copied, 13 s, 76.2 MB/s
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 13.9548 s, 75.1 MB/s

$ dd status=progress if=/media/9TB-Array/urand of=/dev/null
1023459328 bytes (1.0 GB, 976 MiB) copied, 18 s, 56.9 MB/s
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 18.4569 s, 56.8 MB/s

$ dd status=progress if=/media/9TB-Array/zeros of=/dev/null bs=1M count=1000
1009778688 bytes (1.0 GB, 963 MiB) copied, 7 s, 144 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 7.33777 s, 143 MB/s

$ dd status=progress if=/media/9TB-Array/urand2 of=/dev/null bs=1M count=1000
965738496 bytes (966 MB, 921 MiB) copied, 8 s, 121 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.66835 s, 121 MB/s

 

Rebuilding A Disk

If a disk has come out of sync from the array, but is not faulty, it can be re-added to the array to trigger a rebuild of that disk:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdd1[1] sda1[0] sdb1[2]
      8790401640 blocks super 1.2 level 5, 4k chunk, algorithm 2 [4/3] [UUU_]
      bitmap: 11/22 pages [44KB], 65536KB chunk

unused devices: <none>

$ dmesg
...
[   16.657864]  sdb: sdb1
[   16.658648] sd 1:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[   16.659270] sd 1:0:0:0: [sdb] Attached SCSI disk
[   16.688640]  sda: sda1
[   16.689570] sd 0:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[   16.690893] sd 0:0:0:0: [sda] Attached SCSI disk
[   16.843467] md: bind
[   16.902523]  sdc: sdc1
[   16.903293] sd 2:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
[   16.904040] sd 2:0:0:0: [sdc] Attached SCSI disk
[   16.967163]  sdd: sdd1
[   16.968186] sd 3:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16).
[   16.969042] sd 3:0:0:0: [sdd] Attached SCSI disk
[   16.986957] md: bind
[   17.109182] md: bind
[   17.222011] md: bind
[   17.250245] md: kicking non-fresh sdc1 from array!
[   17.250261] md: unbind
[   17.263527] md: export_rdev(sdc1)
[   17.268311] md/raid:md0: device sdd1 operational as raid disk 1
[   17.268316] md/raid:md0: device sda1 operational as raid disk 0
[   17.268318] md/raid:md0: device sdb1 operational as raid disk 2
[   17.271160] md/raid:md0: allocated 4362kB
[   17.271316] md/raid:md0: raid level 5 active with 3 out of 4 devices, algorithm 2
[   17.273380] RAID conf printout:
[   17.273382]  --- level:5 rd:4 wd:3
[   17.273385]  disk 0, o:1, dev:sda1
[   17.273387]  disk 1, o:1, dev:sdd1
[   17.273390]  disk 2, o:1, dev:sdb1
[   17.274980] created bitmap (22 pages) for device md0
[   17.276862] md0: bitmap initialized from disk: read 2 pages, set 123 of 44711 bits
[   17.299383] md0: detected capacity change from 0 to 9001371279360
[   18.083510] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
...

$ sudo umount /media/9TB-Array/
$ sudo mdadm --manage /dev/md0  --re-add /dev/sdc
mdadm: re-added /dev/sdc1
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdc1[4] sdd1[1] sda1[0] sdb1[2]
      8790401640 blocks super 1.2 level 5, 4k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  0.1% (4204764/2930133880) finish=689.0min speed=70766K/sec
      bitmap: 11/22 pages [44KB], 65536KB chunk

unused devices: <none>

$ sudo bash -c "echo 500000 > /proc/sys/dev/raid/speed_limit_min"
$ sudo bash -c "echo 500000 > /proc/sys/dev/raid/speed_limit_max"

# Later, when it's finished:
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdc1[4] sdd1[1] sda1[0] sdb1[2]
      8790401640 blocks super 1.2 level 5, 4k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>

---