A few weeks back Amazon had a good deal on some WD 10Tb USB drives. Since I was going on holiday shortly and probably going to come back with a large amount of photos and video, I thought it would make sense to increase my storage.
Presently I have a 5TB drive and then 2 5TB drives for backup. So by purchasing another 5TB drive to give me 4 5TB drives (of the same model) I could set them up as a fast 10TB RAID 10 disk. And then get a couple of the 10TB drives for backup.
In hindsight, it wasn't needed (I didn't take many photos on holiday), and just buying a normal drive rather than going RAID would have been so much easier and quicker.
The first issue was that since I was needing to wipe the 5TB drives, I had to copy the data from the 'main' disk to both the new backup disks. Then do binary comparisons of the new 10TB backups against the old 5TB backups, to ensure the data had copied okay and avoid any bit-rot issues.
After about a week, I'd got that job done. Now it came to actually installing the 5TB disks in my PC. Unfortunately my PC only has mounting points for 2 3.5″ drives (who uses mechanical drives in their PC nowadays?). I scavenged some parts from work in an attempt to find some way of getting the setup I wanted.
My case is a Corsair Air 540. It has two sides - one for the PSU, 5″ drive bay and SATA SSDs, and the other side for the motherboard, cooling, and 2 3.5″ drives. In the smaller side there are two spots the drives could be placed: above the SSDs (behind the 5″ drive bays); or below the 5″ drive bay.
The problem with placing the drives below the 5″ bay is that there is no airflow in this location and no way to add any. Basically the drives would be surrounded by the 5″ bay (occupied with a backplane) on one side, the solid case front on another, the solid bottom of the case on another, and the PSU (where the cables are attached) on the other.
The other option I did think of was having the drives just in a backplane outside of the case (acting like a DAS). There are holes in the case below where the 3.5″ drives mount, so feeding SATA and power cables out of the case to a backplane just outside wouldn't be too difficult. But this option wasn't good for me as I don't really have space near the PC for this. It's a bit annoying as well having a large case and then having to mount stuff outside of it.
I did also think of using a server tower case. This would accommodate the hardware much better than the Air 540. But airflow would be much worse - front intake would just be the backplane fans pulling in what they can through the backplanes (i.e. not much), and the rear fan is a single 3 pin 120mm. I'd rather stick with the 140mm Noctuas on my Air 540.
I'm also hoping to move to water cooling at some point, which I think will work much better with the Air 540 than a server case that has nowhere to mount radiators. I may actually move to an open air case when I do that though.
So, the only place I could see to 'mount' the drives was above the SSDs. Unfortunately a backplane is too large to mount in this area. Some of the parts I salvaged were 2 parts that take 2 3.5″ drives each. They only just fit in the area. They didn't mount together, so I cabled tied them together and used cotton buds down the 'tube' bits (where the metal edge is bent back on itself) to keep them joined together.
When actually mounting the drives in these, I found that the drives don't mount in the same place - the 'higher' drive mounts further back. This was no good as I didn't have the room for that. So I ended up securing the bottom drive in each bay with a single screw (I forgot to scavenge screws), and then taping everything together to keep the upper drive in each bay in place.
This then only just squeezes in the area when SATA cables and power cables are attached. Airflow is provided by the fan on the rear of the backplane on one side of the drives, and a 90mm fan attached to the rear of the case on the other. I need to get another 90mm fan to cool the bottom 2 drives.
Note the 10TB drive in the above picture wasn't plugged in, it was just to prop up the drive bays and keep them in place - in actual use I use the case in 'desktop' orientation, so they are resting against the side (bottom) of the case). Also, this photo was from when I had them plugged into the motherboard SATA, not the RAID card.
After getting the placement of the drives in case sorted, I booted up the PC only to find Windows wouldn't start. I spent ages trying to fix this (it would boot into safe mode no problems but not normal windows). In the end I just went for the 'Reset' (Reinstall Windows and leave personal files) option. I have tons of software on the PC I'd need to reinstall, so that's why I spent a long time trying other options before I finally gave up and went for this option.
I left it reinstalling windows while I had lunch, and when I came to check on it, it was just stuck on a black screen. Doing a forced off / on, it said it was undoing changes, then after a bit came up with a message saying the install failed and to try again. Clicking OK on this, the machine rebooted and did the undoing changes thing. Then rebooted and did the same again. Then rebooted again, and Windows loaded, with everything as it was (or near enough) before it decided to break.
So the option that said it would wipe Windows and all installed software actually didn't do this at all, but it did fix it. That's Microsoft logic for you.
Once I'd undone some of the stuff I'd done in trying to fix Windows (disabling all non-essential services etc.) I tried setting up a storage space with the 4 drives. At first one of the drives was missing, I checked all the SATA connections and it seemed one was a bit loose. On reboot they all showed in BIOS, and upon boot to Windows, Storage Spaces could now see them as well. But when I tried to create the storage space I got the error Can't create the pool. Check drive connections - Request not supported 0x00000032
.
Searching for this error, I found the fix was simply to uninstall the drives from device manager, then scan for new hardware so they all get added again. Another case of Microsoft logic.
There didn't seem to be any options about number of columns, just the option to create with no mirrors, a single mirror, two mirrors, or parity. So I just chose the single mirror option - I presume this is equivalent to RAID 10 as it ended up with a 10Tb partition. I then ran Crystal DiskMark to get an idea of the speed.
I then destroyed the space and pool and ran Crystal DiskMark on a single drive.
The next stage was to set up the disks with hardware RAID 10 and benchmark that. I got everything connected, but on boot it would say 'F/W Initializing devices', get to 100%, then boot Windows. So how was I meant to get into the RAID card BIOS to set up the disks?
The card I'm using is an LSI 9267-8i MegaRAID card. Googling, I found it was Ctrl + H to get into the card BIOS. But multiple boot runs trying this, I couldn't get anything. I tried a PS2 keyboard in case it didn't like my wireless USB keyboard on a USB 3 port, still the same. Then I found another key combo - Ctrl + P to pause. Pressing this when it was saying 'F/W Initializing devices' it would then pause for about 5s after that got to 100% and would display 'Press Ctrl + H for BIOS'. But pressing Ctrl + H would just boot to Windows.
After more googling and messing with BIOS settings, I found you need to set CSM to BIOS and UEFI, and Interrupt 19 Capture must be turned on. Save the BIOS settings and reboot, press Ctrl + P when it gets to the LSI 'F/W Initializing devices' screen. Press Ctrl +h when prompted, then hammer the F8 key or whichever the boot menu key is for your system. Now choose the RAID card from the boot menu. Now you should finally be able to get into the LSI MegaRAID WebBIOS to set up your array.
However, I then had trouble setting up the RAID array. I found a youtube video that went through the process, and the bit I was missing was that when you create the 2nd drive pool, you must click the button to accept the pool. The button seems like it acts as 'add new pool', but it really is an 'accept pool' button, and if you don't accept the 2nd pool, then on the next step you only have the first pool to choose from and so can't set up the array properly.
When I was creating the array, I got the message that it would use write through as the BBU wasn't available or wasn't charged or was relearning. So I need still need to check on that. It does see the BBU and says it is operational.
These are the test results I got from Crystal Diskmark:
-----------------------------------------------------------------------
CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo
Crystal Dew World : https://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 215.092 MB/s
Sequential Write (Q= 32,T= 1) : 215.584 MB/s
Random Read 4KiB (Q= 8,T= 8) : 1.735 MB/s [ 423.6 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 1.957 MB/s [ 477.8 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 1.777 MB/s [ 433.8 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 1.953 MB/s [ 476.8 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 0.796 MB/s [ 194.3 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 2.077 MB/s [ 507.1 IOPS]
Test : 1024 MiB [G: 0.0% (0.2/4657.5 GiB)] (x5) [Interval=5 sec]
Date : 2019/11/02 13:55:07
OS : Windows 10 Professional [10.0 Build 18362] (x64)
-----------------------------------------------------------------------
CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo
Crystal Dew World : https://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 209.958 MB/s
Sequential Write (Q= 32,T= 1) : 209.387 MB/s
Random Read 4KiB (Q= 8,T= 8) : 6.867 MB/s [ 1676.5 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 2.684 MB/s [ 655.3 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 6.979 MB/s [ 1703.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 2.676 MB/s [ 653.3 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 1.118 MB/s [ 272.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 2.690 MB/s [ 656.7 IOPS]
Test : 1024 MiB [G: 0.0% (0.4/9297.9 GiB)] (x5) [Interval=5 sec]
Date : 2019/11/02 13:46:54
OS : Windows 10 Professional [10.0 Build 18362] (x64)
-----------------------------------------------------------------------
CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo
Crystal Dew World : https://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 553.287 MB/s
Sequential Write (Q= 32,T= 1) : 416.736 MB/s
Random Read 4KiB (Q= 8,T= 8) : 12.968 MB/s [ 3166.0 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 3.112 MB/s [ 759.8 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 12.566 MB/s [ 3067.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 2.724 MB/s [ 665.0 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 2.601 MB/s [ 635.0 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.530 MB/s [ 129.4 IOPS]
Test : 1024 MiB [G: 0.0% (0.4/9314.0 GiB)] (x5) [Interval=5 sec]
Date : 2019/11/02 15:26:59
OS : Windows 10 Professional [10.0 Build 18362] (x64)
Sequential read speed with RAID 10 is very good, similar to SATA SSD speed. Storage Spaces is better than a single drive for random reads and writes, but generally not as good as RAID 10. Random write Q1,T1 on RAID 10 is pretty slow though, I don't know why that would be?
