So new Windows version available in 48 hours or so, so I decided to dig a little deeper, before clean install, also Contacted AMD support, to find out what they have to say if anything.
The web UI raidxpert2 shows the PCIe Nvme (not m.2) in array 2 available options are JOBD, raid0 and raid1, while the BIOS version doesn't list the drive but the array 2 was visible (offline).
One can delete the array, the drive still shows in the raidxpert2 as 2.1 (0.1 and 1.0 for the M.2 raid), and windows don't detect.
Now windows won't detect any of the drives because at least the AMD bottom raid drivers screws things over, took a while to figure out why I can't get the drivers to show up when Nvme Raid is disabled in BIOS, tried installing/uninstalling amd driver packages (clearly not uninstalling everything). Had to manually set to generic drivers for them to show up. First AMD NVMe raid drivers were installed during Windows install.
So since raid is disabled the partition table "vanished" on the drive, but once the AMD Nvme raid drivers were changed testdisk did manage to find the partition table.
little more testing...
Windows 10 install on the NVME not in the NVMe Raid.
It seems one can use the NVMe drives own controller drivers over the AMD bottom device driver (which are needed for raid 0/1/10) , but the drive still gets placed under the drivexprert2 array as legacy/offline. Trying to install the AMD drive package (latest) blue screens windows … if OS is installed on the NVMe. The driver package install won't run if not all NVMe drives are using the bottom device driver (complains about OS being on NVMe). Even windows installation will crash after installing files with only AMD NVMe drivers loaded, array 2 will be offline+failed in BIOS.
With raidxpert2 installed manually, changing the controller back to Intel driver makes the legacy raid go offline and then the array can be removed. The max latency seemed to drop a little, but still it's under some nvme raid layer, since intel toolkit won't work. Somehow being OS drive and using the amd bottom device driver for the lone intel NVMe causes issues, no such issue before when OS was on old SSD. Now computer got stuck on reboots when using the amd driver.
winsat disk test (Intel 900p)
(not OS drive)
NVMe raid disabled with Intel controller driver.
Average Read Time with Sequential Writes 0.021 ms
Latency: 95th Percentile 0.038 ms
Latency: Maximum 0.104 ms
Average Read Time with Random Writes 0.021 ms
(OS drive)
NVMe raid enabled with Intel controller driver.
Average Read Time with Sequential Writes 0.022 ms
Latency: 95th Percentile 0.040 ms
Latency: Maximum 0.129 ms
Average Read Time with Random Writes 0.022 ms
AMD bottom driver (OS)
Average Read Time with Sequential Writes 0.052 ms
Latency: 95th Percentile 0.064 ms
Latency: Maximum 31.889 ms
Average Read Time with Random Writes 0.058 ms
...oh and 16.0 random read is 600MB/s slower than the 2 previous ones.
Crystaldiskmark 4KiB Q8T8 and Q1T1 seems to halved with bottom device driver, seems to be driver cap since the NVMe RAID get same results, apart from Q1T1 which is worse on the RAID.