Fastest Windows drive pooling solution?

[TeeJayHoward]

It's been ages. I remember Storage Spaces being slow as molasses. ZFS has issues with RND4K and isn't "officially" on Windows yet (although it seems to work, there's little issues like the right-click menu not working). I have no idea if there are any modern RAID controllers that work across multiple PCIe slots, and if they'd be any faster than ZFS.

What's the go-to these days for speedy drive pooling? No need for redundancy. Just speed and capacity.

 

[_Gea]

Create a raid-0 alike Storage Space with

- number of Colums=number of disks in the Storage Pool
- Interleave according to workload, default is 256KB (for VMs smaller, AV larger)
- Allocation Unit Size, default is 4KB (slow), prefer a larger value ex 256KB

read for example
https://www.dell.com/support/manual...1af8c9-6b5c-4d8f-be9d-60908d9e418e&lang=en-us

 

[TeeJayHoward]

Create a raid-0 alike Storage Space with

- number of Colums=number of disks in the Storage Pool
- Interleave according to workload, default is 256KB (for VMs smaller, AV larger)
- Allocation Unit Size, default is 4KB (slow), prefer a larger value ex 256KB

read for example
https://www.dell.com/support/manual...1af8c9-6b5c-4d8f-be9d-60908d9e418e&lang=en-us

Holy moly is that terrible. Here's the same drives in a 96% full RaidZ:

Wiped that out, here's a fresh pool, no raidz:

And here's a simple Windows span:

And Windows stripe:

StableBit DrivePool:

Storage Spaces / Windows stuff ain't gonna work. Ceph messes up the entire machine as soon as you install the driver - can't boot Windows. ShareBit's odd - the 4K's almost there, but sequential's crap and it's a paid product anyway. There's just nothing that comes close to ZFS. Maybe I should see if CAD works under Linux yet.

 

[Nobu]

Why not zfs on a storage server, or in a VM that you access like a storage server? You can pass whole disks to a VM, and run the host from a small flash drive or something.

 

[TeeJayHoward]

Why not zfs on a storage server, or in a VM that you access like a storage server? You can pass whole disks to a VM, and run the host from a small flash drive or something.

Can HyperV pass whole disks? Might do that.

 

[Nobu]

Yeah, not sure if the free tools can set it up, but it's supported: https://learn.microsoft.com/en-us/p...ows-server-2012-r2-and-2012/dn803924(v=ws.11)

iirc, there's a way to manually set it up by editing the vm config with a text editor.

 

[Axman]

Why not a NAS?

 

[TeeJayHoward]

Why not a NAS?

This the local storage. Just want it fast. So fast, I have an excuse to buy a 100GbE link to my NAS. Trying out the Hyper-V whole disk thing right now. Seems to be working? fio is not CrystalDiskMark, so exact comparisons are... iffy.

Side note, just realized that if I add up all the disks in my rack I have just over half a petabyte of raw storage in spinning rust. I really should consolidate that into one big pool.

edit: Had a weird issue trying ZFSonWindows again which lead me to throw the drives in a VM instead. I was rsync-ing from the NAS to the local drives and the files that were coming over were being saved as the whole path, not just the file. So there was a file called /Applications/Microsoft/Office/Office.iso, which of course couldn't be viewed from Windows. Really weird thing is that it was stored in the /Applications/Microsoft/Office folder too.

 

[applegrcoug]

I use to use storage spaceson my sppinners, but ick it was slow. I am so glad I moved everything to truenas. I have a pool of spinners and then a pool of striped mirror sas ssds in a truenas VM. I passed through the entire hba. The spinners can saturate my 10Gbe link on large reads and the ssd pool more than saturates my 10Gbe link in both reads and writes.

Local speed is great, but practically speaking, I decided I'm not going to notice much difference between 10Gb and something faster. I use the network space for housing games.

The only time I don't like it is when I copy inside the pools from my client because it has to go from the nas, to my pc, back to the nas.

 

[pendragon1]

seeing lost of people suggesting using windows and just upping the aus to 512, and the interleave to 256.... if your just playing with empty drives right now, might be worth a try....

 

[Nobu]

Had a weird issue trying ZFSonWindows again which lead me to throw the drives in a VM instead. I was rsync-ing from the NAS to the local drives and the files that were coming over were being saved as the whole path, not just the file. So there was a file called /Applications/Microsoft/Office/Office.iso, which of course couldn't be viewed from Windows. Really weird thing is that it was stored in the /Applications/Microsoft/Office folder too

That is weird. I would have to do some research about that.

 

[MrGuvernment]

What is the final use case?

Do you trust Microsoft with that much storage space to manage and perform well?
How much CPU usage does it need...

I run a TrueNAS Scale box and also look for excuses to upgrade since I have a 4 NVMe VM drive over 10Gb I use from my workstation which I can max out that 10Gb link (looking to do 40Gb direct from desktop to truenas eventually..)

 

[twin_savage]

I have no idea if there are any modern RAID controllers that work across multiple PCIe slots, and if they'd be any faster than ZFS.

The 4300 series hardware raid does span across multiple PCIe slots but is NVMe only.
ZFS seems to cap out around 2.5GB/s peak read speeds on the very best systems with very large arrays once the RAM cache is exhausted, and after the pool has been used for awhile expect that number to shrink even more as the pool experiences fragmentation (assuming you are on spinners).

I've never been able to get Windows StorageSpaces to perform well with writes, but I've been using it for parity arrays.

 

[_Gea]

I've never been able to get Windows StorageSpaces to perform well with writes, but I've been using it for parity arrays.

A realtime diskbased raid array writes additional redundancy datablocks what is quite efficient. A Windows Storage Spaces Pool writes more or less whole files additionally to other disks for redundancy. This has advantages as you can pool disks of different size or type with hot/cold data auto tiering but is quite slow especially for parity Spaces. If you need performance, prefer mirror Spaces or Spaces without redundancy and backups.

 

[MrGuvernment]

The 4300 series hardware raid does span across multiple PCIe slots but is NVMe only.
ZFS seems to cap out around 2.5GB/s peak read speeds on the very best systems with very large arrays once the RAM cache is exhausted, and after the pool has been used for awhile expect that number to shrink even more as the pool experiences fragmentation (assuming you are on spinners).

I've never been able to get Windows StorageSpaces to perform well with writes, but I've been using it for parity arrays.

ZFS can do more than 2.5GB/s peak, it all depends on your array configuration as well as the CPU behind it, dedupe, compression, all sorts of settings..

 

[twin_savage]

ZFS can do more than 2.5GB/s peak, it all depends on your array configuration as well as the CPU behind it, dedupe, compression, all sorts of settings..

I should clarify I'm talking about unfragmented RAIDZ with as many VDEVs as is optimal among a reasonable amount of HDDs (<36) and not mirroring and no compression or dedupe; The bottleneck in ZFS read performance seems to be a product of the RMW loop being so far away from the HDDs/SSDs and the peak CPU performance we have today.
I see so many posts with shallow FIO testing data on their pool and then they wonder why significant or full backups take so long in practice.