SLQIO tool differing output with multiple threads vs async requests

Hello,

This is on Windows 2008R2 server, i'm trying to benchmark my SAN, on NTFS and on raw device directly.
I see huge difference in throughput between " -t32 -o1" Vs "-t1 -o32", later being significantly faster. Such difference in IOPs is not seen with raw device directly.

Is there any foul play on NTFS part here?

Thanks,
Shekhar

<<<<<<<<<<<<<
NTFS numbers::

C:\Program Files (x86)\SQLIO>sqlio.exe -kR -t32 -frandom -s300 -b2 -BN -LS -o1 -Fparam.txt
sqlio v1.5.SG
using system counter for latency timings, 2604199 counts per second
parameter file used: param.txt
file F:\Testfile131072.dat with 1 thread (0) using mask 0x0 (0)
1 thread reading for 300 secs from file F:\Testfile131072.dat
using 2KB random IOs
enabling multiple I/Os per thread with 1 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using specified size: 131072 MB for file: F:\Testfile131072.dat
initialization done
interrupted by Ctrl-C - stopping I/Os now
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 238.13
MBs/sec: 0.46
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 3
Max_Latency(ms): 228
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 4 19 24 25 21 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

C:\Program Files (x86)\SQLIO>sqlio.exe -kR -t1 -frandom -s300 -b2 -BN -LS -o32 -Fparam.txt
sqlio v1.5.SG
using system counter for latency timings, 2604199 counts per second
parameter file used: param.txt
file F:\Testfile131072.dat with 1 thread (0) using mask 0x0 (0)
1 thread reading for 300 secs from file F:\Testfile131072.dat
using 2KB random IOs
enabling multiple I/Os per thread with 32 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using specified size: 131072 MB for file: F:\Testfile131072.dat
initialization done
interrupted by Ctrl-C - stopping I/Os now
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 2883.33
MBs/sec: 5.63
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 10
Max_Latency(ms): 144
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 1 7 10 11 10 7 6 5 5 4 3 3 3 2 2 2 2 1 1 1 1 1 1 10
<<<<<<<<<<<<<<<<<<
Raw device numbers:::
C:\Program Files (x86)\SQLIO>sqlio.exe -kR -t32 -frandom -s300 -b2 -BN -LS -o1 -R6
sqlio v1.5.SG
using system counter for latency timings, 2604199 counts per second
32 threads reading for 300 secs from file 6:
using 2KB random IOs
enabling multiple I/Os per thread with 1 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using current size: 1126393 MB for file: 6:
initialization done
interrupted by Ctrl-C - stopping I/Os now
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1865.08
MBs/sec: 3.64
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 16
Max_Latency(ms): 441
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 1 4 6 7 7 6 5 5 4 4 4 3 3 3 3 2 2 2 2 2 2 2 21

C:\Program Files (x86)\SQLIO>sqlio.exe -kR -t32 -frandom -s300 -b2 -BN -LS -o1 -R6
sqlio v1.5.SG
using system counter for latency timings, 2604199 counts per second
32 threads reading for 300 secs from file 6:
using 2KB random IOs
enabling multiple I/Os per thread with 1 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using current size: 1126393 MB for file: 6:
initialization done
interrupted by Ctrl-C - stopping I/Os now
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1865.08
MBs/sec: 3.64
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 16
Max_Latency(ms): 441
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 1 4 6 7 7 6 5 5 4 4 4 3 3 3 3 2 2 2 2 2 2 2 21
<<<<<<<<<<<<<

OOPs the title should be read "SQLIO" and NOT "SLQIO" .. my bad...