Low IOPs for certain block sizes
Bart Van Assche
While running iSCSI performance tests I noticed that the performance
for certain block sizes deviated significantly (more than ten times)
from the performance for other block sizes, both larger and smaller.
This surprised me.
The test I ran was as follows:
* A file of 1 GB residing on a tmpfs filesystem was exported via iSCSI
target software. The test has been repeated with both SCST and STGT.
* On the initiator system open-iscsi version 2.0.870 was used for
performing reads and writes with dd via direct I/O. Read-ahead was set
to zero.
* Both systems were running kernel 2.6.29.1 in run level 3 (no X
server) and the 1 GbE interfaces in the two systems were connected via
a crossed cable. The MTU has been left to its default value, 1500
bytes. Netperf reported a throughput of 600 Mbit/s = 75 MB/s for the
TCP/IP stream test on this setup.
* 128 MB of data has been transferred during each test.
* Each measurement has been repeated three times.
* All caches were flushed before each test.
* The ratio of standard deviation to average was 2% or lower for all
measurements.
* The measurement result are as follows (transfer speeds in MB/s):
Block SCST STGT SCST STGT
size writing writing reading reading
------ ------- ------- ------- -------
64 MB 71.7 63.3 62.1 58.4
32 MB 71.9 63.4 61.7 58.1
16 MB 72.4 63.0 61.7 57.1
8 MB 72.7 63.3 61.7 56.9
4 MB 72.9 63.5 61.3 57.0
2 MB 72.8 59.5 60.3 56.9
1 MB 72.1 38.7 59.4 56.0
512 KB 67.3 21.4 58.0 54.4
256 KB 67.4 22.8 55.5 53.4
128 KB 60.9 22.6 53.3 51.7
64 KB 53.2 22.2 53.0 45.7
32 KB 48.9 21.6 40.0 40.0
16 KB 40.0 20.8 0.6 1.3
8 KB 20.0 19.9 19.9 20.0
4 KB 0.6 1.6 18.9 10.3
All results look normal to me, except the write throughput for a block
size of 4 KB and the read throughput for a block size of 16 KB.
Regarding CPU load: during the 4 KB write test, the CPU load was 0.9
on the initiator system and 0.1 on the target.
Has anyone observed similar behavior before ?
Bart.
Vladislav Bolkhovitin
disabled on both hosts. You can do that by using "ethtool -c"
Bart Van Assche
> I would suggest you to make sure you have any hardware coalescing disabled
> on both hosts. You can do that by using "ethtool -c"
The output of ethtool is as follows:
(target)
$ ethtool -i eth0
driver: sky2
version: 1.22
firmware-version: N/A
bus-info: 0000:02:00.0
$ ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 100
rx-frames: 16
rx-usecs-irq: 20
rx-frames-irq: 16
tx-usecs: 1000
tx-frames: 10
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
(initiator)
$ ethtool -i eth0
driver: atl1
version: 2.1.3
firmware-version: N/A
bus-info: 0000:01:00.0
$ ethtool -c eth0
Coalesce parameters for eth0:
Cannot get device coalesce settings: Operation not supported
Bart.
Bart Van Assche
By this time I have repeated the above test on the same two servers
but this time using a DDR IB network using IPoIB in connected mode
instead of a GbE network (netperf reports 1375 MB/s). The results for
the IB network look normal to me. But it's still not clear to me why
for certain block sizes the number of IOPs was so low on the GbE
network.
Block SCST STGT SCST STGT
size writing writing reading reading
------ ------- ------- ------- -------
64 MB 307 251 658 439
32 MB 460 258 630 431
16 MB 580 258 598 414
8 MB 585 262 566 413
4 MB 665 273 613 428
2 MB 737 240 645 476
1 MB 727 172 631 457
512 KB 623 115 583 326
256 KB 619 120 559 399
128 KB 467 118 499 306
64 KB 309 115 333 251
32 KB 233 107 248 198
16 KB 157 95 162 133
8 KB 97 79 95 78
4 KB 51 42 52 42
Bart.