Disappointing speed with ZFS
Alexey Tarasov
I am trying to use ZFS to store my torrent downloads. I noticed that
hashing in rtorrent works 10 times slower than the same disk with UFS.
I've done tuning as described in WIKI:
vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"
kern.maxvnodes=800000
My DMESG:
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-PRERELEASE #1: Wed Feb 6 15:39:32 MSK 2008
le...@alpha.lexasoft.ru:/usr/obj/usr/src/sys/ALPHA
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Genuine Intel(R) CPU 2160 @ 1.80GHz (1795.51-MHz K8-
class CPU)
Origin = "GenuineIntel" Id = 0x6f2 Stepping = 2
Features
=
0xbfebfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP
,MTRR
,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe39d<SSE3,RSVD2,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 4280516608 (4082 MB)
avail memory = 4125683712 (3934 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (Feb 6 2008 15:37:30)
acpi0: <PTLTD RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci9: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci9
pci10: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x4000-0x401f mem 0xe0200000-0xe021ffff irq 16 at device 0.0 on pci13
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:8f:78:ca
em0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci14: <ACPI PCI bus> on pcib5
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x5000-0x501f mem 0xe0300000-0xe031ffff irq 17 at device 0.0 on pci14
em1: Using MSI interrupt
em1: Ethernet address: 00:30:48:8f:78:cb
em1: [FILTER]
uhci0: <UHCI (generic) USB controller> port 0x3000-0x301f irq 23 at
device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3020-0x303f irq 19 at
device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 18 at
device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3060-0x307f irq 16 at
device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem
0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci15: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0x6000-0x60ff mem
0xe8000000-0xefffffff,0xe0400000-0xe040ffff irq 16 at device 0.0 on
pci15
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH7 SATA300 controller> port 0x30e8-0x30ef,
0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem
0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10
on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,
0xcc000-0xccfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 1.000 msec
hptrr: no controller detected.
acd0: CDROM <CD-224E-N/1.AA> at ata0-master UDMA33
ad4: 157066MB <WDC WD1600YD-01NVB1 10.02E01> at ata2-master SATA150
ad5: 381553MB <WDC WD4000AAJS-00TKA0 12.01C01> at ata2-slave SATA150
ad6: 157066MB <WDC WD1600YD-01NVB1 10.02E01> at ata3-master SATA150
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad4s1a
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6
KERNEL CONFIG is Generic with debug turned off and SCHED_ULE.
Could you please tell me where I am not right? Or what should I do to
make it faster?
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Ivan Voras
> Hello.
>
> I am trying to use ZFS to store my torrent downloads. I noticed that
> hashing in rtorrent works 10 times slower than the same disk with UFS.
I've done some extensive file system testing and here are my results
with bonnie++ for UFS+SU vs ZFS on AMD64, 6 GB RAM (1 GB for kmem), on a
RAID10 volume of 15 kRPM SAS drives:
UFS+SU: write: 109 MB/s, read: 111 MB/s, random file creation: 36500 f/s
ZFS: write: 95 MB/s, read: 180 MB/s (!!), random file creation: 40522 f/s
Read speed for ZFS seems too high to be valid, it's probably some cache
effects (though tests were done on a file more than twice the RAM size).
In any case, ordinary hashing should cause sequential reading, and these
seem really fast.
There could be one more thing: ZFS tries to write data sequentially,
like a log file system, and if the download was done in "parallel", many
pieces from different areas of the file at the same time (which is
normally the case for torrents), it might have gotten very fragmented on
the drive.
You can verify this by creating a similarily-sized ordinary file with dd
(the file should be large enough not to fit in the memory cache, or the
test should be done after a reboot) and then run iostat in one console
while reading the files (separately, one at a time, with dd or cat) in
another. A very fragmented file should have significantly higher tps count.
Alexey Tarasov
But why on this machine ZFS works slower than UFS? When I make UFS
file system on the same disk, rtorrent hashing works 10 times faster.
And while hashing, HDD is used three times intensively with ZFS
(noticed by flashing LED).
I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough
for ZFS?
What kernel tuning can help me?
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
_______________________________________________
Alexey Tarasov
`--# dd if=/dev/zero of=test.tmp bs=1024k count=10000
load: 0.42 cmd: dd 30773 [zfs:(&tx->tx_quiesce_done_cv)] 0.01u 3.56s
5% 1996k
2612+0 records in
2611+0 records out
2737831936 bytes transferred in 65.497687 secs (41800437 bytes/sec)
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 211.643188 secs (49544519 bytes/sec)
`--# dd if=test.tmp of=test2.tmp
(AD6 is my device)
tty ad4 ad5
ad6 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy
in id
1 238 0.00 0 0.00 60.22 18 1.04 64.00 982 61.37 1 0
31 2 67
1 3349 0.00 0 0.00 64.00 7 0.43 58.53 757 43.26 1 0
46 0 53
0 286 0.00 0 0.00 55.81 21 1.13 64.00 487 30.43 1 0
31 1 67
0 269 0.00 0 0.00 64.00 8 0.50 64.00 21 1.31 2 0
49 0 49
0 294 0.00 0 0.00 32.82 34 1.09 63.16 582 35.92 1 0
49 2 48
0 278 0.00 0 0.00 60.00 15 0.86 64.00 983 61.45 0 0
23 0 77
0 280 0.00 0 0.00 56.00 15 0.80 63.64 959 59.60 0 0
23 2 75
0 297 0.00 0 0.00 62.55 11 0.67 47.94 197 9.21 1 0
55 0 44
0 268 0.00 0 0.00 53.80 20 1.05 64.00 481 30.03 1 0
52 0 48
0 290 0.00 0 0.00 57.71 14 0.79 63.23 597 36.89 1 0
54 0 45
0 289 0.00 0 0.00 53.82 11 0.57 64.00 982 61.39 0 0
27 1 71
0 276 0.00 0 0.00 57.18 17 0.94 63.32 963 59.56 0 0
43 1 55
0 296 0.00 0 0.00 61.33 18 1.06 64.00 983 61.44 1 0
37 1 61
0 280 0.00 0 0.00 61.60 14 0.87 63.83 927 57.79 0 0
3 2 94
0 306 16.00 1 0.02 57.82 11 0.62 59.20 486 28.10 1 0
46 0 53
0 251 0.00 0 0.00 62.77 13 0.79 63.17 933 57.57 1 0
35 3 61
0 295 0.00 0 0.00 53.25 16 0.83 42.87 133 5.56 1 0
51 0 48
0 293 0.00 0 0.00 58.00 10 0.57 64.00 501 31.34 2 0
49 0 49
0 265 0.00 0 0.00 38.55 11 0.41 64.00 505 31.57 1 0
33 1 65
0 299 0.00 0 0.00 54.67 18 0.96 62.65 76 4.65 0 0
53 0 47
On 11.02.2008, at 20:21, Ivan Voras wrote:
> On 11/02/2008, Alexey Tarasov <m...@lexasoft.ru> wrote:
>> I've done similar tests on the other machine, and all looks fine.
>>
>> But why on this machine ZFS works slower than UFS? When I make UFS
>> file system on the same disk, rtorrent hashing works 10 times faster.
>> And while hashing, HDD is used three times intensively with ZFS
>> (noticed by flashing LED).
>>
>> I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough
>> for ZFS?
>>
>> What kernel tuning can help me?
>
> Ok, so you're saying that you've done similar testing (meaning
> rtorrent hashing) on other machines, and they work fine, but only this
> one is slow?
>
> There's no usual reason why would it be so.
>
> Can you try the suggestion I made (dd+iostat)?
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
_______________________________________________
Alexey Tarasov
On 11.02.2008, at 20:21, Ivan Voras wrote:
> On 11/02/2008, Alexey Tarasov <m...@lexasoft.ru> wrote:
>> I've done similar tests on the other machine, and all looks fine.
>>
>> But why on this machine ZFS works slower than UFS? When I make UFS
>> file system on the same disk, rtorrent hashing works 10 times faster.
>> And while hashing, HDD is used three times intensively with ZFS
>> (noticed by flashing LED).
>>
>> I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough
>> for ZFS?
>>
>> What kernel tuning can help me?
>
> Ok, so you're saying that you've done similar testing (meaning
> rtorrent hashing) on other machines, and they work fine, but only this
> one is slow?
No, I mean bonnie++ tests.
>
>
> There's no usual reason why would it be so.
>
> Can you try the suggestion I made (dd+iostat)?
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
_______________________________________________
Ivan Voras
> `--# dd if=test.tmp of=test2.tmp
>
> tty ad4 ad5 ad6 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 1 238 0.00 0 0.00 60.22 18 1.04 64.00 982 61.37 1 0 31 2 67
> 1 3349 0.00 0 0.00 64.00 7 0.43 58.53 757 43.26 1 0 46 0 53
> 0 286 0.00 0 0.00 55.81 21 1.13 64.00 487 30.43 1 0 31 1 67
> 0 269 0.00 0 0.00 64.00 8 0.50 64.00 21 1.31 2 0 49 0 49
Ok, now do the same with the file that rtorrent is hashing in the "if"
argument.
You should have used "bs=1m" for both tests, but it's too late now - do
the other test also without "bs".
Alexey Tarasov
4341952 -rw-r--r-- 1 lexa wheel 4,1G 7 фев 16:24 CentOS-5.1-
x86_64-bin-DVD.iso
`--# dd if=CentOS-5.1-x86_64-bin-DVD.iso of=test2
tty ad4 ad5
ad6 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy
in id
0 351 7.33 1 0.01 57.37 44 2.44 43.02 44 1.87 1 0
1 0 98
0 594 0.00 0 0.00 59.00 15 0.89 62.63 301 18.38 0 0
9 0 91
0 287 0.00 0 0.00 59.20 25 1.44 64.00 325 20.29 1 0
50 0 49
0 328 0.00 0 0.00 55.11 18 0.97 64.00 291 18.17 1 0
49 1 49
0 264 16.00 1 0.02 53.33 12 0.62 64.00 345 21.55 2 0
36 0 62
0 285 19.50 32 0.61 54.73 22 1.17 64.00 320 19.98 1 0
52 1 47
0 341 0.00 0 0.00 64.00 17 1.06 64.00 365 22.78 1 0
47 1 51
0 345 0.00 0 0.00 52.50 16 0.81 64.00 328 20.49 1 0
32 1 66
0 299 0.00 0 0.00 57.38 26 1.46 64.00 302 18.86 2 0
49 0 49
0 308 0.00 0 0.00 58.59 17 0.94 60.70 383 22.73 0 0
16 0 84
0 307 0.00 0 0.00 46.06 33 1.48 54.20 369 19.51 1 0
51 0 48
0 305 0.00 0 0.00 51.80 20 1.01 63.38 303 18.74 1 0
47 0 52
0 269 0.00 0 0.00 54.86 21 1.11 63.78 285 17.73 1 0
37 0 62
0 307 0.00 0 0.00 55.56 18 0.98 64.00 285 17.79 0 0
50 2 48
0 267 4.00 1 0.00 53.33 12 0.60 63.86 868 54.16 0 0
4 0 96
0 300 6.71 28 0.18 56.70 23 1.26 63.75 505 31.43 0 0
47 2 52
0 307 0.00 0 0.00 60.55 22 1.30 64.00 694 43.39 1 0
67 2 30
0 309 0.00 0 0.00 52.42 19 0.97 64.00 967 60.44 0 0
61 1 38
0 286 0.00 0 0.00 50.91 21 1.07 62.91 380 23.36 0 0
20 1 79
0 301 0.00 0 0.00 58.74 18 1.05 64.00 606 37.87 0 0
4 1 95
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
_______________________________________________
Ivan Voras
> ls -lash CentOS-5.1-x86_64-bin-DVD.iso
>
> 4341952 -rw-r--r-- 1 lexa wheel 4,1G 7 фев 16:24
> CentOS-5.1-x86_64-bin-DVD.iso
>
> `--# dd if=CentOS-5.1-x86_64-bin-DVD.iso of=test2
Ok, now try this one:
# dd if=CentOS-5.1-x86_64-bin-DVD.iso of=/dev/null bs=1m
>
> tty ad4 ad5 ad6 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 0 351 7.33 1 0.01 57.37 44 2.44 43.02 44 1.87 1 0 1 0 98
> 0 594 0.00 0 0.00 59.00 15 0.89 62.63 301 18.38 0 0 9 0 91
> 0 287 0.00 0 0.00 59.20 25 1.44 64.00 325 20.29 1 0 50 0 49
> On 11.02.2008, at 20:53, Ivan Voras wrote:
>
>> Alexey Tarasov wrote:
>>
>>> `--# dd if=test.tmp of=test2.tmp
>>
>>>
>>> tty ad4 ad5
>>> ad6 cpu
>>> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy
>>> in id
>>> 1 238 0.00 0 0.00 60.22 18 1.04 64.00 982 61.37 1 0 31
>>> 2 67
>>> 1 3349 0.00 0 0.00 64.00 7 0.43 58.53 757 43.26 1 0 46
>>> 0 53
>>> 0 286 0.00 0 0.00 55.81 21 1.13 64.00 487 30.43 1 0 31
>>> 1 67
>>> 0 269 0.00 0 0.00 64.00 8 0.50 64.00 21 1.31 2 0 49
>>> 0 49
>>
>> Ok, now do the same with the file that rtorrent is hashing in the "if"
>> argument.
_______________________________________________
Alexey Tarasov
ad6 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy
in id
1 0 98
0 606 0.00 0 0.00 58.62 31 1.76 63.02 246 15.14 0 0
3 0 97
2 5334 0.00 0 0.00 57.14 7 0.37 64.00 355 22.20 0 0
2 0 97
2 3194 0.00 0 0.00 0.00 0 0.00 63.35 295 18.26 0 0
2 1 97
0 148 0.00 0 0.00 0.00 0 0.00 63.90 336 20.97 0 0
0 1 99
0 453 0.00 0 0.00 55.87 87 4.72 64.00 290 18.14 1 0
3 0 96
0 339 0.00 0 0.00 60.22 35 2.04 61.31 342 20.49 0 0
3 0 96
0 311 0.00 0 0.00 60.00 24 1.41 61.48 347 20.84 0 0
3 0 96
0 317 0.00 0 0.00 50.37 26 1.28 64.00 327 20.42 0 0
3 0 97
0 488 0.00 0 0.00 54.05 37 1.98 64.00 284 17.78 1 0
2 0 97
0 309 0.00 0 0.00 54.21 28 1.47 63.22 312 19.27 0 0
3 0 97
0 303 16.00 14 0.23 37.51 51 1.87 64.00 334 20.85 1 0
1 1 97
0 297 0.00 0 0.00 57.89 35 1.96 63.78 345 21.49 1 0
3 0 97
0 287 0.00 0 0.00 51.86 28 1.41 64.00 325 20.34 0 0
2 0 98
0 319 0.00 0 0.00 60.52 30 1.76 63.24 299 18.46 0 0
4 1 94
0 277 0.00 0 0.00 63.83 22 1.38 64.00 274 17.11 1 0
2 1 97
0 298 0.00 0 0.00 55.29 16 0.88 63.21 286 17.63 0 0
2 0 97
0 338 12.12 31 0.36 57.65 33 1.84 62.08 299 18.13 1 0
4 0 95
0 306 2.00 1 0.00 58.55 21 1.21 64.00 332 20.73 2 0
2 1 95
0 269 0.00 0 0.00 59.68 24 1.40 64.00 304 18.98 1 0
2 0 96
On 11.02.2008, at 21:12, Ivan Voras wrote:
> Alexey Tarasov wrote:
>> ls -lash CentOS-5.1-x86_64-bin-DVD.iso
>>
>> 4341952 -rw-r--r-- 1 lexa wheel 4,1G 7 фев 16:24
>> CentOS-5.1-x86_64-bin-DVD.iso
>>
>> `--# dd if=CentOS-5.1-x86_64-bin-DVD.iso of=test2
>
> Ok, now try this one:
> # dd if=CentOS-5.1-x86_64-bin-DVD.iso of=/dev/null bs=1m
>
>>
>> tty ad4 ad5
>> ad6 cpu
>> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni
>> sy in id
>> 0 351 7.33 1 0.01 57.37 44 2.44 43.02 44 1.87 1 0
>> 1 0 98
>> 0 594 0.00 0 0.00 59.00 15 0.89 62.63 301 18.38 0 0
>> 9 0 91
>> 0 287 0.00 0 0.00 59.20 25 1.44 64.00 325 20.29 1 0
>> 50 0 49
>
>> On 11.02.2008, at 20:53, Ivan Voras wrote:
>>
>>> Alexey Tarasov wrote:
>>>
>>>> `--# dd if=test.tmp of=test2.tmp
>>>
>>>>
>>>> tty ad4 ad5
>>>> ad6 cpu
>>>> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni
>>>> sy
>>>> in id
>>>> 1 238 0.00 0 0.00 60.22 18 1.04 64.00 982 61.37 1 0 31
>>>> 2 67
>>>> 1 3349 0.00 0 0.00 64.00 7 0.43 58.53 757 43.26 1 0 46
>>>> 0 53
>>>> 0 286 0.00 0 0.00 55.81 21 1.13 64.00 487 30.43 1 0 31
>>>> 1 67
>>>> 0 269 0.00 0 0.00 64.00 8 0.50 64.00 21 1.31 2 0 49
>>>> 0 49
>>>
>>> Ok, now do the same with the file that rtorrent is hashing in the
>>> "if"
>>> argument.
>
Kris Kennaway
> I've done similar tests on the other machine, and all looks fine.
>
> But why on this machine ZFS works slower than UFS? When I make UFS file
> system on the same disk, rtorrent hashing works 10 times faster. And
> while hashing, HDD is used three times intensively with ZFS (noticed by
> flashing LED).
>
> I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough for
> ZFS?
>
> What kernel tuning can help me?
I'd guess this is just related to the ZFS design. As Ivan says, it
prefers to do all writes sequentially. This means that reads (as with
reading of hashes) may be very fragmented and require lots of drive
seeking, which will reduce performance a lot. ZFS does do aggressive
prefetching of data to try and offset this problem, but if your disk
bandwidth is low (e.g. you are not using a fast disk array) then it may
not help much (and can also introduce big I/O latency for other operations).
As for what can be done about this, I don't know, but you should look
into the general ZFS literature (ZFS support mailing lists, etc).
Kris
Niki Denev
> I've done similar tests on the other machine, and all looks fine.
>
> But why on this machine ZFS works slower than UFS? When I make UFS
> file system on the same disk, rtorrent hashing works 10 times faster.
> And while hashing, HDD is used three times intensively with ZFS
> (noticed by flashing LED).
>
> I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough
> for ZFS?
>
> What kernel tuning can help me?
>
>
What happens when you disable ZFS prefetching by putting :
vfs.zfs.prefetch_disable=1
in your /boot/loader.conf?
--Niki
Joao Barros
> On Feb 11, 2008 7:05 PM, Alexey Tarasov <m...@lexasoft.ru> wrote:
> > I've done similar tests on the other machine, and all looks fine.
> >
> > But why on this machine ZFS works slower than UFS? When I make UFS
> > file system on the same disk, rtorrent hashing works 10 times faster.
> > And while hashing, HDD is used three times intensively with ZFS
> > (noticed by flashing LED).
> >
> > I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough
> > for ZFS?
> >
> > What kernel tuning can help me?
> >
> >
>
> What happens when you disable ZFS prefetching by putting :
> vfs.zfs.prefetch_disable=1
> in your /boot/loader.conf?
>
This would be my recommendation as well.
I had to disable prefetch to be able to play video files(1.1GB
~43mins) from a 4 disk raidz without jitter. I can even play 2 video
files concurrently now :)
Try and show us the results.
--
Joao Barros
Ivan Voras
> tty ad4 ad5 ad6 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 0 606 0.00 0 0.00 58.62 31 1.76 63.02 246 15.14 0 0 3 0 97
> 2 5334 0.00 0 0.00 57.14 7 0.37 64.00 355 22.20 0 0 2 0 97
> 2 3194 0.00 0 0.00 0.00 0 0.00 63.35 295 18.26 0 0 2 1 97
> 0 148 0.00 0 0.00 0.00 0 0.00 63.90 336 20.97 0 0 0 1 99
Sorry, can't help you. I was expecting to see a lot of small
transactions, indicating seeking across the file system, but the numbers
seem normal enough.
Is the drive & ZFS topology "normal" - you don't perhaps have a zpool
made of multiple partitions/slices on the same drive? Can you send the
output of "zpool status"?
Alexey Tarasov
pool: storage
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
ad6 ONLINE 0 0 0
errors: No known data errors
--
Alexey Tarasov
On 12.02.2008, at 14:59, Asbjørn Clemmensen wrote:
> I've never experienced that error, but when it happens, how much
> free memory
> does the system have? Are you running FreeBSD as i386 or amd64? I
> had to
> switch to amd64 because performance was extremely poor running with
> a i386
> system.
>
> - Asbjørn
>
> On Tue, Feb 12, 2008 at 01:36:39PM +0300, Alexey Tarasov wrote:
>> Hello.
>>
>> Thanks, hashing works good with this options.
>>
>> Another issue. Sometimes I get the following error in rtorrent:
>>
>> Storage error: [File chunk write error: Cannot allocate memory.]
>>
>> Is it an issue of ZFS or Rtorrent bug?
>>
>> On 12.02.2008, at 8:47, Asbjørn Clemmensen wrote:
>>
>>> On Mon, Feb 11, 2008 at 03:49:40PM +0300, Alexey Tarasov wrote:
>>>> Hello.
>>>>
>>>> I am trying to use ZFS to store my torrent downloads. I noticed
>>>> that hashing in rtorrent works 10 times slower than the same disk
>>>> with UFS.
>>>
>>> You might want to try changing rtorrent's settings for hashing
>>> files. I had
>>> the same problem when switching to ZFS - files hashed at < 1mb/
>>> sec. I ended up
>>> using these settings:
>>>
>>> hash_read_ahead = 32
>>> hash_max_tries = 1
>>> hash_interval = 1
>>>
>>> This way the drives are utilized much more, and same goes for the
>>> CPU. Now I'm
>>> getting 20-30 mb/sec when hashing.
>>>
>>> (7.0-PRERELEASE amd64, Athlon64 3800+ X2, 3 Gb of RAM)
>>>
>>>>
>>>> I've done tuning as described in WIKI:
>>>>
>>>> vm.kmem_size_max="1073741824"
>>>> vm.kmem_size="1073741824"
>>>>
>>>> kern.maxvnodes=800000
>>>>
>>>> My DMESG:
>>>>
>>>> Copyright (c) 1992-2008 The FreeBSD Project.
>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992,
>>>> 1993, 1994
>>>> The Regents of the University of California. All rights
>>>> reserved.
>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>> FreeBSD 7.0-PRERELEASE #1: Wed Feb 6 15:39:32 MSK 2008
>>>> le...@alpha.lexasoft.ru:/usr/obj/usr/src/sys/ALPHA
>>>> Timecounter "i8254" frequency 1193182 Hz quality 0
>>>> CPU: Genuine Intel(R) CPU 2160 @ 1.80GHz (1795.51-MHz
>>>> K8-class CPU)
>>> - Asbjørn Clemmensen
Pawel Jakub Dawidek
> Alexey Tarasov wrote:
> >I've done similar tests on the other machine, and all looks fine.
> >
> >But why on this machine ZFS works slower than UFS? When I make UFS file
> >system on the same disk, rtorrent hashing works 10 times faster. And
> >while hashing, HDD is used three times intensively with ZFS (noticed by
> >flashing LED).
> >
> >I have an amd64 Core2Duo processor, 4 Gb of RAM, what is not enough for
> >ZFS?
> >
> >What kernel tuning can help me?
>
> I'd guess this is just related to the ZFS design. As Ivan says, it
> prefers to do all writes sequentially. This means that reads (as with
> reading of hashes) may be very fragmented and require lots of drive
> seeking, which will reduce performance a lot. ZFS does do aggressive
> prefetching of data to try and offset this problem, but if your disk
> bandwidth is low (e.g. you are not using a fast disk array) then it may
> not help much (and can also introduce big I/O latency for other operations).
>
> As for what can be done about this, I don't know, but you should look
> into the general ZFS literature (ZFS support mailing lists, etc).
It may also be because ZFS send FLUSH requests to the disk quite often.
If disk or controller handles those requests slowly, it may case
performance drop when comparing to UFS, which doesn't send FLUSH
requests at all. To verify this, one should try disable FLUSH requests
sending by adding:
vfs.zfs.cache_flush_disable=1
to /boot/loader.conf.
--
Pawel Jakub Dawidek http://www.wheel.pl
p...@FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!