[Lustre-discuss] Slow NFS/CIFS when exporting Lustre
183 views
Skip to first unread message
Dan
Sep 22, 2011, 11:31:47 AM9/22/11
to lustre-...@lists.lustre.org
Hi,
After researching this in the archives it seems 85% of Lustre's native performance is reasonable via NFS. Over GigE I'm seeing 114 MB/sec read/write via Lustre native. NFS writes at 44 MB/sec via NFS (about 38%) but reads at 95 MB/sec. CIFS is lucky to see 18 MB/sec writes after tuning Samba (12 MB/sec prior). I've tried tuning the Lustre client and NFS client and server but nothing impacts the performance except setting noatime on NFS, which gets another 1.5 MB/sec on writes. What are some of you seeing in this configuration? Suggestions?
My configuration is a separate MDS and OSS with one OST running Lustre 1.8.6 on GigE only. Testing is currently simple using dd. Obdfilter-survey results below (single Areca 1680 w/ 10 3TB disks in RAID 6, no stripe/stride tuning or external journal, using the stock kernel driver w/ latest firmware):
Wed Sep 21 20:07:02 PDT 2011 Obdfilter-survey for case=disk from localhost.localdomain
ost 1 sz 16777216K rsz 1024K obj 1 thr 1 write 183.91 [ 126.86, 241.73] rewrite 202.90 [ 163.57, 241.73] read 753.34 [ 152.83,3440.19]
ost 1 sz 16777216K rsz 1024K obj 1 thr 2 write 384.32 [ 255.70, 486.45] rewrite 516.70 [ 290.68, 598.34] read 1570.43 [ 376.59,3589.96]
ost 1 sz 16777216K rsz 1024K obj 1 thr 4 write 415.34 [ 332.63, 488.45] rewrite 653.74 [ 38.96, 877.01] read 1915.92 [ 554.28,4865.86]
ost 1 sz 16777216K rsz 1024K obj 1 thr 8 write 401.46 [ 305.66, 541.39] rewrite 664.63 [ 383.57,1030.43] read 2081.41 [ 803.67,5404.30]
ost 1 sz 16777216K rsz 1024K obj 1 thr 16 write 406.12 [ 290.68, 541.39] rewrite 700.57 [ 0.00,1112.77] read 1563.54 [ 63.93,1936.96]
ost 1 sz 16777216K rsz 1024K obj 2 thr 2 write 373.83 [ 293.66, 458.50] rewrite 569.01 [ 60.99, 806.09] read 1380.53 [ 307.19,3737.88]
ost 1 sz 16777216K rsz 1024K obj 2 thr 4 write 443.58 [ 351.64, 517.43] rewrite 591.88 [ 334.62, 852.04] read 1633.55 [ 7.99,4560.89]
ost 1 sz 16777216K rsz 1024K obj 2 thr 8 write 459.65 [ 374.58, 532.95] rewrite 602.29 [ 358.60,1039.84] read 1416.61 [ 37.96,3275.32]
ost 1 sz 16777216K rsz 1024K obj 2 thr 16 write 461.84 [ 366.59, 517.42] rewrite 663.96 [ 262.71,1056.77] read 1366.43 [ 51.94,2379.27]
ost 1 sz 16777216K rsz 1024K obj 4 thr 4 write 434.83 [ 281.67, 518.41] rewrite 561.85 [ 326.63, 859.03] read 1225.48 [ 4.99,3352.31]
ost 1 sz 16777216K rsz 1024K obj 4 thr 8 write 481.20 [ 378.57, 566.37] rewrite 602.96 [ 143.84, 989.18] read 1478.49 [ 14.98,4548.44]
ost 1 sz 16777216K rsz 1024K obj 4 thr 16 write 493.07 [ 373.58, 583.34] rewrite 628.67 [ 304.66,1044.05] read 1475.33 [ 57.93,3221.92]
ost 1 sz 16777216K rsz 1024K obj 8 thr 8 write 436.75 [ 4.00, 535.33] rewrite 556.52 [ 249.72,1003.87] read 1473.04 [ 20.98,3419.10]
ost 1 sz 16777216K rsz 1024K obj 8 thr 16 write 483.68 [ 383.57, 588.34] rewrite 627.79 [ 226.74,1194.59] read 1490.97 [ 41.95,4202.74]
ost 1 sz 16777216K rsz 1024K obj 16 thr 16 write 469.65 [ 377.57, 552.84] rewrite 544.70 [ 257.71,1216.66] read 1375.23 [ 31.96,1133.43]
done!
Thank you,
Dan
After researching this in the archives it seems 85% of Lustre's native performance is reasonable via NFS. Over GigE I'm seeing 114 MB/sec read/write via Lustre native. NFS writes at 44 MB/sec via NFS (about 38%) but reads at 95 MB/sec. CIFS is lucky to see 18 MB/sec writes after tuning Samba (12 MB/sec prior). I've tried tuning the Lustre client and NFS client and server but nothing impacts the performance except setting noatime on NFS, which gets another 1.5 MB/sec on writes. What are some of you seeing in this configuration? Suggestions?
My configuration is a separate MDS and OSS with one OST running Lustre 1.8.6 on GigE only. Testing is currently simple using dd. Obdfilter-survey results below (single Areca 1680 w/ 10 3TB disks in RAID 6, no stripe/stride tuning or external journal, using the stock kernel driver w/ latest firmware):
Wed Sep 21 20:07:02 PDT 2011 Obdfilter-survey for case=disk from localhost.localdomain
ost 1 sz 16777216K rsz 1024K obj 1 thr 1 write 183.91 [ 126.86, 241.73] rewrite 202.90 [ 163.57, 241.73] read 753.34 [ 152.83,3440.19]
ost 1 sz 16777216K rsz 1024K obj 1 thr 2 write 384.32 [ 255.70, 486.45] rewrite 516.70 [ 290.68, 598.34] read 1570.43 [ 376.59,3589.96]
ost 1 sz 16777216K rsz 1024K obj 1 thr 4 write 415.34 [ 332.63, 488.45] rewrite 653.74 [ 38.96, 877.01] read 1915.92 [ 554.28,4865.86]
ost 1 sz 16777216K rsz 1024K obj 1 thr 8 write 401.46 [ 305.66, 541.39] rewrite 664.63 [ 383.57,1030.43] read 2081.41 [ 803.67,5404.30]
ost 1 sz 16777216K rsz 1024K obj 1 thr 16 write 406.12 [ 290.68, 541.39] rewrite 700.57 [ 0.00,1112.77] read 1563.54 [ 63.93,1936.96]
ost 1 sz 16777216K rsz 1024K obj 2 thr 2 write 373.83 [ 293.66, 458.50] rewrite 569.01 [ 60.99, 806.09] read 1380.53 [ 307.19,3737.88]
ost 1 sz 16777216K rsz 1024K obj 2 thr 4 write 443.58 [ 351.64, 517.43] rewrite 591.88 [ 334.62, 852.04] read 1633.55 [ 7.99,4560.89]
ost 1 sz 16777216K rsz 1024K obj 2 thr 8 write 459.65 [ 374.58, 532.95] rewrite 602.29 [ 358.60,1039.84] read 1416.61 [ 37.96,3275.32]
ost 1 sz 16777216K rsz 1024K obj 2 thr 16 write 461.84 [ 366.59, 517.42] rewrite 663.96 [ 262.71,1056.77] read 1366.43 [ 51.94,2379.27]
ost 1 sz 16777216K rsz 1024K obj 4 thr 4 write 434.83 [ 281.67, 518.41] rewrite 561.85 [ 326.63, 859.03] read 1225.48 [ 4.99,3352.31]
ost 1 sz 16777216K rsz 1024K obj 4 thr 8 write 481.20 [ 378.57, 566.37] rewrite 602.96 [ 143.84, 989.18] read 1478.49 [ 14.98,4548.44]
ost 1 sz 16777216K rsz 1024K obj 4 thr 16 write 493.07 [ 373.58, 583.34] rewrite 628.67 [ 304.66,1044.05] read 1475.33 [ 57.93,3221.92]
ost 1 sz 16777216K rsz 1024K obj 8 thr 8 write 436.75 [ 4.00, 535.33] rewrite 556.52 [ 249.72,1003.87] read 1473.04 [ 20.98,3419.10]
ost 1 sz 16777216K rsz 1024K obj 8 thr 16 write 483.68 [ 383.57, 588.34] rewrite 627.79 [ 226.74,1194.59] read 1490.97 [ 41.95,4202.74]
ost 1 sz 16777216K rsz 1024K obj 16 thr 16 write 469.65 [ 377.57, 552.84] rewrite 544.70 [ 257.71,1216.66] read 1375.23 [ 31.96,1133.43]
done!
Thank you,
Dan
Oleg Drokin
Sep 26, 2011, 8:06:58 AM9/26/11
to Dan, lustre-...@lists.lustre.org
Hello!
On Sep 22, 2011, at 11:31 AM, Dan wrote:
> After researching this in the archives it seems 85% of Lustre's native performance is reasonable via NFS. Over GigE I'm seeing 114 MB/sec read/write via Lustre native. NFS writes at 44 MB/sec via NFS (about 38%) but reads at 95 MB/sec. CIFS is lucky to see 18 MB/sec writes after tuning Samba (12 MB/sec prior). I've tried tuning the Lustre client and NFS client and server but nothing impacts the performance except setting noatime on NFS, which gets another 1.5 MB/sec on writes. What are some of you seeing in this configuration? Suggestions?
One issue is likely frequent attempts to use xattrs (there is a patch to speed this up in recent Lustre, LU-144 I think it is) and flocks.
Also I heard to cifs only does small IO - 64kb packets over the wire (and corresponding small reads), if you have high latency link, that would
probably have pretty negative impact too.
Bye,
Oleg
--
Oleg Drokin
Senior Software Engineer
Whamcloud, Inc.
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
Dan
Oct 6, 2011, 1:38:07 PM10/6/11
to Oleg Drokin, lustre-...@lists.lustre.org
Hi Oleg,
Thank your for your suggestions. I've upgraded from GigE to IB and moved from Lustre 1.8.6 to 2.1 on my test cluster. This includes the LU-144 fix for xattrs now but doesn't seem to help. I've eliminated Windows for now and I'm using a RHEL 5.7 machine as a CIFS client since Windows 7 network tuning has its own challenges.
Typical write speed is 44 MB/sec but from time to time it doubles to the low 90 MB/sec then slows back down (see collectl logs from Lustre client/Samba export server below). This cycle can happen twice during a 10 GB write test for no reason I can currently see. I've also noticed that when I break out of a write using dd via ctrl-c the performance jumps to 96 MB/sec until a cache somewhere runs out. This also happens right before the end of a write - the last few hundred MB write at full speed (seen at the end of the collectl log below).
I've tried tuning vfs_cache_pressure, max_rpcs_in_flight and max_dirty_mb on the Lustre client exporting Samba. Tweaking cache pressure seems to make no difference but setting max_rpcs and max_dirty to 256 seems optimal. FWIW - NFS performance is rock solid at 100 MB/sec, what else might I try tuning?
Thank you,
Dan
[root@test2 ~]# obdfilter-survey
Wed Oct 5 14:06:50 PDT 2011 Obdfilter-survey for case=disk from test2
ost 1 sz 16777216K rsz 1024K obj 1 thr 1 write 216.00 [ 101.89, 301.66] rewrite 285.53 [ 23.97, 400.56] read 194.76 [ 94.82, 249.72]
ost 1 sz 16777216K rsz 1024K obj 1 thr 2 write 457.09 [ 234.73, 658.27] rewrite 601.92 [ 214.76, 814.08] read 1436.60 [ 295.60,3624.32]
ost 1 sz 16777216K rsz 1024K obj 1 thr 4 write 598.33 [ 390.56, 835.21] rewrite 616.84 [ 360.60, 886.00] read 2361.10 [ 639.82,5046.86]
ost 1 sz 16777216K rsz 1024K obj 1 thr 8 write 638.57 [ 192.59, 985.76] rewrite 667.73 [ 114.87, 970.75] read 2026.85 [ 451.50,2823.11]
ost 1 sz 16777216K rsz 1024K obj 1 thr 16 write 646.61 [ 485.50, 862.04] rewrite 641.01 [ 0.00,1040.83] read 1872.22 [ 716.21,4353.40]
ost 1 sz 16777216K rsz 1024K obj 2 thr 2 write 529.88 [ 288.68, 774.13] rewrite 505.26 [ 230.75, 824.10] read 1072.79 [ 8.99,4203.22]
ost 1 sz 16777216K rsz 1024K obj 2 thr 4 write 644.92 [ 277.41, 782.12] rewrite 604.37 [ 367.59, 907.99] read 1508.34 [ 12.99,5049.31]
ost 1 sz 16777216K rsz 1024K obj 2 thr 8 write 688.28 [ 63.93,1012.86] rewrite 675.73 [ 159.82, 996.89] read 1678.51 [ 19.00,3427.55]
ost 1 sz 16777216K rsz 1024K obj 2 thr 16 write 677.00 [ 48.95, 968.92] rewrite 630.08 [ 324.63,1030.83] read 1595.35 [ 42.95,2839.49]
ost 1 sz 16777216K rsz 1024K obj 4 thr 4 write 652.07 [ 92.90, 936.98] rewrite 580.03 [ 337.61, 938.83] read 1363.86 [ 7.99,4020.78]
ost 1 sz 16777216K rsz 1024K obj 4 thr 8 write 684.45 [ 220.75, 997.88] rewrite 577.64 [ 206.83,1009.07] read 1357.85 [ 18.98,3411.98]
ost 1 sz 16777216K rsz 1024K obj 4 thr 16 write 711.37 [ 138.86,1109.76] rewrite 579.74 [ 257.71, 904.99] read 1528.18 [ 33.96,4054.37]
ost 1 sz 16777216K rsz 1024K obj 8 thr 8 write 617.76 [ 0.00,1075.80] rewrite 515.14 [ 315.65, 815.08] read 1383.29 [ 19.98,4909.61]
ost 1 sz 16777216K rsz 1024K obj 8 thr 16 write 531.77 [ 0.00,1173.02] rewrite 605.71 [ 170.81,1195.53] read 1602.59 [ 32.96,3676.41]
ost 1 sz 16777216K rsz 1024K obj 16 thr 16 write 579.66 [ 210.54,1203.26] rewrite 523.99 [ 95.89, 962.91] read 1484.40 [ 31.96,2635.00]
done!
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
56 10 0 0 42151 28993 669 12001 431 10439 41629 20639 1 0 0 40544 724
60 13 0 0 42209 29045 684 12099 421 10180 40588 20125 1 0 0 40656 726
48 11 0 0 42212 29030 664 11907 431 10440 41629 20640 1 0 0 40600 725
44 11 0 0 42247 29063 684 12109 432 10442 41629 20642 1 0 0 40712 727
56 11 436 3 41238 28390 671 12057 420 10188 40629 20143 1 0 0 39632 708
48 12 1340 46 42529 29260 710 12593 431 10440 41629 20640 1 0 0 40992 732
40 9 0 0 42353 29134 684 12279 421 10180 40588 20125 1 0 0 40712 727
56 14 0 0 36304 24977 601 10640 377 9134 36425 18059 1 0 0 35000 625
60 12 0 0 41730 28701 671 12048 420 10169 40548 20104 1 0 0 40168 717
36 7 444 5 42703 29378 709 12587 432 10441 41629 20641 1 0 0 41104 734
48 11 796 123 41875 28797 665 11921 420 10179 40588 20124 1 0 0 40264 719
32 8 0 0 41784 28746 670 11861 420 10179 40588 20124 1 0 0 40264 719
64 9 0 0 41670 28662 661 11861 431 10439 41629 20639 1 0 0 40096 716
88 11 0 0 41948 28861 677 11975 421 10180 40588 20125 1 0 0 40376 721
36 9 432 5 41914 28833 668 11991 421 10189 40629 20144 1 0 0 40360 721
40 7 0 0 48113 33099 865 15455 525 12228 48303 24070 1 0 0 46312 827
56 10 0 0 47996 33011 771 13828 577 12177 46872 23685 1 0 0 46200 825
76 14 0 0 48159 33128 779 13824 575 12263 47323 23883 1 0 0 46312 827
64 14 0 0 48708 33522 810 14567 575 12338 47716 24056 1 0 0 46872 837
36 6 440 3 47734 32838 774 13723 560 12085 46811 23580 1 0 0 45986 821
52 11 0 0 47396 32595 761 13656 573 12074 46442 23476 1 0 0 45640 815
52 12 20 3 47473 32657 770 13651 578 11990 45891 23253 1 0 0 45696 816
28 6 4 1 63469 43650 840 14886 657 15921 63484 31475 1 0 0 60984 1089
28 7 0 0 88140 60619 928 15999 889 21463 85610 42448 1 0 0 84896 1516
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
76 10 372 4 93116 64047 1184 20928 942 22855 91104 45169 1 0 0 89600 1600
36 9 0 0 92165 63392 1162 20365 939 22730 90633 44937 1 0 0 88737 1585
72 15 0 0 90408 62177 1027 18007 906 21925 87421 43345 1 0 0 87024 1554
60 12 0 0 90731 62406 1101 19224 928 22451 89504 44381 1 0 0 87024 1554
20 3 0 0 86886 59746 976 17095 874 21142 84298 41797 1 0 0 83944 1499
0 0 404 3 88630 60957 1035 18011 906 21924 87421 43344 1 0 0 85288 1523
0 0 0 0 92254 63439 1127 19864 936 22661 90362 44801 1 0 0 88751 1585
0 0 0 0 90122 61993 1037 18027 917 22207 88550 43904 1 0 0 86775 1550
0 0 0 0 90662 62365 1084 19077 916 22185 88461 43860 1 0 0 87304 1559
4 1 0 0 87496 60176 930 16037 957 21686 85416 42596 1 0 0 84168 1503
120 26 184 3 89070 61251 1024 17969 905 21923 87421 43343 1 0 0 85736 1531
136 28 0 0 93832 64533 1254 22082 949 22969 91584 45409 1 0 0 90328 1613
108 25 0 0 88874 61114 983 17193 896 21685 86466 42871 1 0 0 85542 1528
120 26 12 2 85982 59130 904 15576 876 21144 84299 41798 1 0 0 82768 1478
112 24 0 0 95007 65339 1193 21067 965 23444 93492 46352 1 0 0 91448 1633
108 24 392 5 98034 67418 1286 22632 982 23753 94706 46957 1 0 0 94360 1685
112 27 0 0 92812 63822 1105 19430 948 22966 91583 45406 1 0 0 89376 1596
108 27 0 0 95831 65913 1195 21106 970 23468 93572 46395 1 0 0 92196 1646
144 27 0 0 94472 64971 1230 21623 959 23227 92624 45922 1 0 0 90944 1624
116 26 0 0 91589 62983 1094 19259 927 22446 89502 44376 1 0 0 88200 1575
116 25 380 4 87707 60325 1008 17527 884 21402 85339 42312 1 0 0 84392 1507
156 26 0 0 89071 61254 1009 17684 906 21925 87421 43344 1 0 0 85736 1531
104 24 0 0 88192 60656 968 16752 895 21663 86380 42828 1 0 0 84952 1517
112 26 0 0 95460 65643 1183 20874 974 23541 93822 46529 1 0 0 91820 1640
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
112 26 0 0 92606 63693 914 15654 999 22998 90541 45197 1 0 0 89096 1591
64 13 360 3 98450 67697 1170 20575 1065 24594 96847 48327 1 0 0 94864 1694
68 9 0 0 101979 70135 1255 21960 1091 25245 99496 49629 1 0 0 98112 1752
64 10 0 0 95772 65854 1048 18314 1029 23780 93685 46742 1 0 0 92196 1646
52 10 0 0 100530 69147 1333 23471 1059 24827 98158 48882 1 0 0 96768 1728
48 10 0 0 99672 68550 1147 20132 1087 24841 97625 48778 1 0 0 95928 1713
52 12 420 3 96497 66371 1129 19678 1051 23968 94140 47049 1 0 0 92904 1659
72 13 0 0 93024 63972 883 15220 1017 23275 91480 45700 1 0 0 89600 1600
88 12 844 106 93913 64589 1112 19382 969 23100 91772 45600 1 0 0 90328 1613
48 8 12 2 92992 63951 1114 19606 948 22967 91584 45417 1 0 0 89544 1599
52 11 0 0 91506 62935 1086 18929 918 22208 88550 43914 1 0 0 88064 1573
32 7 384 4 92093 63329 1173 20741 937 22706 90543 44901 1 0 0 88704 1584
60 15 0 0 88196 60659 909 15625 891 21521 85844 42572 1 0 0 84896 1516
136 30 0 0 89004 61201 946 16490 898 21751 86691 42992 1 0 0 85624 1529
128 30 0 0 88558 60917 928 15990 905 21923 87421 43353 1 0 0 85232 1522
132 30 0 0 91930 63222 1019 17829 927 22426 89414 44343 1 0 0 88504 1580
124 31 264 3 96038 66051 1289 22707 981 23749 94705 46964 1 0 0 92456 1651
124 31 0 0 94318 64857 1129 19878 949 22969 91584 45419 1 0 0 90776 1621
148 30 0 0 90540 62266 1018 17669 916 22184 88462 43868 1 0 0 87136 1556
132 30 0 0 90719 62380 1069 18791 927 22446 89502 44386 1 0 0 87360 1560
136 31 0 0 94233 64813 1289 22738 949 22969 91584 45419 1 0 0 90720 1620
124 31 160 3 95711 65814 1265 22435 971 23514 93759 46496 1 0 0 92100 1645
196 34 0 0 92808 63832 1111 19377 938 22707 90543 44901 1 0 0 89320 1595
128 31 0 0 94191 64768 1153 20335 959 23228 92625 45933 1 0 0 90664 1619
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
104 26 12 2 93648 64405 1196 20987 939 22708 90543 44903 1 0 0 90216 1611
124 24 0 0 92767 63796 1115 19624 947 22943 91492 45371 1 0 0 89231 1593
112 22 332 5 91806 63141 1117 19516 928 22447 89502 44386 1 0 0 88368 1578
112 22 0 0 90513 62240 1097 19321 916 22185 88462 43870 1 0 0 87136 1556
104 26 0 0 92454 63586 1212 21303 938 22707 90543 44901 1 0 0 88984 1589
108 23 0 0 96394 66287 1236 21875 981 23750 94705 46966 1 0 0 92792 1657
88 21 0 0 59498 40932 915 16210 593 14358 57241 28389 1 0 0 57344 1024
116 21 452 3 41957 28859 671 12040 421 10190 40629 20149 1 0 0 40416 722
124 24 0 0 42094 28958 681 12065 431 10439 41629 20643 1 0 0 40488 723
96 21 0 0 42365 29138 684 12275 421 10181 40589 20131 1 0 0 40824 729
96 21 0 0 46792 32194 759 13456 545 12034 46890 23543 1 0 0 45024 804
96 22 0 0 48329 33239 775 13903 569 12234 47340 23860 1 0 0 46536 831
96 22 416 3 48720 33517 793 14072 569 12350 47946 24131 1 0 0 46881 837
108 26 0 0 47621 32761 765 13721 579 12107 46448 23509 1 0 0 45808 818
80 20 0 0 47541 32708 775 13753 590 12242 46840 23739 1 0 0 45752 817
104 22 0 0 47716 32819 765 13719 584 12182 46710 23648 1 0 0 45976 821
100 22 0 0 44963 30936 730 12937 484 10964 43000 21511 1 0 0 43232 772
92 20 404 5 42537 29255 683 12261 431 10440 41629 20645 1 0 0 40992 732
100 24 0 0 42514 29253 689 12208 432 10450 41671 20665 1 0 0 40921 731
124 26 0 0 42689 29361 683 12259 431 10440 41629 20645 1 0 0 41048 733
84 19 0 0 42591 29307 693 12270 432 10442 41630 20646 1 0 0 40992 732
120 27 0 0 42659 29343 690 12380 431 10440 41629 20645 1 0 0 41104 734
88 22 964 95 42590 29323 699 12398 441 10700 42670 21159 1 0 0 40992 732
80 19 0 0 42485 29219 680 12200 421 10171 40548 20110 1 0 0 40895 730
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
112 24 0 0 42509 29246 686 12152 428 10428 41574 20615 1 0 0 40880 730
128 25 0 0 42473 29212 678 12149 431 10440 41629 20644 1 0 0 40880 730
148 21 0 0 42568 29315 698 12364 431 10440 41629 20645 1 0 0 40992 732
116 21 404 4 42679 29361 680 12198 436 10476 41796 20728 1 0 0 41048 733
96 23 0 0 42420 29185 687 12160 426 10345 41218 20440 1 0 0 40880 730
108 23 0 0 42162 29000 674 12099 432 10450 41671 20665 1 0 0 40585 725
216 36 348 10 41016 28221 669 11840 410 9918 39548 19612 1 0 0 39480 705
104 23 0 0 42535 29287 692 12422 432 10441 41629 20646 1 0 0 40936 731
112 25 12 3 42429 29194 690 12223 431 10440 41629 20645 1 0 0 40824 729
108 25 0 0 42395 29160 681 12217 431 10441 41629 20645 1 0 0 40824 729
100 20 0 0 42660 29348 708 12564 431 10430 41587 20623 1 0 0 41063 733
100 21 444 4 45535 31315 770 13849 464 11223 44751 22193 1 0 0 43792 782
76 19 0 0 42415 29183 690 12220 431 10440 41629 20644 1 0 0 40824 729
84 20 12 2 51770 35606 742 13233 528 12789 50996 25290 1 0 0 49784 889
100 18 0 0 97679 67181 1210 21179 999 24143 96302 47757 1 0 0 94024 1679
100 24 0 0 97904 67331 1245 22013 994 24083 95995 47605 1 0 0 94192 1682
96 22 436 4 95940 65983 1272 22382 972 23515 93759 46497 1 0 0 92436 1651
104 25 0 0 27413 18867 338 5956 250 6010 23938 11877 1 0 0 26496 474
124 24 0 0 3 22 10 13 11 261 1041 516 1 0 0 0 0
84 21 0 0 1 10 0 2 0 0 0 0 1 0 0 0 0
112 24 0 0 2 16 10 14 0 0 0 0 1 0 0 0 0
Thank your for your suggestions. I've upgraded from GigE to IB and moved from Lustre 1.8.6 to 2.1 on my test cluster. This includes the LU-144 fix for xattrs now but doesn't seem to help. I've eliminated Windows for now and I'm using a RHEL 5.7 machine as a CIFS client since Windows 7 network tuning has its own challenges.
Typical write speed is 44 MB/sec but from time to time it doubles to the low 90 MB/sec then slows back down (see collectl logs from Lustre client/Samba export server below). This cycle can happen twice during a 10 GB write test for no reason I can currently see. I've also noticed that when I break out of a write using dd via ctrl-c the performance jumps to 96 MB/sec until a cache somewhere runs out. This also happens right before the end of a write - the last few hundred MB write at full speed (seen at the end of the collectl log below).
I've tried tuning vfs_cache_pressure, max_rpcs_in_flight and max_dirty_mb on the Lustre client exporting Samba. Tweaking cache pressure seems to make no difference but setting max_rpcs and max_dirty to 256 seems optimal. FWIW - NFS performance is rock solid at 100 MB/sec, what else might I try tuning?
Thank you,
Dan
[root@test2 ~]# obdfilter-survey
Wed Oct 5 14:06:50 PDT 2011 Obdfilter-survey for case=disk from test2
ost 1 sz 16777216K rsz 1024K obj 1 thr 1 write 216.00 [ 101.89, 301.66] rewrite 285.53 [ 23.97, 400.56] read 194.76 [ 94.82, 249.72]
ost 1 sz 16777216K rsz 1024K obj 1 thr 2 write 457.09 [ 234.73, 658.27] rewrite 601.92 [ 214.76, 814.08] read 1436.60 [ 295.60,3624.32]
ost 1 sz 16777216K rsz 1024K obj 1 thr 4 write 598.33 [ 390.56, 835.21] rewrite 616.84 [ 360.60, 886.00] read 2361.10 [ 639.82,5046.86]
ost 1 sz 16777216K rsz 1024K obj 1 thr 8 write 638.57 [ 192.59, 985.76] rewrite 667.73 [ 114.87, 970.75] read 2026.85 [ 451.50,2823.11]
ost 1 sz 16777216K rsz 1024K obj 1 thr 16 write 646.61 [ 485.50, 862.04] rewrite 641.01 [ 0.00,1040.83] read 1872.22 [ 716.21,4353.40]
ost 1 sz 16777216K rsz 1024K obj 2 thr 2 write 529.88 [ 288.68, 774.13] rewrite 505.26 [ 230.75, 824.10] read 1072.79 [ 8.99,4203.22]
ost 1 sz 16777216K rsz 1024K obj 2 thr 4 write 644.92 [ 277.41, 782.12] rewrite 604.37 [ 367.59, 907.99] read 1508.34 [ 12.99,5049.31]
ost 1 sz 16777216K rsz 1024K obj 2 thr 8 write 688.28 [ 63.93,1012.86] rewrite 675.73 [ 159.82, 996.89] read 1678.51 [ 19.00,3427.55]
ost 1 sz 16777216K rsz 1024K obj 2 thr 16 write 677.00 [ 48.95, 968.92] rewrite 630.08 [ 324.63,1030.83] read 1595.35 [ 42.95,2839.49]
ost 1 sz 16777216K rsz 1024K obj 4 thr 4 write 652.07 [ 92.90, 936.98] rewrite 580.03 [ 337.61, 938.83] read 1363.86 [ 7.99,4020.78]
ost 1 sz 16777216K rsz 1024K obj 4 thr 8 write 684.45 [ 220.75, 997.88] rewrite 577.64 [ 206.83,1009.07] read 1357.85 [ 18.98,3411.98]
ost 1 sz 16777216K rsz 1024K obj 4 thr 16 write 711.37 [ 138.86,1109.76] rewrite 579.74 [ 257.71, 904.99] read 1528.18 [ 33.96,4054.37]
ost 1 sz 16777216K rsz 1024K obj 8 thr 8 write 617.76 [ 0.00,1075.80] rewrite 515.14 [ 315.65, 815.08] read 1383.29 [ 19.98,4909.61]
ost 1 sz 16777216K rsz 1024K obj 8 thr 16 write 531.77 [ 0.00,1173.02] rewrite 605.71 [ 170.81,1195.53] read 1602.59 [ 32.96,3676.41]
ost 1 sz 16777216K rsz 1024K obj 16 thr 16 write 579.66 [ 210.54,1203.26] rewrite 523.99 [ 95.89, 962.91] read 1484.40 [ 31.96,2635.00]
done!
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
56 10 0 0 42151 28993 669 12001 431 10439 41629 20639 1 0 0 40544 724
60 13 0 0 42209 29045 684 12099 421 10180 40588 20125 1 0 0 40656 726
48 11 0 0 42212 29030 664 11907 431 10440 41629 20640 1 0 0 40600 725
44 11 0 0 42247 29063 684 12109 432 10442 41629 20642 1 0 0 40712 727
56 11 436 3 41238 28390 671 12057 420 10188 40629 20143 1 0 0 39632 708
48 12 1340 46 42529 29260 710 12593 431 10440 41629 20640 1 0 0 40992 732
40 9 0 0 42353 29134 684 12279 421 10180 40588 20125 1 0 0 40712 727
56 14 0 0 36304 24977 601 10640 377 9134 36425 18059 1 0 0 35000 625
60 12 0 0 41730 28701 671 12048 420 10169 40548 20104 1 0 0 40168 717
36 7 444 5 42703 29378 709 12587 432 10441 41629 20641 1 0 0 41104 734
48 11 796 123 41875 28797 665 11921 420 10179 40588 20124 1 0 0 40264 719
32 8 0 0 41784 28746 670 11861 420 10179 40588 20124 1 0 0 40264 719
64 9 0 0 41670 28662 661 11861 431 10439 41629 20639 1 0 0 40096 716
88 11 0 0 41948 28861 677 11975 421 10180 40588 20125 1 0 0 40376 721
36 9 432 5 41914 28833 668 11991 421 10189 40629 20144 1 0 0 40360 721
40 7 0 0 48113 33099 865 15455 525 12228 48303 24070 1 0 0 46312 827
56 10 0 0 47996 33011 771 13828 577 12177 46872 23685 1 0 0 46200 825
76 14 0 0 48159 33128 779 13824 575 12263 47323 23883 1 0 0 46312 827
64 14 0 0 48708 33522 810 14567 575 12338 47716 24056 1 0 0 46872 837
36 6 440 3 47734 32838 774 13723 560 12085 46811 23580 1 0 0 45986 821
52 11 0 0 47396 32595 761 13656 573 12074 46442 23476 1 0 0 45640 815
52 12 20 3 47473 32657 770 13651 578 11990 45891 23253 1 0 0 45696 816
28 6 4 1 63469 43650 840 14886 657 15921 63484 31475 1 0 0 60984 1089
28 7 0 0 88140 60619 928 15999 889 21463 85610 42448 1 0 0 84896 1516
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
76 10 372 4 93116 64047 1184 20928 942 22855 91104 45169 1 0 0 89600 1600
36 9 0 0 92165 63392 1162 20365 939 22730 90633 44937 1 0 0 88737 1585
72 15 0 0 90408 62177 1027 18007 906 21925 87421 43345 1 0 0 87024 1554
60 12 0 0 90731 62406 1101 19224 928 22451 89504 44381 1 0 0 87024 1554
20 3 0 0 86886 59746 976 17095 874 21142 84298 41797 1 0 0 83944 1499
0 0 404 3 88630 60957 1035 18011 906 21924 87421 43344 1 0 0 85288 1523
0 0 0 0 92254 63439 1127 19864 936 22661 90362 44801 1 0 0 88751 1585
0 0 0 0 90122 61993 1037 18027 917 22207 88550 43904 1 0 0 86775 1550
0 0 0 0 90662 62365 1084 19077 916 22185 88461 43860 1 0 0 87304 1559
4 1 0 0 87496 60176 930 16037 957 21686 85416 42596 1 0 0 84168 1503
120 26 184 3 89070 61251 1024 17969 905 21923 87421 43343 1 0 0 85736 1531
136 28 0 0 93832 64533 1254 22082 949 22969 91584 45409 1 0 0 90328 1613
108 25 0 0 88874 61114 983 17193 896 21685 86466 42871 1 0 0 85542 1528
120 26 12 2 85982 59130 904 15576 876 21144 84299 41798 1 0 0 82768 1478
112 24 0 0 95007 65339 1193 21067 965 23444 93492 46352 1 0 0 91448 1633
108 24 392 5 98034 67418 1286 22632 982 23753 94706 46957 1 0 0 94360 1685
112 27 0 0 92812 63822 1105 19430 948 22966 91583 45406 1 0 0 89376 1596
108 27 0 0 95831 65913 1195 21106 970 23468 93572 46395 1 0 0 92196 1646
144 27 0 0 94472 64971 1230 21623 959 23227 92624 45922 1 0 0 90944 1624
116 26 0 0 91589 62983 1094 19259 927 22446 89502 44376 1 0 0 88200 1575
116 25 380 4 87707 60325 1008 17527 884 21402 85339 42312 1 0 0 84392 1507
156 26 0 0 89071 61254 1009 17684 906 21925 87421 43344 1 0 0 85736 1531
104 24 0 0 88192 60656 968 16752 895 21663 86380 42828 1 0 0 84952 1517
112 26 0 0 95460 65643 1183 20874 974 23541 93822 46529 1 0 0 91820 1640
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
112 26 0 0 92606 63693 914 15654 999 22998 90541 45197 1 0 0 89096 1591
64 13 360 3 98450 67697 1170 20575 1065 24594 96847 48327 1 0 0 94864 1694
68 9 0 0 101979 70135 1255 21960 1091 25245 99496 49629 1 0 0 98112 1752
64 10 0 0 95772 65854 1048 18314 1029 23780 93685 46742 1 0 0 92196 1646
52 10 0 0 100530 69147 1333 23471 1059 24827 98158 48882 1 0 0 96768 1728
48 10 0 0 99672 68550 1147 20132 1087 24841 97625 48778 1 0 0 95928 1713
52 12 420 3 96497 66371 1129 19678 1051 23968 94140 47049 1 0 0 92904 1659
72 13 0 0 93024 63972 883 15220 1017 23275 91480 45700 1 0 0 89600 1600
88 12 844 106 93913 64589 1112 19382 969 23100 91772 45600 1 0 0 90328 1613
48 8 12 2 92992 63951 1114 19606 948 22967 91584 45417 1 0 0 89544 1599
52 11 0 0 91506 62935 1086 18929 918 22208 88550 43914 1 0 0 88064 1573
32 7 384 4 92093 63329 1173 20741 937 22706 90543 44901 1 0 0 88704 1584
60 15 0 0 88196 60659 909 15625 891 21521 85844 42572 1 0 0 84896 1516
136 30 0 0 89004 61201 946 16490 898 21751 86691 42992 1 0 0 85624 1529
128 30 0 0 88558 60917 928 15990 905 21923 87421 43353 1 0 0 85232 1522
132 30 0 0 91930 63222 1019 17829 927 22426 89414 44343 1 0 0 88504 1580
124 31 264 3 96038 66051 1289 22707 981 23749 94705 46964 1 0 0 92456 1651
124 31 0 0 94318 64857 1129 19878 949 22969 91584 45419 1 0 0 90776 1621
148 30 0 0 90540 62266 1018 17669 916 22184 88462 43868 1 0 0 87136 1556
132 30 0 0 90719 62380 1069 18791 927 22446 89502 44386 1 0 0 87360 1560
136 31 0 0 94233 64813 1289 22738 949 22969 91584 45419 1 0 0 90720 1620
124 31 160 3 95711 65814 1265 22435 971 23514 93759 46496 1 0 0 92100 1645
196 34 0 0 92808 63832 1111 19377 938 22707 90543 44901 1 0 0 89320 1595
128 31 0 0 94191 64768 1153 20335 959 23228 92625 45933 1 0 0 90664 1619
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
104 26 12 2 93648 64405 1196 20987 939 22708 90543 44903 1 0 0 90216 1611
124 24 0 0 92767 63796 1115 19624 947 22943 91492 45371 1 0 0 89231 1593
112 22 332 5 91806 63141 1117 19516 928 22447 89502 44386 1 0 0 88368 1578
112 22 0 0 90513 62240 1097 19321 916 22185 88462 43870 1 0 0 87136 1556
104 26 0 0 92454 63586 1212 21303 938 22707 90543 44901 1 0 0 88984 1589
108 23 0 0 96394 66287 1236 21875 981 23750 94705 46966 1 0 0 92792 1657
88 21 0 0 59498 40932 915 16210 593 14358 57241 28389 1 0 0 57344 1024
116 21 452 3 41957 28859 671 12040 421 10190 40629 20149 1 0 0 40416 722
124 24 0 0 42094 28958 681 12065 431 10439 41629 20643 1 0 0 40488 723
96 21 0 0 42365 29138 684 12275 421 10181 40589 20131 1 0 0 40824 729
96 21 0 0 46792 32194 759 13456 545 12034 46890 23543 1 0 0 45024 804
96 22 0 0 48329 33239 775 13903 569 12234 47340 23860 1 0 0 46536 831
96 22 416 3 48720 33517 793 14072 569 12350 47946 24131 1 0 0 46881 837
108 26 0 0 47621 32761 765 13721 579 12107 46448 23509 1 0 0 45808 818
80 20 0 0 47541 32708 775 13753 590 12242 46840 23739 1 0 0 45752 817
104 22 0 0 47716 32819 765 13719 584 12182 46710 23648 1 0 0 45976 821
100 22 0 0 44963 30936 730 12937 484 10964 43000 21511 1 0 0 43232 772
92 20 404 5 42537 29255 683 12261 431 10440 41629 20645 1 0 0 40992 732
100 24 0 0 42514 29253 689 12208 432 10450 41671 20665 1 0 0 40921 731
124 26 0 0 42689 29361 683 12259 431 10440 41629 20645 1 0 0 41048 733
84 19 0 0 42591 29307 693 12270 432 10442 41630 20646 1 0 0 40992 732
120 27 0 0 42659 29343 690 12380 431 10440 41629 20645 1 0 0 41104 734
88 22 964 95 42590 29323 699 12398 441 10700 42670 21159 1 0 0 40992 732
80 19 0 0 42485 29219 680 12200 421 10171 40548 20110 1 0 0 40895 730
#<----------Disks-----------><----------Network----------><-----------InfiniBand-----------><--------Lustre Client-------->
#KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut KBIn PktIn KBOut PktOut Errs KBRead Reads KBWrite Writes
112 24 0 0 42509 29246 686 12152 428 10428 41574 20615 1 0 0 40880 730
128 25 0 0 42473 29212 678 12149 431 10440 41629 20644 1 0 0 40880 730
148 21 0 0 42568 29315 698 12364 431 10440 41629 20645 1 0 0 40992 732
116 21 404 4 42679 29361 680 12198 436 10476 41796 20728 1 0 0 41048 733
96 23 0 0 42420 29185 687 12160 426 10345 41218 20440 1 0 0 40880 730
108 23 0 0 42162 29000 674 12099 432 10450 41671 20665 1 0 0 40585 725
216 36 348 10 41016 28221 669 11840 410 9918 39548 19612 1 0 0 39480 705
104 23 0 0 42535 29287 692 12422 432 10441 41629 20646 1 0 0 40936 731
112 25 12 3 42429 29194 690 12223 431 10440 41629 20645 1 0 0 40824 729
108 25 0 0 42395 29160 681 12217 431 10441 41629 20645 1 0 0 40824 729
100 20 0 0 42660 29348 708 12564 431 10430 41587 20623 1 0 0 41063 733
100 21 444 4 45535 31315 770 13849 464 11223 44751 22193 1 0 0 43792 782
76 19 0 0 42415 29183 690 12220 431 10440 41629 20644 1 0 0 40824 729
84 20 12 2 51770 35606 742 13233 528 12789 50996 25290 1 0 0 49784 889
100 18 0 0 97679 67181 1210 21179 999 24143 96302 47757 1 0 0 94024 1679
100 24 0 0 97904 67331 1245 22013 994 24083 95995 47605 1 0 0 94192 1682
96 22 436 4 95940 65983 1272 22382 972 23515 93759 46497 1 0 0 92436 1651
104 25 0 0 27413 18867 338 5956 250 6010 23938 11877 1 0 0 26496 474
124 24 0 0 3 22 10 13 11 261 1041 516 1 0 0 0 0
84 21 0 0 1 10 0 2 0 0 0 0 1 0 0 0 0
112 24 0 0 2 16 10 14 0 0 0 0 1 0 0 0 0
Reply all
Reply to author
Forward
0 new messages