vdbench issues

[Noa Joo]

Hi isilon Users

I am testing vdbench. I tested it 4 times and the results keep getting slower. 

I'd like to know why.

Parameter	I/O type			Threads	Shared	Elapsed	Warmup
4-1	90%	Sequential 64K	Read 10%	128	yes	1,200	60
			Write 90%	512
	10%	Random 8K	Read 10%	64
			Write 90%	128
Results
FIXED(10000)	OPS(avg)	OPS(max)	RT	Throughput(MB/s)	Mount
1	10001		2.094	498.39	default	with customer
2	9998.2		11.187	497.46	default	with customer
3	8353.2		57.502	415.72
4	7499.2		64.245	373.12

vdbench parameter 

## Multihost Definition
hd=default,vdbench=/vdbench,shell=ssh,user=root
hd=host1,system=Server1
hd=host2,system=Server2

## Filesystem Definition
fsd=fsd41host1,anchor=/fsdata/data41,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd42host2,anchor=/fsdata/data42,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd43host3,anchor=/fsdata/data43,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd44host4,anchor=/fsdata/data44,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd45host5,anchor=/fsdata/data45,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd46host6,anchor=/fsdata/data46,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd47host7,anchor=/fsdata/data47,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes
fsd=fsd48host8,anchor=/fsdata/data48,depth=4,width=2,files=300000,size=(50k,10,100k,20,200k,20,300k,20,400k,20,450k,10),openflags=o_direct,distribution=bottom,shaed=yes

## Filesystem Workload Definition
fwd=fwd411,fsd=fsd4*,operation=read,xfersize=64k,fileio=sequential,fileselect=random,threads=128,skew=9
fwd=fwd412,fsd=fsd4*,operation=write,xfersize=64k,fileio=sequential,fileselect=random,threads=512,skew=81
fwd=fwd413,fsd=fsd4*,operation=read,xfersize=8k,fileio=random,fileselect=random,threads=64,skew=1
fwd=fwd414,fsd=fsd4*,operation=write,xfersize=8k,fileio=random,fileselect=random,threads=128,skew=9

## Run Definition
rd=rd41,fwd=fwd41*,fwdrate=???,format=no,elapsed=1200,warmup=60,interval=1

[Noa Joo]

I send it back to you including the configuration as well.

2024년 2월 23일 (금) 오후 7:12, Noa Joo <jwtl...@gmail.com>님이 작성:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/CAF18haLseck6%3DihzPZdp7rLy3ELsuM3Z99L8Hvwx_ijqQLRBrQ%40mail.gmail.com.

[Youssef GHORBAL]

Hi,

Based on a rather long experience on Isilon clusters (> 10 years) you'll never get a consistent synthetic benchmark results on an Isilon (corollary: you'll also never get consistent performance metrics in a production enviroment) All you can assess, is a range going from "the speed of light" and "NFS not responding".

From my experience the most "performance impactful" enviroment variables:

- Internal cluster jobs (MediaScan, FSAnalyse, etc) 

- Caching (data, metadata memory and/or SSD L3 cache) -> drop all caches (client and server side) between different runs (echo > 3 /proc/sys/vm/drop_caches, isi_flush, isi_flush --l3-full etc)

- Random access patternes (be it read, write, metadata etc)

- Small files patterns

Also, you'll never get a clear cut answer from Dell Support/Engeneering/etc on whether what you are reporting as results are "expected" (based on your setup) or are symptoms of an issue. All you'll get is "it depends" :D

Youssef Ghorbal

--------------------------

On 23 Feb 2024, at 11:19, Noa Joo <jwtl...@gmail.com> wrote:

I send it back to you including the configuration as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/CAF18haKxKyCUAsoeOAOGjGdWzrb67S6XHDSQrSmZpS9PGUpFBA%40mail.gmail.com.

[Michael Ebert]

You could always install an InsightIQ server to monitor and provide real-time analysis of your clusters.  It is a free diagnostic and reporting tool that can be downloaded from the Dell support site. This might help you glean insight into your performance issues.

Regards,

Mike