Load patterns generation

[Paolo Forte]

Hi everybody,

I need to apply different load patterns to a generator. I am currently using the Voldemort's Performance Tool and it doesn't provide such functionalities.

More specifically, I want to keep track of the response time of Voldemort's servers while varying the load pattern (e.g. increasing the number of clients linearly during the time or using a poisson model).

Has anyone ever developed such a load generator for Voldemort?

[Felix GV]

Hi Paolo,

We have a Voldemort Perf Lab we use internally to identify performance regressions and test improvements.

I think it does not support changing the load pattern midway through a test. Rather, we configure many different settings we wish to test (in terms of load characteristics and/or in terms of client/server configs) and VPL then goes through every desired permutations for 30 minutes or more each time, and gathers and plots the data.

I don't know if our perf lab internally re-uses the open-source perf tool and just adds extra tooling and automation around it, or if it's completely different. Xu might be able to give more details.

Is there something specific you're interested in when looking at changing load patterns, versus just doing a few different (completely separate) tests?

--
 
Felix GV
Data Infrastructure Engineer
Distributed Data Systems
LinkedIn
 
f...@linkedin.com
linkedin.com/in/felixgv

---

From: project-...@googlegroups.com [project-...@googlegroups.com] on behalf of Paolo Forte [forte....@gmail.com]
Sent: Tuesday, January 27, 2015 6:28 AM
To: project-...@googlegroups.com
Subject: [project-voldemort] Load patterns generation

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Paolo Forte]

My goal is to study the server's behaviour in a dynamic scenario, looking at some internal variables (using sysstat).

This is the reason because I need a dynamic load generator. I guess I have to figure it out in some other way. :\

Suggestions are welcome! :D

[Felix GV]

All right, well, if you end up creating a new perf tool or extending the current one to fit your needs, don't hesitate to send a pull request for it ;)

--
 
Felix GV
Data Infrastructure Engineer
Distributed Data Systems
LinkedIn
 
f...@linkedin.com
linkedin.com/in/felixgv

From: project-...@googlegroups.com [project-...@googlegroups.com] on behalf of Paolo Forte [forte....@gmail.com]

Sent: Tuesday, January 27, 2015 8:04 AM
To: project-...@googlegroups.com
Cc: Xu Ha
Subject: Re: [project-voldemort] Load patterns generation

[Paolo Forte]

Do you perhaps know any other traffic generator for Voldemort?
For instance, I need to send traffic according to a Poisson distribution. 

[Arunachalam]

We don't have anything in the open source for perf testing. We have something in house but it is tightly coupled with our inhouse infrastructure.

The closest I can think of is, you can start from these unit test utilities. There are many dumps for database testing, take them and modify these to send the traffic based on them. You might need to bootstrap warmup and other usual nuances of perf testing.

https://github.com/voldemort/voldemort/blob/master/test/unit/voldemort/client/ClientTrafficGenerator.java

https://github.com/voldemort/voldemort/blob/master/test/unit/voldemort/client/ClientTrafficVerifier.java

Hope it helps.

Thanks,

Arun.

[Paolo Forte]

Thanks. I'll read through the codes you linked.

What I have been thinking to is theoretically simple (and dirty): I want to start from the performance tool code and add somewhere some sleeps, in order to shape the traffic of the generator.