Testing

[Ed Papa]

Hello,

I have developed a custom priority queue (bounded unique thread-safe priority queue) that I want to benchmark with multiple threads, each invoking a different operation of the priority queue. My goal is to benchmark the priority queue's operations under heavy thread contention.

M threads will call add()
N threads will call take()
L threads will call remove()

I want to run all threads at the same time and benchmark the time each operation takes.

Can caliper help with this use case? If it can, I would appreciate some guidance one how to set up the benchmark class.

Thank you

Ed

[Gregory Kick]

Caliper doesn't have any explicit support for multi-threaded benchmarks.  It's possible to make something reasonable, and a few people have, but there is no out-of-the-box solution for the type of benchmark you're trying to create.

Unfortunately, the person that has the most direct experience with what you're trying to do (he has actually written multi-threaded benchmarks for queues) is on vacation.  I've bcc'd him, so when he comes back he'll hopefully be able to lend some advice.

--
--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")
 
---
You received this message because you are subscribed to the Google Groups "caliper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caliper+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Greg Kick
Java Core Libraries Team

[Luke Sandberg]

A few tips i figured out out the hard way.

* Expect to increase the caliper time-limit significantly, I had to up mine to 5 minutes to get results (this is likely because anything involving multiple threads + context switches has a large amount of variance which means it will take awhile for caliper to be confident in its results, e.g. calling take() on an ArrayBlockingQueue will take ~70 ns on a modern machine if the lock is free and the queue has an item waiting, but if the lock is contended and the thread has to block that can add 12 microseconds of latency which is hundreds of times larger that the actual operation)

* You will mostly likely have to disable the allocation instrument when running with multiple threads, this is because synchronization primitives often allocate objects when they are contended (e.g. a node in the wait queue), which means that your allocation behavior will be non-deterministic (which caliper doesn't like).  To measure allocation behavior you should write a separate benchmark that uses a single thread (or very carefully interleaves threads to measure particular cases of contention)

* Some of the things you likely want to measure would be something like queue throughput (put-takes per second) and caliper can't help you with that (at least not in a way that i can figure out).

* if you want to measure how long it takes to add, take, or remove an item from a queue you are going to have to write a different benchmark for each operation you want to measure (caliper can only measure one thing at a time) and you are probably going to want to measure operations in pairs like put/take or add/remove otherwise you're going to fill your queues or spend too much time allocating and drown out your actual operation.  e.g. i wrote my benchmarks to measure how long it takes to push reps numbers of items through a queue using various queue operations and numbers of threads. i.e.  It is hard to individually measure an add operation without also measuring a remove operation.

It would be nice if caliper offered some help with writing these sorts of benchmarks, but I'm not sure how it could :/

[Kevin Burton]

Can you implement a near real-world situations similar to your benchmark?

Maybe, outside of the benchmark, startup N threads and then have them wait on a CountDownLatch and then from within your benchmark method call the countDown method on the latch.

The main issue I could foresee is dealing with 'reps'... but perhaps you could create it so that you can block on a countDownLatch each time. 

Though the latch might impose some additional overhead.

[David Beaumont]

This feel like diminishing returns to me. It seems quite unlikely that a benchmark like this would be usefully comparable to a real world application.

Whenever I've tried things like this it's always been in terms of letting the system start up and reach an active steady state then flip a flag to trigger the counting of reps and have a timer task that records the timestamp at which the test was halted. Each thread counts reps locally and they get summed at the end.

That avoids having to be clever with sharding work 'equally' in any sense or having strong control over threads.

However, I still didn't find the results _that_ useful.

Cheers,

    David

--

--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")
 
---
You received this message because you are subscribed to the Google Groups "caliper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caliper+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
David Beaumont :: Îñţérñåţîöñåļîžåţîờñ Libraries :: Google
Google Switzerland GmbH., Brandschenkestrasse 110, CH-8002, Zürich - Switzerland
Tel +41 44 668 1800 :: Fax +41 44 668 1818