measuring "cpu time" instead of "time time" during performance tests

[Westley Hennigh]

I would like to run *lots* of julia performance tests on my machine. Probably the reasonable way to do this would be to shut down other applications and run each of the tests, one by one, in a single julia process. But that's going to take a long time, and I'd be running single threaded tests on my multi-core machine. Wouldn't it be nice if I could spin up multiple julia processes and run different tests in parallel?

To do that, I think I need to find a way to measure the "cpu time" that each test takes, rather than the "time time". Does that seem reasonable? Does anyone have any advice?

Acceptable answers to this question include variants of "that's a terrible idea, for the love of god don't do that."

[ele...@gmail.com]

Short answer "Correct" :)

Longer answer, the cores in your machine share a number of resources, L1 cache, memory etc.  If you run several tests at once they will interact in unknown ways as they access the shared resources making the tests impossible to compare.

Cheers

Lex

 

[Westley Hennigh]

Ah, that's a really good point, thanks!

[Christian Peel]

Here's a stackoverflow discussion on the same topic: http://stackoverflow.com/questions/24427185/measuring-elapsed-cpu-time-in-julia   which resulted in the following package to measure CPU time:  https://github.com/schmrlng/CPUTime.jl   A discussion of the scenarios in which this package is useful is at https://github.com/schmrlng/CPUTime.jl/issues/1   My understanding is if one calls disable_threaded_libs()   (which just calls blas_set_num_threads(1) right now) then the package gives useful results.  I know that there is work to add multithreading elsewhere in Julia (see https://github.com/JuliaLang/julia/pull/6741) but I'm not certain where it's at.

I'd very much like someone to correct me if there are places where multi-threaded computation is used under the hood which make CPUTime not useful (assuming nprocs() is 1 and no explicit parallel proc is used).

--

chris...@ieee.org

[Amit Murthy]

For parallel processing, it will be useful to have a package or support in Base itself to time

- aggregate CPUTime used across all workers for a code block executing in parallel

- calculate the communication/serialization overhead when executing across different nodes

- list CPU counters like L1/L2 cache misses 

This information can help in quickly detecting the main cause of poor scaling with multiple cores for certain workloads.