understand the "Time" and "CPU" time on the following case

[zhihui....@gmail.com]

Here is the code:

int num_threads = 1;

static void BM_RawAccess(benchmark::State &state) {
for (auto _ : state)
raw_access();
}

BENCHMARK(BM_RawAccess)->Threads(num_threads);

static void BM_MutexAccess(benchmark::State &state) {
for (auto _ : state)
mutex_access();
}

BENCHMARK(BM_MutexAccess)->Threads(num_threads);

the result is:

--------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------
BM_RawAccess/threads:4 0.435 ns 1.59 ns 431479512
BM_MutexAccess/threads:4 51.8 ns 155 ns 5073936

My questions are:

1. Why the cpu time can greater than time?
2. what I want to benchmark is : the latency and the throughput among different benchmark, for here, it is "MutexAccess and RawAccess". per the "Time" or "CPU time", it works for "latency" purpose. but the "iterations" doesn't work for throughput purpose since they doesn't run in the equal time duration. what is the suggested way to do this?
3. how can I set the num_threads easily during run time? simply change "num_threads = 4" to num_threads = std::max(std::stoi(std::getenv("xxx")), 1); will crash the program. the error is:

* thread #1, stop reason = signal SIGSTOP
* frame #0: 0x00007fff5bfacd8b libsystem_c.dylib`strtol_l + 80
frame #1: 0x000000010ce33860 perf`_GLOBAL__sub_I_perf.cpp + 32
frame #2: 0x000000010cf83cc8 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 518
frame #3: 0x000000010cf83ec6 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
frame #4: 0x000000010cf7f0da dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 358
frame #5: 0x000000010cf7e254 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 134
frame #6: 0x000000010cf7e2e8 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 74
frame #7: 0x000000010cf6d774 dyld`dyld::initializeMainExecutable() + 199
frame #8: 0x000000010cf7278f dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 6237
frame #9: 0x000000010cf6c4f6 dyld`dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*) + 1154
frame #10: 0x000000010cf6c036 dyld`_dyld_start + 54

Thanks

[zhihui....@gmail.com]

copy/paste error. the "num_threads = 1" should be "num_threads = 4".

[Roman Lebedev]

On Mon, Jul 1, 2019 at 3:19 AM <zhihui....@gmail.com> wrote:
>
> Here is the code:
>
> int num_threads = 1;
>
> static void BM_RawAccess(benchmark::State &state) {
> for (auto _ : state)
> raw_access();
> }
>
> BENCHMARK(BM_RawAccess)->Threads(num_threads);
>
>
> static void BM_MutexAccess(benchmark::State &state) {
> for (auto _ : state)
> mutex_access();
> }
>
> BENCHMARK(BM_MutexAccess)->Threads(num_threads);
>
> the result is:
>
> --------------------------------------------------------------------------
> Benchmark Time CPU Iterations
> --------------------------------------------------------------------------
> BM_RawAccess/threads:4 0.435 ns 1.59 ns 431479512
> BM_MutexAccess/threads:4 51.8 ns 155 ns 5073936
>
> My questions are:
>
> 1. Why the cpu time can greater than time?

See https://github.com/google/benchmark/issues/769
"Time" is indeed printed divided by num_threads.
I can't tell if it is a bug or not, but it is what it is.

> 2. what I want to benchmark is : the latency and the throughput among different benchmark, for here, it is "MutexAccess and RawAccess". per the "Time" or "CPU time", it works for "latency" purpose. but the "iterations" doesn't work for throughput purpose since they doesn't run in the equal time duration. what is the suggested way to do this?
> 3. how can I set the num_threads easily during run time? simply change "num_threads = 4" to num_threads = std::max(std::stoi(std::getenv("xxx")), 1); will crash the program. the error is:

Define runtime. The most trivial solution is to use
https://github.com/google/benchmark#using-registerbenchmarkname-fn-args
and call `->Threads()` with `std::max(std::stoi(std::getenv("xxx")), 1);`

> * thread #1, stop reason = signal SIGSTOP
> * frame #0: 0x00007fff5bfacd8b libsystem_c.dylib`strtol_l + 80
> frame #1: 0x000000010ce33860 perf`_GLOBAL__sub_I_perf.cpp + 32
> frame #2: 0x000000010cf83cc8 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 518
> frame #3: 0x000000010cf83ec6 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
> frame #4: 0x000000010cf7f0da dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 358
> frame #5: 0x000000010cf7e254 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 134
> frame #6: 0x000000010cf7e2e8 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 74
> frame #7: 0x000000010cf6d774 dyld`dyld::initializeMainExecutable() + 199
> frame #8: 0x000000010cf7278f dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 6237
> frame #9: 0x000000010cf6c4f6 dyld`dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*) + 1154
> frame #10: 0x000000010cf6c036 dyld`_dyld_start + 54
>
>
> Thanks

Roman.

> --
> You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/benchmark-discuss/106c43df-8c4e-462d-afca-b6271b701f3e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.