We attached both the jacoco agent and our custom method-level-only agent ("funccover") to some sample codes and averaged their run execution time.
We attached “jacocoagent” and “funccover” to Google Java Formatter to test and compare “funccover”s performance. Results are the average of 200 runs.
Normally formatter’s execution time is around 200 seconds, with “jacocoagent” it runs around 97 seconds (48% overhead) slower and for “funccover” it runs around 30 seconds slower (15% overhead). So our agent adds x3 less overhead for this case.
2) Merge Sort Algorithm
In this test we implemented a program that sorts an array of size 10^7 with a merge sort algorithm. In each binary, we sort the array 50 times per run to be able to ignore the constant overhead of instrumentation since it is only done once per run. There is no I/O bottleneck and most of the time is spent on the CPU. Binary runs around 216 seconds without the instrumentation. Our agent "funccover" adds around 1.97% overhead whilst jacoco adds around 11.4%. Our instrumentation adds 5x less overhead. Results are average of 50x runs.
3) Simple Function Calls - Filtered
We attached “jacocoagent” and “funccover” to a simple program that calls the following function 10^8 times with numbers from 1 to 10^8.
static int f(int x) {
if(x == 0) return 0;
if(x % 2 == 0) return f(x / 2);
return f(x / 2) + 1;
}
Results are average of 500x runs.
Here jacoco adds 17% overhead whilst funccover adds less than 7% overhead.
All this makes sense of course, you are updating less often the coverage data if you only add a single probe per method.