[llvm-dev] How to get branch coverage by using 'source-based code coverage'
LoveZhi Xie via llvm-dev
Phipps, Alan via llvm-dev
Hi Ted,
Presently, branch coverage is not supported in clang source-based code coverage, but I am working on implementing branch condition coverage right now. It will track True/False branches for each leaf-level condition, including within Boolean expressions comprised of logical operators (“&&”, “||”). Technically, this condition-based coverage is more granular than how “branch coverage” is often defined (tracking on that control flow decisions allow for all regions to be hit), and is closer to the granularity that GCOV provides, but it’s anchored to the source code, won’t count “hidden branches”, and isn’t negatively impacted by optimization (although the reverse isn’t true: coverage instrumentation may impact optimization).
My work is a step closer to getting us toward Modified Condition/Decision Coverage (MC/DC). I hope to upstream my work within the next few months, so unfortunately it won’t help you in the short-term. Region coverage may be enough for what you need if you can ascertain based on the coverage that control-flow decisions have evaluated to both true and false, but I’ll let others comment here with suggestions.
-Alan Phipps
LoveZhi Xie via llvm-dev
- Line coverage
- Function coverage
- Branch coverage/region coverage
- Condition coverage
- Modified Condition/Decision Coverage
Phipps, Alan via llvm-dev
+ Vedant Kumar
Hi Ted!
I would not say that clang region coverage is branch coverage(‘aka decision coverage’). I would say that it may be possible to extrapolate branch decision coverage based on region coverage. For example, consider the following:
[region0]
if (decision) {
[region1]
}
Region coverage will tell you that whether [region1] was executed, and this will be enough to tell you that the branch decision evaluated to True at least once. However, this will not tell you how many times the decision evaluated to False, which is also important when considering branch coverage. In this case, you can calculate this value by subtracting the number of times [region0] was executed from the number of times [region1] was executed.
What I am implementing for branch condition coverage will do this for you but will provide more granular True/False counts for individual conditions that comprise all Boolean expression (which also generate branches), and will tally all of these cases as branches that are either covered or not covered.
I hope that helps.
As for the future goals, I do plan on addressing what’s required for MC/DC, but that is work that I have not yet planned.
Thanks!
-Alan Phipps
LoveZhi Xie via llvm-dev
#include <string>// If not comment this line, the branch coverage won't reach to 100%// #include <iostream>int main(int argc, const char* argv[]) {std::string str = "foo";str = argv[1];if (str == "foo" || str == "bar") {int a = 5;a = 7;} else {int b = 6;b = 4;}return 0;}
#!/bin/bashg++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage simple.cc./simple "foo"./simple "bar"./simple "hello1"./simple "hello2"./simple "hello3"./simple "hello4"./simple "hello5"./simple "hello6"./simple "hello7"./simple "hello8"lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file simple.lcov.infogenhtml simple.lcov.info --function-coverage --branch-coverage --output-directory lcov_out/gcov -b -c simple.gcno
- What's the "official" evaluation formula on branch coverage?
- Are there any existing post-processing tools based on clang/llvm source based coverage?
- How to disable STL (or other) noise in gcov branch coverage?
Phipps, Alan via llvm-dev
Hi Ted,
I’ll attempt to answer your questions:
1.) What's the "official" evaluation formula on branch coverage? Sorry I can't find any compelling documents to describe the accurate algorithm on this issue. I just thought gcov/lcov won't consider how many times the decision evaluated to False.
a. [Alan]: I share your concern here. My reference is the ISO26262 standard for functional safety, which defines branch coverage as “percentage of branches of the control flow of a computer program executed during a test. Note 1: 100% branch coverage implies 100% statement coverage., Note 2: An if-statement always has two branches – condition true and condition false – independent of the existence of an else-clause”
i. Basically, condition true and condition false for every decision in a program. In contrast, MC/DC applies to single condition outcomes that affect a decision.
b. [Alan] Your GCOV output is producing branch information for each condition of the decision, and each of those branches corresponds to the true and false evaluation of each condition. The problem with GCC/GCOV is that, because of the way it instruments, it’s not obvious how branches correspond to the source code.
2.) Are there any existing post-processing tools based on clang/llvm source based coverage
a. [Alan] I’m not aware of them, but others may know
3.) How to disable STL (or other) noise in gcov branch coverage?
a. [Alan] I think this is a perennial problem – gcov reports hidden branches that are created, but the data isn’t necessarily useful from a source-code coverage perspective.
-Alan