Getting function coverage drcov2lcov

[Edgar Neubauer]

Greetings,

I use drcov to generate a coverage report. I convert this to lcov using drcov2lcov tool, then use genhtml to generate a html report. This reports only shows the called source code lines but not the number functions called (it's always 0/0). Is there a special parameter in drcov or drcov2lcov that generates this info?

Regards

[Derek Bruening]

drcov2lcov today does not output function info into the coverage.info file: this is in the tracker as https://github.com/DynamoRIO/dynamorio/issues/1435.  If someone has time to contribute to that feature it would be appreciated.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/ec7775a0-4019-48ec-8e21-d7bbd08da857n%40googlegroups.com.

[Edgar Neubauer]

Thanks for your answer. Another question: A recent run I tried didn't list a function as triggered which I now was triggered by log. The app uses g++ with O2 and RelWithDbgInfo. Might this cause problems?

[Edgar Neubauer]

It seems to be that only code in the main thread is captured. Are there any special options one has to take in this case?

[Edgar Neubauer]

-thread_private doesn't change it.

[Edgar Neubauer]

It works with the X64 Linux version. Aarch64 only records main thread. Latest cronbuild. 9.0.19368

[Derek Bruening]

On Mon, Jan 23, 2023 at 8:12 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Thanks for your answer. Another question: A recent run I tried didn't list a function as triggered which I now was triggered by log. The app uses g++ with O2 and RelWithDbgInfo. Might this cause problems?

Mapping back to source relies on debug info.  Maybe something was inlined and not perfectly mapped back in the debug info?  You could drill down into the details by finding the precise PC sequence for the code in question, ensuring that PC range was recorded by drcov's raw data, and observing what source code lines those turn into in the post-processing step.

 

Derek Bruening schrieb am Donnerstag, 19. Januar 2023 um 16:59:15 UTC+1:

drcov2lcov today does not output function info into the coverage.info file: this is in the tracker as https://github.com/DynamoRIO/dynamorio/issues/1435.  If someone has time to contribute to that feature it would be appreciated.

On Thu, Jan 19, 2023 at 4:06 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Greetings,

I use drcov to generate a coverage report. I convert this to lcov using drcov2lcov tool, then use genhtml to generate a html report. This reports only shows the called source code lines but not the number functions called (it's always 0/0). Is there a special parameter in drcov or drcov2lcov that generates this info?

Regards

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/ec7775a0-4019-48ec-8e21-d7bbd08da857n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/9de08f07-2a99-41ae-bd88-cf4ea55fddb4n%40googlegroups.com.

[Edgar Neubauer]

As stated, the problem is only on Aarch64. The missing lines are those that would be called in other threads.

[Edgar Neubauer]

Okay, I know the addresses the function has in the raw binary. How do I obtain the covered addresses from the raw binary log file? Or do I need dump_text?

[Edgar Neubauer]

Okay, with dump_text the address of the log output is mentioned in the log file:

module[   0]: 0x000000000151dd74,     8

Does this mean coverage? If yes, how can I check this for dump_binary?

[Edgar Neubauer]

Okay, I also find the address in a binary generated log file as 00 74 DD 51 01 08 . But drcov2lcov does not generate a hit in the function. It seems to open the right files for the symbols though, I can observe it opening the app binary in strace.

[Edgar Neubauer]

Is it a problem to use the X64 drcov2lcov with the Aarch64 binary log file and symbols?

[Derek Bruening]

On Tue, Jan 24, 2023 at 6:45 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Is it a problem to use the X64 drcov2lcov with the Aarch64 binary log file and symbols?

Hmm, my first reaction is that if the bitwidth is the same then the ELF access should work -- but I am not sure whether the DWARF lookup would be different in some way due to maybe relocation differences or something.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/d911ec9e-7268-490a-958d-26ff85ae12a5n%40googlegroups.com.

[Derek Bruening]

Did you run drcov2lcov with -verbose 5 or somesuch to see all the line addresses it checked against recorded executed addresses and look for the target address?

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/287ca258-dfb7-44d0-8298-ea67e6c02eb7n%40googlegroups.com.

[Edgar Neubauer]

Hmm ok the path in the module table seems to point to the right binary but the source code address it pulls from 0x000000000151dd74 is very different from what addr2line says on the same file.

[Edgar Neubauer]

[DRCOV2LCOV] INFO(3):    module table 0x5557299eae80, 63926272

[DRCOV2LCOV] INFO(4):    Create module table 0x5557299eae80 for module /media/user/SSD1TB/dynamoriotest/Symbols/usr/bin/myapp
[DRCOV2LCOV] INFO(5):    Module: 1, 0x1bb000, /usr/bin/myapp
[DRCOV2LCOV] INFO(2):    Substituting |/media/user/SSD1TB/dynamoriotest/Symbols/usr/bin/myapp| for |/usr/bin/myapp|
...

22418634-[DRCOV2LCOV] INFO(5):    lookup 0x151dd74 in module table 0x5557299eae80
22418635-[DRCOV2LCOV] INFO(5):    skip: [DRCOV2LCOV] INFO(5):    < very wrong sourcecode location >, 152, 0x151dd74

[Derek Bruening]

I would suggest double-checking the binary given the "Substituting" log indicating you told it to swap in a different path, double-checking any separate debug info file paths (or is the debug info in the same file?), and then checking multiple other source line to address mappings to see whether all differ or just one, and finally you would want to walk the DWARF info to get into details: but addr2line and the libelftc used by drcov2lcov should be reading the same DWARF info so a binary or separate debuginfo file path difference seems the most likely explanation.  Another sanity check could be comparing addr2line to the symquery tool which uses the same libelftc -- it is part of the drmemory repo but is a general tool allowing for various symbol queries using the same library as drcov2lcov.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/6e3b8f2e-d69d-40f0-8be1-6bdec09f4babn%40googlegroups.com.

[Edgar Neubauer]

Indeed,

dynamorio/bin64/symquery -v  -e /media/user/SSD1TB/dynamoriotest/Symbols/usr/bin/myapp -a 0x000000000151dd74

and

addr2line -i -f -e /media/user/SSD1TB/dynamoriotest/Symbols/usr/bin/myapp 0x000000000151dd74

Return different things. addr2line gives the same result as GDB.

[Derek Bruening]

First please double-check that you are passing the right value to "symquery -a": as the help message says it takes an offset "address relative to module base" while addr2line takes an absolute address as if the library were loaded at its preferred base (known as the "ELF address").  These will only be the same for an executable if it's PIE with a 0 preferred base.  (As cross-platform tools all the DR-based tools use a simple offset from the module base, which is easier to deal with than the "ELF address" which requires knowing the preferred base and performing conditional computations depending on whether it was loaded there or not.)

Next: this is the x86-64 symquery and addr2line?  (Earlier there were messages about some problem only happening on AArch64.)

Next, please check what format your debug info is in: is it DWARF2?  That is supposed to be fully supported by libelftc and is what the drsyms library is documented as supporting.  I have not checked what happens with DWARF3: whether it is supported by libelftc, whether it would fail up front, or whether it could possibly have mysterious silent inaccuracies (seems unlikely though).

I would also do some sanity checks by looking up some functions by name with symquery.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/304eb9f8-343e-4849-b55c-272ba3605ef8n%40googlegroups.com.

[Edgar Neubauer]

The module is the main exe so I assumed there is no offset involved. How would I find this out at runtime? This is the output from /proc/pid/maps:

cat /proc/3797/maps
00400000-040f7000 r-xp 00000000 b3:06 477                                /usr/bin/myapp
040f7000-04107000 ---p 00000000 00:00 0 
04107000-04202000 r--p 03cf7000 b3:06 477                                /usr/bin/myapp
04202000-04214000 rw-p 03df2000 b3:06 477                                /usr/bin/myapp

Yes, I used the X64 symquery and addr2line. There is no aarch64 version of symquery I think although there is an aarch64 version of drcov2lcov.

I cross-compile for Aarch64. The my app binary and its symbols are for Aarch64. 

Readelf says dwarf version 4 from Compilation Unit.

[Edgar Neubauer]

Question: addr2line may return multiple source code file positions for inlined functions. symquery only seems to resolve one. How does drcov2lcov handle this?

[Edgar Neubauer]

In other words: Will the inline be unwinded correctly and all source code positions get a hit or only the one on the top?

The symquery address would be 0x151dd74 - 0x400000 = 0x111dd74 right

[Derek Bruening]

On Thu, Jan 26, 2023 at 6:22 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Question: addr2line may return multiple source code file positions for inlined functions. symquery only seems to resolve one. How does drcov2lcov handle this?

You can see for yourself that drcov2lcov walks the lines, not the functions: https://github.com/DynamoRIO/dynamorio/blob/master/clients/drcov/postprocess/drcov2lcov.cpp#L1176.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/25bc0e52-a772-453d-ac2b-aed40980d63en%40googlegroups.com.

[Edgar Neubauer]

Where is the program counter mapped to the line? And will it map to multiple lines in the case of inline unwinding? 

[Derek Bruening]

In the previously linked drcov2lcov code you can see that each line's address is obtained from DWARF info: https://github.com/DynamoRIO/dynamorio/blob/master/ext/drsyms/drsyms_dwarf.c#L413.  See also symquery --lines which I think should print out the same info.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/6c92f797-4325-4e0a-b339-ffdc8e5ac0f6n%40googlegroups.com.

[Edgar Neubauer]

Ok, made another run for drcov2lcov. Verbose level 5 gave me some output for the right source code line in the right file. But all of them have the mark "skip".

Does that mean there is no hit for the corresponding address in the log file?

I took one of the addresses and searched for it in the recorded *.log file and it was present.

Any more ideas on how to debug this?

[Edgar Neubauer]

Addendum:  symquery also shows the line with the corresponding address.

[Edgar Neubauer]

Ok, to state it again:

The address is listed in module[0] when I use drove with -dump_text. drcov2lcov walks though the dwarf lines and eventually finds a line with this address. But module_table_Bb_lookup which is used in drcov2lcov.cpp cannot find it in the module table for some reason.

p table->bb_table.bitmap[ info-°line_addr / 8 ] yields 0 for some reason.

[Edgar Neubauer]

drcov, not drove

[Edgar Neubauer]

Ok, it seems that after almost 250000 bb entries drcov2lcov just stops reading from the input log file. I have no idea why.

[Edgar Neubauer]

I wonder if something is wrong with the size entry of the BB Table: entry in the log file. It says it has about 250000 elements. One bb_entry_t has a size of 8 byte. That means we have about 2 MB of bbt entries. But the whole file is about 8 MB in size. When I create a run with dump_text it also shows around 250000 bb Table entries but the file has about 1000000 lines. 

Is it right that one bbt entry has one line in the dump_text log? If yes, then the bb Table size entry is wrong by about factor 4.

[Derek Bruening]

On Mon, Jan 30, 2023 at 7:02 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Ok, to state it again:

The address is listed in module[0] when I use drove with -dump_text. drcov2lcov walks though the dwarf lines and eventually finds a line with this address. But module_table_Bb_lookup which is used in drcov2lcov.cpp cannot find it in the module table for some reason.

If the address is not found, then it was not executed and so was not covered.  If drcov2lcov only considers the start address of the line, that does seem like a potential problem if the line contains subsequent basic blocks that can be reached without executing the first block the line maps to.  No idea how often that might happen though: requires unusual code with multiple statements stacked on one line?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/e878773d-301d-4ffa-8420-77fb1730ed10n%40googlegroups.com.

[Edgar Neubauer]

Is it normal that the BB table entry count in the log file is lower than the actual number of addresses / lines  in the file?

[Edgar Neubauer]

Do your tests include large programs that may include hundreds of thousands bb table entries?

[Edgar Neubauer]

Tried some runs with -dump_text on Aarch64 with the same binary. With short runs and few samples, the table size at the beginning of the file is correct. Once it pushes the million, it becomes incorrect. I did not have this on an X64 run with the same app.

Maybe some variable overflowing on Aarch64?

Derek Bruening schrieb am Dienstag, 31. Januar 2023 um 00:59:58 UTC+1:

[Derek Bruening]

There appears to be a format string error with a 32-bit output type for a 64-bit entry count for -dump_text: https://github.com/DynamoRIO/dynamorio/blob/master/ext/drcovlib/drcovlib.c#L142
This would not affect binary output however.

I filed https://github.com/DynamoRIO/dynamorio/issues/5845

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/f9a1c229-4284-483f-8da7-79d120e6d330n%40googlegroups.com.

[Edgar Neubauer]

Well, at least we have one thing for starters...

My logs with dump_binary also include "BB Table:"  . Is this from a different function where the format string is correct?

[Edgar Neubauer]

Btw can you tell me the cmake call for the Aarch64  cronbuild? I try to link against libc 2.31 but it always uses 2.34.

[Derek Bruening]

On Wed, Feb 1, 2023 at 3:56 AM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Well, at least we have one thing for starters...

My logs with dump_binary also include "BB Table:"  . Is this from a different function where the format string is correct?

Ah yes, that text line looks like it is used by the binary format too, so it is not just the text.

However, this format specifier mismatch does not seem like it could cause any behavior change,

as there is no subsequent specifier.  On a big-endian machine you would expect it to always get

the count wrong but little-endian should be fine.  So it doesn't seem like this can explain what

you are seeing; especially with different behavior at ~a million as that is not a 32-bit boundary.

But maybe you could patch in https://github.com/DynamoRIO/dynamorio/pull/5854 just to see.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/7f7b56e7-e828-47cd-ae80-bd7fc5d72f95n%40googlegroups.com.

[Edgar Neubauer]

I would love to try but I have problems with building because it always tries to link against libc 2.34 but my target has 2.31. How would one build for it to be as flexible as your cronbuild?

[Edgar Neubauer]

Does drcov have problems with non standard program exit functions like _exit ?

Derek Bruening schrieb am Sonntag, 5. Februar 2023 um 01:18:28 UTC+1:

[Edgar Neubauer]

- Tried to start drrun with gdb but info shared lists no libraries and breakpoints don't work even after loading symbols.

- How can I manually build for Aarch64 so it links against the right libc version? The Aaatch64 toolchain file always gives me the wrong glibc.

Derek Bruening schrieb am Sonntag, 5. Februar 2023 um 01:18:28 UTC+1:

[Derek Bruening]

On Sun, Feb 5, 2023 at 4:24 PM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

I would love to try but I have problems with building because it always tries to link against libc 2.34 but my target has 2.31. How would one build for it to be as flexible as your cronbuild?

Do you mean the drrun target pulls in some versioned symbol in 2.34?  Those kinds of things have no nice solution; easiest is to build with an older toolchain.
In this case we don't care about the drrun being new: you can use the cronbuild drrun but a recent libdrcovlib.so.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/bdfac8c7-b6ca-4204-9388-5ce7bd30e54fn%40googlegroups.com.

[Derek Bruening]

On Mon, Feb 6, 2023 at 4:03 PM 'Edgar Neubauer' via DynamoRIO Users <dynamor...@googlegroups.com> wrote:

Does drcov have problems with non standard program exit functions like _exit ?

No, being terminated by an agent outside of the process is the only issue: for that, the -nudge_kills option is on by default and covers some common such scenarios.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/d8e938aa-cfb0-48e9-aea5-8353db3178c5n%40googlegroups.com.

[Edgar Neubauer]

I tried the latest cronbuild, the count still seems to be wrong in the output file.

How about adding an assert in debug mode that compares the bb table entry count with the count of entries in the vector and check if it is always consistent?

[Derek Bruening]

Happy to look at a pull request with such added checks.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/e5adbbb2-6ab1-44a5-a735-2bbbbf392eb7n%40googlegroups.com.

[Edgar Neubauer]

Hard to do if I cannot build.

[Edgar Neubauer]

Is it possible to build without libunwind?

Derek Bruening schrieb am Donnerstag, 16. Februar 2023 um 16:54:36 UTC+1:

[Edgar Neubauer]

I try to get it compiled, but I get a cmake error.

DR_DO_NOT_DEFINE_bool.c  fails to compile in line 27 because "error: 'bool' undeclared here (not in a function)

Derek Bruening schrieb am Donnerstag, 16. Februar 2023 um 16:54:36 UTC+1:

[Edgar Neubauer]

Ok, here another log from the failing compile maybe that helps:

Derek Bruening schrieb am Donnerstag, 16. Februar 2023 um 16:54:36 UTC+1:

[Edgar Neubauer]

Greetings, I finally managed to compile dynamorio for my target and have some new info regarding the too low BB table number. Apparently, the resulting drcov* file has two table entries:

DRCOV VERSION: 3
DRCOV FLAVOR: drcov-64
Module Table: version 5, count 211
Columns: id, containing_id, start, end, entry, offset, preferred_base, path
<...>
drcov.myapp.14722.0000.proc.log:216:BB Table: 256053 bbs
<...>
VERSION: 3
DRCOV FLAVOR: drcov-64
Module Table: version 5, count 221
Columns: id, containing_id, start, end, entry, offset, preferred_base, path
<...>
drcov.myapp.14722.0000.proc.log:5860:BB Table: 602217 bbs
<...>

I assume that the second table is not processed by drcov2lcov for some reason.

Regards

[Edgar Neubauer]

Are there supposed to be two tables anyways? I don't use per thread data.