[llvm-dev] Adding llvm-undname to the llvm-cov bot

[Nico Weber via llvm-dev]

Hi Vedant and Chris,

is the config for http://lab.llvm.org:8080/coverage/coverage-reports/index.html public somewhere? If so, where? (I looked in zorg but didn't find it.)

If not, could you add "llvm-undname" to the list of binaries passed to llvm/utils/prepare-code-coverage-artifact.py so that llvm/lib/Demangle/MicrosoftDemangle.cpp (and friends) show up? (If the config is public, I can send you a patch.)

Also, is there a reason llvm/utils/prepare-code-coverage-artifact.py doesn't have +x set, or is that just an oversight?

Thanks,
Nico

[Nico Weber via llvm-dev]

Vedant or Chris: Ping :)

[Chris Bieneman via llvm-dev]

Hey Nico,

I'm actually not sure where the configurations for that bot are stored. I suspect Duncan may have a better idea.

I'm reasonably certain that the missing +x is just an oversight.

-Chris

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Duncan Exon Smith via llvm-dev]

+Chris Matthews, do you know where the configs are stored for this?

[Chris Matthews via llvm-dev]

Probably this job:

lab.llvm.org:8080/green/job/clang-stage2-coverage-R/

💬 from 📱

[via llvm-dev]

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

As for +x permissions on llvm/utils/prepare-code-coverage-artifact.py, it's an oversight that they are missing. I wasn't able to land a permissions change via the new monorepo (I get: "Committed c3b9398d101 to svn", but the commit does not appear). Perhaps you'll have better luck?

best,

vedant

[Nico Weber via llvm-dev]

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html). What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname). I figured what the bot does is run `check-llvm` and then pass all binaries that run as part of `llvm-check` to the report generation script, and I had assumed llvm-undname was just missing on that list of all binaries. But maybe that's not how that coverage list is computed?

 

As for +x permissions on llvm/utils/prepare-code-coverage-artifact.py, it's an oversight that they are missing. I wasn't able to land a permissions change via the new monorepo (I get: "Committed c3b9398d101 to svn", but the commit does not appear). Perhaps you'll have better luck?

I couldn't figure out how to do it via git-svn / `git-llvm push` either, but I added the +x bit in r362561 using an old svn checkout I had lying around.

[via llvm-dev]

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

I figured what the bot does is run `check-llvm` and then pass all binaries that run as part of `llvm-check` to the report generation script, and I had assumed llvm-undname was just missing on that list of all binaries. But maybe that's not how that coverage list is computed?

The bot runs check-llvm, but passes a predefined list of binaries to llvm-cov to save time. llvm-undname was missing in the list of binaries to group under the 'all' and 'llvm' entries.

As for +x permissions on llvm/utils/prepare-code-coverage-artifact.py, it's an oversight that they are missing. I wasn't able to land a permissions change via the new monorepo (I get: "Committed c3b9398d101 to svn", but the commit does not appear). Perhaps you'll have better luck?

I couldn't figure out how to do it via git-svn / `git-llvm push` either, but I added the +x bit in r362561 using an old svn checkout I had lying around.

Thanks!

vedant

[Nico Weber via llvm-dev]

On Wed, Jun 5, 2019 at 1:33 PM <v...@apple.com> wrote:

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html , that's great!

Is there a way to see which revision the report was built at?  That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?

[Nico Weber via llvm-dev]

On Thu, Jun 6, 2019 at 12:56 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, Jun 5, 2019 at 1:33 PM <v...@apple.com> wrote:

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html , that's great!

Is there a way to see which revision the report was built at?  That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?

Even more mysteriously, in the "all" report (http://lab.llvm.org:8080/coverage/coverage-reports/all/index.html), MicrosoftDemangle.cpp's line coverage is just  45.21% (967/2139).

I assumed that "llvm" is the result of `check-llvm` and "all" is `check-all`. Both of those should run the tests in llvm/test/Demangle, but it feels like that doesn't happen in either case. Can you think of a reason for why the bot wouldn't run the tests in llvm/test/Demangle?

[via llvm-dev]

On Jun 6, 2019, at 9:56 AM, Nico Weber <tha...@chromium.org> wrote:

On Wed, Jun 5, 2019 at 1:33 PM <v...@apple.com> wrote:

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html , that's great!

Is there a way to see which revision the report was built at?

For now, it's possible to look up this up via the Jenkins job status page and clicking on the description of the latest successful job (e.g yesterday's #4089). Ideally the svn revision would be included in the report, but I haven't worked out the right steps to take in Jenkins to make that happen. This is something I'd like to tackle during the monorepo transition.

  That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?

The discrepancy may be due to an llvm-cov limitation when processing multiple binaries: it ignores coverage records from a binary if it has a name that's already been seen.

vedant

[via llvm-dev]

On Jun 7, 2019, at 6:47 AM, Nico Weber <tha...@chromium.org> wrote:

On Thu, Jun 6, 2019 at 12:56 PM Nico Weber <tha...@chromium.org> wrote:

On Wed, Jun 5, 2019 at 1:33 PM <v...@apple.com> wrote:

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html , that's great!

Is there a way to see which revision the report was built at?  That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?

Even more mysteriously, in the "all" report (http://lab.llvm.org:8080/coverage/coverage-reports/all/index.html), MicrosoftDemangle.cpp's line coverage is just  45.21% (967/2139).

That's strange. You might have hit a race condition while the bot was busy uploading new reports? Currently, under the "all" and "llvm" reports, MicrosoftDemangle.cpp's line coverage is 74%.

I assumed that "llvm" is the result of `check-llvm` and "all" is `check-all`. Both of those should run the tests in llvm/test/Demangle, but it feels like that doesn't happen in either case. Can you think of a reason for why the bot wouldn't run the tests in llvm/test/Demangle?

There's just one set of tests which run once, and I can't think of any reason why tests in llvm/test/Demangle wouldn't run.e

vedant

[Nico Weber via llvm-dev]

On Mon, Jun 10, 2019 at 2:11 PM <v...@apple.com> wrote:

On Jun 6, 2019, at 9:56 AM, Nico Weber <tha...@chromium.org> wrote:

On Wed, Jun 5, 2019 at 1:33 PM <v...@apple.com> wrote:

On Jun 4, 2019, at 4:41 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Jun 3, 2019 at 2:06 PM <v...@apple.com> wrote:

Hi Nico,

Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.

Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html).

For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry.

What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).

This should be fixed on the next successful run.

Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html , that's great!

Is there a way to see which revision the report was built at?

For now, it's possible to look up this up via the Jenkins job status page and clicking on the description of the latest successful job (e.g yesterday's #4089). Ideally the svn revision would be included in the report, but I haven't worked out the right steps to take in Jenkins to make that happen. This is something I'd like to tackle during the monorepo transition.

  That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?

The discrepancy may be due to an llvm-cov limitation when processing multiple binaries: it ignores coverage records from a binary if it has a name that's already been seen.

Thanks, that sounds like a good explanation. Is there a PR for this problem?

[Max Moroz via llvm-dev]

Hmm, I think we've fixed that behavior in https://reviews.llvm.org/D46478. For instance, we run 400+ binaries in Chrome, each having its own LLVMFuzzerTestOneInput function, and we're able to combine all .profdata together and generate an aggregate report.

However, it feels like there is some other condition which may still lead to a collision. We've hit it in OSS-Fuzz recently (https://github.com/google/oss-fuzz/issues/2330), but the developer ended up changing the sources of the binaries and the issue went away.

Where are the logs from the buildbot? Are there any warnings reported like the following one:

warning: /out/dumps/simple_round_trip.profdata: Function basic block count change detected (counter mismatch)

[via llvm-dev]

On Jun 10, 2019, at 12:45 PM, Max Moroz <mmo...@chromium.org> wrote:

Hmm, I think we've fixed that behavior in https://reviews.llvm.org/D46478. For instance, we run 400+ binaries in Chrome, each having its own LLVMFuzzerTestOneInput function, and we're able to combine all .profdata together and generate an aggregate report.

Oh, thanks for pointing that out. This should cover the common case, excepting symbols like "main" that really do create ambiguity.

However, it feels like there is some other condition which may still lead to a collision. We've hit it in OSS-Fuzz recently (https://github.com/google/oss-fuzz/issues/2330), but the developer ended up changing the sources of the binaries and the issue went away. Where are the logs from the buildbot? Are there any warnings reported like the following one:

warning: /out/dumps/simple_round_trip.profdata: Function basic block count change detected (counter mismatch)

The logs should be visible via the Jenkins job. Here's one: http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/4092/consoleText (I do see a 'counter mismatch' warning here).

Coverage for a symbol is dropped if there's a mismatch between the hash recorded in the profile and the hash recorded in the code coverage blob. Typically this occurs when the profile is out of sync with the build products, but that's not the case on the bot. It's also possible for there to be conflicting definitions of a function present in different binaries. These definitions can have different numbers of counters or AST hashes due to source-level differences. For this case, I think it's reasonable to not display coverage.

vedant