How to resolve Missing DeclaredToolchainInfo?

[Garrett Kajmowicz]

As a part of my adventures in Toolchain land, I took the bazel clang toolchain tutorial and modified for my own needs. Unfortunately, I'm now running into an error message of:

ERROR: While resolving toolchains for target //thing/to:build: invalid registered toolchain '//tools/build/product-build:my_toolchain': target does not provide the DeclaredToolchainInfo provider.

A search online shows a small number of people reporting the same issue without any resolution. I've found a few bazel dev postings about the implementation of the Java class from a few years ago, but it isn't superficially obvious to me what the system wants to be provided with.

How can I address this now? And might a more detailed error message be provided for the future?

-     Garrett

[John Cater]

Probably this means that you called register_toolchains for the cc_toolchain (which provides ToolchainInfo and CcToolchainProvider), instead of calling register_toolchain on the toolchain rule itself (which provides DeclaredToolchainInfo). The relevant documentation is at https://docs.bazel.build/versions/master/toolchains.html#registering-and-building-with-toolchains

I agree that the error is slightly confusing. Can you file an issue so that I can take a look at improving it?

Thanks,

John C

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/83a235a3-680b-4421-95cd-d6ca193aff03n%40googlegroups.com.

[Garrett Kajmowicz]

With a fresh set of eyes in the morning, I suspect that the underlying issue is that I was passing the cc_toolchain rather than the cc_toolchain_suite to register_toolchains(). At the very least, switching which one I was passing in got rid of the original error message.

Nonetheless, I still filed the ticket as you requested:
https://github.com/bazelbuild/bazel/issues/13556

This runs into another annoyance in that there doesn't seem to be a way to preserve the existing set of cc_toolchain objects. In my case, I need to output binaries for multiple platforms during a single build operation; I need to have concurrent compilers for the host and target platforms. The toolchain I'm trying to add is wrapper around a provided binary version of gcc for compiling arm64 targets. But when I create and register my toolchain_suite, I then get an error message:

ERROR: /var/tmp/garrettkajmowicz/repo_bazel/tools/build/ product-build/BUILD:27:19: in cc_toolchain_suite rule //tools/build/product-build:marvell_suite: cc_toolchain_suite '//tools/build/product-build:my_suite' does not contain a toolchain for cpu 'k8'

ERROR: While resolving toolchains for target //thing/to:build: invalid registered toolchain '//tools/build/product-build:my_suite': Analysis of target '//tools/build/nfp-firewall-build:my_suite' failed

If I then try and add back the host toolchain (yes, this is really hacky) by adding an additional item to my doing:

cc_toolchain_suite(

    name = "my_suite",

    toolchains =

    {

        "aarch64": ":my_toolchain",

        "k8": "@local_config_cc//:cc-compiler-k8",

    },

)

But then I'm back to the "target does not provide the DeclaredToolchainInfo provider" error again.

I spent some time trying to better understand by what you meant by "calling register_toolchain on the toolchain rule itself"

The best that I could manage was to alternate between the DeclaredToolchainInfo and "does not contain a toolchain for cpu 'k8'" by adding/removing a ":" character in front of the value for toolchain_config passed to cc_toolchain().

The referenced information about toolchains seemed to be a better description of how to add support for a new type of compiler altogether rather than a gcc instance for a specific platform. I referred to this tutorial:
https://docs.bazel.build/versions/master/tutorial/cc-toolchain-config.html

and these references:
https://docs.bazel.build/versions/master/be/c-cpp.html#cc_toolchain
https://docs.bazel.build/versions/master/cc-toolchain-config-reference.html

And this platform constraint seems to suggest that adding additional architectures is something which can be done as well:

https://docs.bazel.build/versions/master/be/platform.html#constraint_value_examples

Thanks again for your help,

-     Garrett

[John Cater]

You seem to be swinging back and forth between the legacy and the platform-based ways of resolving CC toolchains. I doubt that you are doing this on purpose, but it is very confusing.

Both systems define the actual CC toolchain using the cc_toolchain rule, and this describes what tools are available, what headers and libraries the system provides, etc. The difference is in how Bazel discovers which toolchain to use:

- The legacy system uses the --crosstool_top flag (or --host_crosstool_top for host builds) to find a cc_toolchain_suite target. This target is then used to find the correct cc_toolchain to use, based on the --cpu and --compiler flags (and that is, of course, --host_cpu and --host_compiler in host builds).

- The new system uses register_toolchain, and the actual `toolchain` rule, to let Bazel know "there are toolchains available for rules that want the @bazel_tools//tools/cpp:toolchain_type toolchain, they have these constraints". Bazel then matches those constraints against the target platform and the execution platforms to choose the best toolchain and execution platform for this target. Please take a look at the documentation at https://docs.bazel.build/versions/master/platforms-intro.html and https://docs.bazel.build/versions/master/toolchains.html for further details.

The errors you have received saying "in cc_toolchain_suite rule //tools/build/product-build:marvell_suite: cc_toolchain_suite '//tools/build/product-build:my_suite' does not contain a toolchain for cpu 'k8'" are for the legacy system: something is checking the value of --crosstool_top, it's pointing to //tools/build/product-build:marvell_suite, that cc_toolchain_suite doesn't contain anything for the default --cpu value of "k8", and that's your error.

The errors saying "target does not provide the DeclaredToolchainInfo provider" are from the new platforms-and-toolchains system: it means you have called `register_toolchains` in your WORKSPACE file, but the label you gave is not the actual `toolchain` rule, it is something else.

Please note that it would be an error to call `register_toolchains('//tools/build/product-build:marvell_suite')`, because `//tools/build/product-build:marvell_suite` is the `cc_toolchain_suite` rule, not the `toolchain` rule.

If you are comfortable sticking with the legacy CC toolchains, since they are still supported, remove the `register_toolchains` calls from your WORKSPACE.

I suspect the problem with your cc_toolchain_suite is that host builds are trying to build for k8 (since that's the default for --host_cpu), but you're not defining a toolchain for k8 in your cc_toolchain_suite. You can either add the k8 toolchain, like you did, or make sure that --host_crosstool_top is set to `@bazel_tools//tools/cpp:toolchain`.

Hopefully this will help clear up the problems that you are having.

John Cater

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/cf8f2d9b-7e3c-4f06-a3eb-932d338cb832n%40googlegroups.com.

[Garrett Kajmowicz]

You managed to hit the nail on the head. I was incompletely using both toolchain mechanisms which obviously results in strange behavior. I still don't have everything working (see below), but things are much, much closer. I decided instead to go with the platforms approach as it will likely address our needs better.

For posterity, for my situation:

register_toolchains() takes a labelled toolchain.

toolchain() takes a labelled cc_toolchain. This is non-obvious and I thought that cc_toolchain was effectively a derived class of toolchain, but it is not.

cc_toolchain() takes a labelled cc_toolchain_config. These can be generated via tools like:

https://github.com/bazelbuild/bazel/blob/master/tools/cpp/cc_toolchain_config.bzl

Having created platforms and constraints for everything that I am trying to use, I'm able to get the relevant constraints activated when specifying the appropriate --platforms list on the command line. It's great.

The last oddity that I'm facing is that, for some reason, the actual gcc binary being called is the one for the build system and not the one specified in the toolchain definition. It is indeed resolving the toolchain correctly - if I remove the register_toolchains() call, the build fails complaining that there's no matching toolchain for the underlying cpu. If you look at the appended output,  you can see that the compiler being called is "/usr/bin/gcc". And yet I believe I am passing in a full list of tools which are rooted in /opt/marvell-tools-249.0/bin/. I say believe as I'm uncertain if there's a way to get more information out of the toolchain system to trace the internal flow of data building up the toolchain data structures.

-     Garrett

$ bazel --output_user_root=/var/tmp/garrettkajmowicz/bazel_cache build --verbose_failures --sandbox_debug --toolchain_resolution_debug --verbose_failures --platforms=//usfp-firewall:armada tools/... usfp-firewall/usfp/mflow/usfp_hash/...

INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //usfp-firewall:armada: execution @local_config_platform//:host: Selected toolchain //tools/build/nfp-firewall-build:marvell_toolchain

INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //usfp-firewall:armada: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm, android

INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //usfp-firewall:armada: Rejected toolchain @local_config_cc//:cc-compiler-k8; mismatching values: x86_64

INFO: ToolchainResolution: Target platform //usfp-firewall:armada: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain //tools/build/nfp-firewall-build:marvell_toolchain

INFO: Analyzed 4 targets (1 packages loaded, 12 targets configured).

INFO: Found 4 targets...

ERROR: /var/tmp/garrettkajmowicz/repo_bazel/usfp-firewall/usfp/mflow/usfp_hash/BUILD:3:11: Compiling usfp-firewall/usfp/mflow/usfp_hash/usfp_hash_cuckoo.c failed: (Exit 1): linux-sandbox failed: error executing command 

  (cd /var/tmp/garrettkajmowicz/bazel_cache/38e339f273195e47ba0c555e49e21f34/sandbox/linux-sandbox/409/execroot/__main__ && \

  exec env - \

    PATH=/home/garrettkajmowicz/bin:/opt/marvell/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/storage/build_tools:/storage/pit-dev-lab \

    PWD=/proc/self/cwd \

    TMPDIR=/tmp \

  /var/tmp/garrettkajmowicz/bazel_cache/install/f95ca91ebc34d56aa0f8ad499de91220/linux-sandbox -t 15 -w /var/tmp/garrettkajmowicz/bazel_cache/38e339f273195e47ba0c555e49e21f34/sandbox/linux-sandbox/409/execroot/__main__ -w /tmp -w /dev/shm -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild/bin/usfp-firewall/usfp/mflow/usfp_hash/_objs/usfp_hash/usfp_hash_cuckoo.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/usfp-firewall/usfp/mflow/usfp_hash/_objs/usfp_hash/usfp_hash_cuckoo.pic.o' -fPIC -DALLOW_EXPERIMENTAL_API -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/marvell-dpdk -iquote bazel-out/k8-fastbuild/bin/external/marvell-dpdk -iquote external/kernel_source_armada -iquote bazel-out/k8-fastbuild/bin/external/kernel_source_armada -Ibazel-out/k8-fastbuild/bin/external/marvell-dpdk/_virtual_includes/marvell-dpdk-armada -Ibazel-out/k8-fastbuild/bin/external/kernel_source_armada/_virtual_includes/kernel_dpdk_headers -Iusfp-firewall/usfp -Iusfp-firewall/usfp/mflow/usfp_hash -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c usfp-firewall/usfp/mflow/usfp_hash/usfp_hash_cuckoo.c -o bazel-out/k8-fastbuild/bin/usfp-firewall/usfp/mflow/usfp_hash/_objs/usfp_hash/usfp_hash_cuckoo.pic.o)

[Austin Schuh]

Are you using --incompatible_enable_cc_toolchain_resolution ?  I forget exactly where we are in the c/c++ toolchain conversion, but I don't recall it being fully done.  Last time I tried, I was unable to fully switch over and filed some bugs.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/585f9441-d923-4f3d-bdb7-148c74a20586n%40googlegroups.com.

[John Cater]

I'd second Austin's question: are you actually enabling "--incompatible_enable_cc_toolchain_resolution"? Without it, toolchain resolution still happens (so a lack of toolchains matching your platforms will raise errors), but the results will be ignored.

Beyond that, I'd need to see a minimal example of your project to diagnose further, I'm afraid.

John C

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DHV_SMMyv9S%3Du8yDqtO4DqtVpAaWdJSyk2%3DPq0hFZc-gQ%40mail.gmail.com.

[Garrett Kajmowicz]

You and Austin were right about the missing "--incompatible_enable_cc_toolchain_resolution" flag. I misinterpreted the intent of that flag.

I have my toolchain now working. I'll start new questions in a new thread.

Thank you!