Feature order in CROSSTOOL

[Tim Blakely]

I've been toying around with getting the Xtensa lx106 toolchain working in Bazel. I'm pretty close, but it looks like the the archiver doesn't like the default flags of "rcsD" and instead needs "cru". 0.5.2 appears to have removed ar_flag in favor of setting up action_configs, but I'm not having much luck in my attempts to get it working. 

The original .params file for the default archiving step looks like this:

rcsD

<target>.a

<file>.o

I tried adding a feature to the compiler based off the msvc CROSSTOOL file:

  feature {

    name: 'archiver_flags'

    flag_set {

      action: 'c++-link-static-library'

      action: 'c++-link-alwayslink-static-library'

      action: 'c++-link-pic-static-library'

      action: 'c++-link-alwayslink-pic-static-library'

      flag_group {

        flag: 'cru'

      }

    }

  }

The changes appeared to be picked by by Blaze, but the .param file got a bit mangled in the process:

<file>.o

cru

In addition to the archive disappearing, the updated archiver_flags are inserted after the object files instead of before. Apparently the ar binary that ships with the Xtensa toolchain is particular about parameter order (not sure if the wider gnu toolset is the same). My next attempt was to add output_execpath that I found in the default action config in CppActionConfigs.java:

  feature {

    name: 'archiver_flags'

    expand_if_all_available: 'output_execpath'

    flag_set {

      action: 'c++-link-static-library'

      action: 'c++-link-alwayslink-static-library'

      action: 'c++-link-pic-static-library'

      action: 'c++-link-alwayslink-pic-static-library'

      flag_group {

        flag: 'cru'

        flag: '%{output_execpath}'

      }

    }

  }

Gets closer, but the object files are still ahead of the flags:

<file>.o

cru

<file>.a

I tried overriding the action_config for c++-link-static-library, but blaze fails to parse the CROSSTOOL file with "multiple action configs for action 'c++-link-static-library'." 

So I'm a bit stumped. Am I missing something else? Does the "implies" field in action_config determine order of feature flags? If so, why does overriding archiver_flags a change the order of the base action_config? If not, is there another way to control feature order?

[Irina Iancu]

+Marcel

[Marcel Hlopko]

Hi Tim,

welcome to the crosstool sufferers anonymous club :) Hope you didn't suffer for too long before asking here.

1. Features are expanded and put into command line in the order they appear in the CROSSTOOL. So to put 'cru' before .o files, you need to put 'archiver_flags' feature above the feature that iterates libraries_to_link. If you don't iterate anything like that, I'm surprised that the archiver_flags appear after .o files (and we have an internal crosstool in which they appear before). Can you share your crosstool in that case?

2. You probably want to use flag: '%{output_execpath}' there (also msvc crosstool does) - that's where archiver finds out the output filename.

3. Regarding multiple action configs error, that's suspicious, what bazel version do you use? Anyway, if you start adding action_configs, I strongly suggest to add all of them (all of them are present in CppActionConfigs, check version at HEAD), then also check features that get added in CppConfiguration.java and take those that are relevant to your toolchain, and then add "no_legacy_features" feature, so Bazel will stop trying to be smart and will not add duplicated action_configs. The bad news is that I'm doing many cleanups in this area (introducing new build variables such as output_execpath so all command line building is expressed in the crosstool as bazel stays platform independent) and will probably break you 3-4 times until it's finished. I don't think fixes will be hard, you will just copy paste the change from CppActionConfigs over to your crosstool. If you don't want to do this now (you'll have to eventually though), don't define any action_configs, and keep archiver_flags feature there. Bazel is smart to use it when it finds it. But it might also stop working at some point.

Hope I made at least something clearer, keep the questions coming :)

 

--

-- 

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Tim Blakely]

Heya Marcel!

Ha! Good club to be in ;) Thanks for the reply. To be clear: I'm just trying to get the xtensa-lx106 toolchain up since I loathe the Arduino IDE workflow, so it's just a hobby for now :) I had some success back in March getting the xtensa-lx6 toolchain working and managed to flash a few ESP32 chips, so I know it's at least was possible to do! :)

1. Happy to share the CROSSTOOL file, but let me get it in a non-hacky place before I do. Since I posted, I ended up going... a bit deeper than I intended, heh. I looked at cc_configure and ended up writing a skylark rule that generates the CROSSTOOL file automatically a la windows_cc_configure. This automatically generates the CROSSTOOL features and action_configs based on the specifications of the cpu specified by the --cpu flag. I've gotten it working to the point where I can generate elf/bin files, but they're slightly different than the ones produced by the standard SDK - specifally the .rodata.str1.1 sections of the resulting binaries - but I think that's in my court to figure out.  

2. Yup, figured that out, but I also had to disable another feature that's on by default to keep it from using it earlier (exactly which one escapes me at the moment).

3. Not sure if the internal google package repo for bazel that's screwed up or the 0.5.2 release itself, but it appears it doesn't include the actionsAreConfigured change that skips the default actions if "c++" is defined somewhere. Cloning and building at HEAD picked up the change.  And don't worry about breakage; this is me just toying around with it and it looks like CROSSTOOL is going to be under pretty heavy flux in the next quarter or so. If I can just it working for now, I'll just stay at that bazel release for a while until the CROSSTOOL changes settle down.

So given that I got it (somewhat) working, the only real headaches I've had when writing/generating the CROSSTOOL are:

a) There's no built-in rule to read in a file as a string. I had to do a combination of cepository_ctx.execute, cat, and stripping/returning stdout.

b) Finding external include directories. repository_ctx has the "path" function, but only supports either relative paths to the current repository or labels to another target. This makes it painful on two fronts: it's impossible to get a path to a directory where another label is located e.g. for compiler includes since the compiler is actually in its own WORKSPACE rule, and you can't depend on a filegroup() rule declared in the external compiler's new_http_rule since bazel includes the full subpath. The latter can be solved by adding a new cc_library() that includes only the header files and a strip_prefix argument, but that complicates the new_http_archive's BUILD rules quite a bit. The former requires targeting a single file within the desired directory and the running a "dirname" command on it; also less than ideal

c) Templates must be written to a file; there's no way to read a file, template it, and store it as a string. I've combined the rule from a) with a .format() call, but it'd be nice to have that baked into repository_ctx.

d) Pretty sure I'd sell my soul for a regex string-matching function in [repository_]ctx... ;)

So onto more CROSSTOOL questions!

- The old CROSSTOOL "docs"* suggest writing wrappers to include things like ld_library_path, etc. Do you expect these to be necessary down the road, given that more and more of the configuration will be in the CROSSTOOL file itself?

- I found out about output_execpath via rooting around j/c/g/devtools/build/lib/rules/cpp, and there appear to be a bunch of others sprinkled around those source files like "libraries_to_link", the various compilation stages like "c++-header-parsing","c++-module-compile" and "preprocess-assemble", etc. Is there a list somewhere? If not, mind generating one or pointing me to which files have important templated vars?

- Is there a way to get bazel to dump the entire, post-templated, post-processed, this-is-the-final-proto-its-using-to-compile-all-the-things CROSSTOOL file?

Thanks, and keep up the great work!

-Tim

* Sorry, couldn't help a tease... ;) Even crosstool_config.proto razzes on the lack of docs, though with CROSSTOOL changing rapidly in the near future I totally understand not wanting to write them until they're stable :)

[Marcel Hlopko]

Hi Tim,

comments inline.

On Wed, Jul 5, 2017 at 9:40 PM Tim Blakely <bla...@google.com> wrote:

Heya Marcel!

Ha! Good club to be in ;) Thanks for the reply. To be clear: I'm just trying to get the xtensa-lx106 toolchain up since I loathe the Arduino IDE workflow, so it's just a hobby for now :) I had some success back in March getting the xtensa-lx6 toolchain working and managed to flash a few ESP32 chips, so I know it's at least was possible to do! :)

Very cool!

 

1. Happy to share the CROSSTOOL file, but let me get it in a non-hacky place before I do. Since I posted, I ended up going... a bit deeper than I intended, heh. I looked at cc_configure and ended up writing a skylark rule that generates the CROSSTOOL file automatically a la windows_cc_configure. This automatically generates the CROSSTOOL features and action_configs based on the specifications of the cpu specified by the --cpu flag. I've gotten it working to the point where I can generate elf/bin files, but they're slightly different than the ones produced by the standard SDK - specifally the .rodata.str1.1 sections of the resulting binaries - but I think that's in my court to figure out.  

So you generate the crosstool depending on --cpu value? How we expected the crosstool to work is to have toolchain definitions for all possible cpus, and then --cpu flag would select one of them. The problem is that if you specify one --cpu flag, use that to generate the crosstool, and then use different --cpu flag, the crosstool won't be regenerated. We use environment variables to customize generation, changes to those will trigger regeneration.

 

2. Yup, figured that out, but I also had to disable another feature that's on by default to keep it from using it earlier (exactly which one escapes me at the moment).

3. Not sure if the internal google package repo for bazel that's screwed up or the 0.5.2 release itself, but it appears it doesn't include the actionsAreConfigured change that skips the default actions if "c++" is defined somewhere. Cloning and building at HEAD picked up the change.  And don't worry about breakage; this is me just toying around with it and it looks like CROSSTOOL is going to be under pretty heavy flux in the next quarter or so. If I can just it working for now, I'll just stay at that bazel release for a while until the CROSSTOOL changes settle down.

Current behavior @HEAD is that the crosstool is patched unless you define 'no_legacy_features' feature. Ideally, bazel wouldn't patch crosstool at all, but since I'm doing many changes there, it's easier to do this logic in bazel and do a big migration step when it's all done. Note also that if you do define 'no_legacy_features' it's very possible that my work will break you. Sorry in advance for that :)

 

So given that I got it (somewhat) working, the only real headaches I've had when writing/generating the CROSSTOOL are:

a) There's no built-in rule to read in a file as a string. I had to do a combination of cepository_ctx.execute, cat, and stripping/returning stdout.

Sounds like a good feature request, would you mind creating an issue on github and mention me?

 

b) Finding external include directories. repository_ctx has the "path" function, but only supports either relative paths to the current repository or labels to another target. This makes it painful on two fronts: it's impossible to get a path to a directory where another label is located e.g. for compiler includes since the compiler is actually in its own WORKSPACE rule, and you can't depend on a filegroup() rule declared in the external compiler's new_http_rule since bazel includes the full subpath. The latter can be solved by adding a new cc_library() that includes only the header files and a strip_prefix argument, but that complicates the new_http_archive's BUILD rules quite a bit. The former requires targeting a single file within the desired directory and the running a "dirname" command on it; also less than ideal

Didn't play enough with skylark repos to know right away what you mean, could you file an issue and elaborate there?

 

c) Templates must be written to a file; there's no way to read a file, template it, and store it as a string. I've combined the rule from a) with a .format() call, but it'd be nice to have that baked into repository_ctx.

Another feature request? :)

 

d) Pretty sure I'd sell my soul for a regex string-matching function in [repository_]ctx... ;)

We all would :) It's not present in Skylark rules by design, but in repository rules it shouldn't hurt. So feature request :)

 

So onto more CROSSTOOL questions!

- The old CROSSTOOL "docs"* suggest writing wrappers to include things like ld_library_path, etc. Do you expect these to be necessary down the road, given that more and more of the configuration will be in the CROSSTOOL file itself?

Nope, I'd like to get rid of all that boilerplate.

 

- I found out about output_execpath via rooting around j/c/g/devtools/build/lib/rules/cpp, and there appear to be a bunch of others sprinkled around those source files like "libraries_to_link", the various compilation stages like "c++-header-parsing","c++-module-compile" and "preprocess-assemble", etc. Is there a list somewhere? If not, mind generating one or pointing me to which files have important templated vars?

There's no list, it's all too hot right now, I'm still extracting hardcoded gcc-style flags and intoducing new build variables. Once I'm done, I'll make them more discoverable and documented. Until then, the best place is to look at https://github.com/bazelbuild/bazel/blob/6ff634d3f3582c74190a5dd5051a4b0253aec604/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java 

- Is there a way to get bazel to dump the entire, post-templated, post-processed, this-is-the-final-proto-its-using-to-compile-all-the-things CROSSTOOL file?

Nope, intentionally :) We have such feature internally at Google, and we regret introducing it. People started depending on it, they parse this crosstool and do weird stuff with it. This makes it hard to evolve this. Also, we're still not sure proto is the best format for crosstool, and have plans to experiment with having the crosstool written in skylark.

 

Thanks, and keep up the great work!

-Tim

* Sorry, couldn't help a tease... ;) Even crosstool_config.proto razzes on the lack of docs, though with CROSSTOOL changing rapidly in the near future I totally understand not wanting to write them until they're stable :)

:) I'm aware. It's very very very painful to write crosstool with the amount of documentation provided. The plan is to finish my work asap, polish it, try skylark, and when done write the documentation that will last and we won't be ashamed of. We're talking end of this year or early next year.

Thanks for your feedback (and patience with crosstool), feel free to ask more, and let's discuss your feature requests on github.

Cheers!

[setr...@gmail.com]

Hi Tim, would you please share how you fix this problem? I come across the same problem. In my CROSSTOOL file I write only on feature for ios cross build on a ubuntu x86_64 server :

feature {
name: 'archiver_flags'
flag_set {
action: 'c++-link-static-library'
action: 'c++-link-alwayslink-static-library'
action: 'c++-link-pic-static-library'
action: 'c++-link-alwayslink-pic-static-library'
flag_group {

flag: '-r'
flag: '%{output_execpath}'
}
expand_if_all_available: 'output_execpath'
expand_if_all_available: "uses_action_configs_for_cc_archiving"
}
}
It results in ar <object>.o -r <archive>.a. And I have no features iterates over 'libraries_to_link'. I guess this may have a default feature, but I tried to put my archiver_flags in the front of my toolchain definition section, nothing changes.

[Marcel Hlopko]

It's coming from https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java?q=CppActionConfigs, either add:

feature { name: "libraries_to_link" }

so bazel doesn't add the default one, or:

feature { name: "no_legacy_features" }

so bazel will not patch your crosstool at all.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5dc118b1-8b79-4f95-9b19-d37054054534%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-- 

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[setr...@gmail.com]

Will it affect the default build behaviors by adding 'feature { name: "libraries_to_link" }'?

I've checked out CppActionConfigs.java, and it appears to that custom feature will overwrite default ones.

在 2018年3月6日星期二 UTC+8下午4:08:52，Marcel Hlopko写道：

[Marcel Hlopko]

Oh now I see I replied to two emails dealing with the single issue.

So your problem is that flags coming from archiver_flags feature appear after libraries_to_link, is that right?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/59d10644-c5d2-4a10-95f5-b0edf20b7ad5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[setr...@gmail.com]

Yep, exactly the same problem with Tim.

在 2018年3月6日星期二 UTC+8下午4:50:09，Marcel Hlopko写道：

[Marcel Hlopko]

Then either use bazel@HEAD (since it has this cl in https://github.com/bazelbuild/bazel/commit/6d24ca921bb987155c201bbcfae641a6180c23d2) and add:


feature { name: 'archiver_flags' enabled: true

feature {
    name: 'archiver_flags'
    enabled: true

    flag_set {
      action: 'c++-link-static-library'
      action: 'c++-link-alwayslink-static-library'
      action: 'c++-link-pic-static-library'
      action: 'c++-link-alwayslink-pic-static-library'

      flag_group {
        expand_if_all_available: 'output_execpath'               flag: '-the_flag_you_need_there'
        flag: '%{output_execpath}'
      }      flag_group {        flag_group {
          expand_if_all_available: "libraries_to_link"
          iterate_over: "libraries_to_link"
          flag_group {
            expand_if_equal {
              variable: "libraries_to_link.type"
              value: "object_file"
            }
            flag: "%{libraries_to_link.name}"
          }
          flag_group {
            expand_if_equal {
              variable: "libraries_to_link.type"
              value: "object_file_group"
            }
            iterate_over: "libraries_to_link.object_files"
            flag: "%{libraries_to_link.object_files}"
          }
        }
      }    }
  }

to your crosstool,
or copy entire libraries_to_link feature and put it into your crosstool below archiver_flags from above,
or replace ar with a shell script that will reshuffle the flags.


Sorry for the troubles...

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/da0b9ab3-86ee-4c9e-aa19-55310e278dca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.Then either use bazel@HEAD (since it has this cl in https://github.com/bazelbuild/bazel/commit/6d24ca921bb987155c201bbcfae641a6180c23d2) and add:


feature { name: 'archiver_flags' enabled: true

flag_set { action: 'c++-link-static-library' action: 'c++-link-alwayslink-static-library' action: 'c++-link-pic-static-library' action: 'c++-link-alwayslink-pic-static-library'

flag_group { expand_if_all_available: 'output_execpath' flag: '-whatever_flag_is_needed' flag: '%{output_execpath}' }
flag_group { flag_group { expand_if_all_available: "libraries_to_link" iterate_over: "libraries_to_link" flag_group { expand_if_equal { variable: "libraries_to_link.type" value: "object_file" } flag: "%{libraries_to_link.name}" } flag_group { expand_if_equal { variable: "libraries_to_link.type" value: "object_file_group" } iterate_over: "libraries_to_link.object_files" flag: "%{libraries_to_link.object_files}" } } } } }to your crosstool,
or copy entire libraries_to_link feature and put it into your crosstool below archiver_flags from above,
or replace ar with a shell script that will reshuffle the flags.


Sorry for the mess

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/da0b9ab3-86ee-4c9e-aa19-55310e278dca%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[setr...@gmail.com]

Thanks very much.
That solves my problem.
Hope that 'feature' and 'action_config' wiil soon be released and documented. :-)

在 2018年3月6日星期二 UTC+8下午5:01:45，Marcel Hlopko写道：

[Marcel Hlopko]

Me too :)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5132bf1b-e889-4615-86fd-4a52b4614a31%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.