Variant build_requires (or, how to handle gcc build requirements?)

[Paul Molodowitch]

Hi - we've been using rez as the heart of our build system for a bit, but we've come across an issue that we don't really have a great answer for: namely, what to do if we want to build different versions of a package, with different versions of gcc?

ie, say I want to build the package stegosaurus using both gcc-4.1.2 and gcc-4.8.3.  To this point, we've been creating two variants.  However, say we also have some other package - starscream - which requires gcc-4.4.3.  Now, starscream and stegosaurus don't need to interact with each other at a binary level, so it should be fine that one was built with gcc-4.8.3, and the other with gcc-4.1.2.  However, rez now will not allow me to create an environment with both packages, because their gcc version will conflict.

Is there a "standard" solution to this issue?  It seems like, ideally, we'd have some sort of concept of "build variants" - which would be just like variants, except their requirements would only be added to the "build_requires", and not to the "requires".  Or, even better, if there were a way to mark some requests in a variant list as being build requirements - ie,

variants = [

    {"requires": ["maya-2016"]

     "build_requires": ["gcc-4.8.3"},

    {"requires": ["maya-2016"]

     "build_requires": ["gcc-4.1.2"},

    {"requires": ["maya-2015"]

     "build_requires": ["gcc-4.1.2"},

]

or perhaps:

variants = [

    ["maya-2016", "{B}gcc-4.8.3"],

    ["maya-2016", "{B}gcc-4.1.2"],

    ["maya-2015", "{B}gcc-4.1.2"],

]

Suggestions / thoughts?

[Allan Johns]

Hey Paul,

Yeah you've stumbled across one of the edge cases that rez isn't good at.

The correct fix as I see it is your first suggestion. Currently, variants are only lists of requirements. The concept of a variant needs to be extended so that it can also have its own build requirements. Further to this, a variant should be able to specify its installation subpath.

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Fede Naum]

Hi Paul, 

I think we bumped into some sort of the same issue early on when we started using rez a couple of years ago. 

I'm not 100% sure if we added the concept of "private_build_requires" or if it is standard in rez 

The difference between "private_build_requires" and "build_requires" is that the requirements listed there are not transitive

Then we split gcc and stdlib

---------------------

maya-2011 and 2012 package  have

private_build_requires:

- gcc-4.1.2

requires

- stdlib-4.1.2+<5

--------------------------

maya-2014 and 2015 package  have

private_build_requires:

- gcc-4.4.6

requires

- stdlib-4.4.6+<5

---------------------

Then a package that needs to use maya has the variants 

private_build_requires:

- gcc-4.4.6

requires:

- stdlib-4.4.6+<5

variants

- [maya-2011]

- [maya-2012]

- [maya-2014]

- [maya-2015]

Not sure if this solves the same issue but it is close enough that I think it was worth mentioning.

Hope this helps

Fede

[b.f...@toonboxent.com]

Hi Fede,

would you mind making it a pull request?
We kind of have similar issues and your solution sounds like a viable fix.

Cheers,
Blazej

[Allan Johns]

https://github.com/nerdvegas/rez/wiki/Package-Definition-Guide#private_build_requires

;)

[b.f...@toonboxent.com]

Sweet.

RTFM always gets you :)

[Fabien Castan]

Hi,

I'm not sure we can deal with gcc like we do with cmake. Cmake, doxygen and stuff like that are only used at build time whereas gcc provide runtime libraries and you need to meet these runtime requirements.

The difference with all other packages is that gcc provides full binary compatibility between all minor versions.

So, currently, I have added the 'gxx' in the requirements with an expression: 'gxx-4.2+' which works for compilation and runtime.

Is there a problem with this basic solution?

[b.f...@toonboxent.com]

I can not contribute to Fabien's question. I have a question on my own.

Correct me if I'm wrong:

I can not do something like:

rez-env --private_build=packageA,packageB packageB

The reason I'm looking for this functionality is that in case the private_build_requires is not only build generative (like doxygen) but more like test modules I can not create an environment with the test packages for my IDE.

Is this a valid request or is there a better way to deal with tests and other dev only packages.

Cheers,
Blazej

On Tuesday, 5 July 2016 17:08:03 UTC-4, allan.johns wrote:

[Allan Johns]

Hey Blazej,

A college and I were just talking about this. I would like to add 'test_requires' and 'test_commands' package attributes, and a 'rez-test' tool to run them. We would find this useful for running automated CI tests on our released packages, and you're right, sometimes you have a test-only package requirement.

Does this sound like what you're after?

A

[Fede Naum]

Hi Allan, 

test_requires will be awesome. !!

We workarounded by using private_build_requires but it causes some headaches sometimes.

Cheers

Fede

[b.f...@toonboxent.com]

test_requires/commands sounds like must have.

Please make it available as rez-env flag though, so we can hook up our IDE to the test environment for development, too.

[Allan Johns]

Yeah was planning on that. Might make sense to ditch the rez-env --build flag and instead have a --type=[build,test] instead.

A

[b.f...@toonboxent.com]

Since --build is not really an internal interface I doubt someone will miss it.

I hope the --type is accumulative so you can do build,test like build and test.

Private on the other hand is a different beast as what you might want to flag is something like --private_requires=package1,package2 to include the private requires for specific packages.

I assume that is a bit more work and for our case the test solution is pretty much what we need.

[sergei nazarenko]

Hi, though about it and maybe it would be better to use something like this:

variants = [

  [maya-2011]

 ,[maya-2012]

 ,[maya-2014]

 ,[maya-2015]

]

variants_private_build_requirements =

{

   0: [gcc-4.1.2]

  ,1: [gcc-4.1.2]

  ,2: [gcc-4.4.6]

  ,3: [gcc-4.8.2]

}

 

[Allan Johns]

Hey Sergei,

I think going in that direction is just putting off what we really need to do, and that's have more descriptive variants. I am thinking something like so:

with variant("py26") as v:
    requires = ["python-2.6"]
    build_requires = ["cmake"]

with variant("py27") as v:

    requires = ["python-2.7"]

    build_requires = ["cmake"]

Basically, within each variant scope you'd be able to override many of the package attributes, if not all. The label given here would become the installation subpath - it's an annoying limitation atm that the subpath has to match the variant dependencies.

A

--

You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[sergei nazarenko]

Got it, 

thanks for explanation, Allan

[b.f...@toonboxent.com]

That's awesome.
It's amazing to see that whenever we hit a problem it has been already discussed or even solved.

It seems rez really is going in the right direction of becoming an industry standard.
Keep it up people

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[b.f...@toonboxent.com]

Wouldn't it make sense to be able to override commands per variant with this method.
Currently we do many if system.platforms in the commands. Sometimes I even wish the package file was per variant as opposed to per version. But then you would have much duplication.
What about something like this instead:

with variant("py26") as v:

    requires = ["python-2.6", "platform-windows"]
    build_requires = ["cmake"]
    commands = windows_commands

def windows_commands():
    # Call base commands if necessary
    commands()
    # Do windows related stuff

On Friday, 5 August 2016 11:47:21 UTC-4, allan.johns wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[Allan Johns]

Yep, you should be able to override commands per-variant. I * think* you could just do this though:

with variant("py26") as v:
    requires = ["python-2.6"]
    build_requires = ["cmake"]

    def commands():

        pass

A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[b.f...@toonboxent.com]

Neat.

Other related questions: We think about cleaning up our gcc dependencies with private_build_requires ;)

It was mentioned that one practice is to separate gcc from libstdc++.
Is the backwards compatibility of libstdc++ ok or did you encounter problems in cases of mixed builds within the same environment.

For example is it save to call a gcc-4.1.2 (curse you Nuke!) library against libstdc++ from gcc 4.8.3?

We kind of would love to avoid statically link libstdc++ as it kind of adds up with its ~4 MB.

Cheers,
Blazej

[b.f...@toonboxent.com]

I just read through:
https://gcc.gnu.org/onlinedocs/libstdc++/manual/backwards.html

Apparently 4.x should be ABI compatible. There is a major change in 5.1 (dual ABI) and later but in our case it should be safe.

Does anyone have first hand experiences on that topic?

[Adrian Krupa]

Hi Allan,

Is something from this thread going to be implemented? We have some packages that need to have per variant requirements. Labeling variants would also be great. Do you have any roadmap?

Adrian

[Allan Johns]

Hi Adrian,

This work is high on my list. Specifically, what we require is:

* Ability to express more than just an extra set of requirements per variant (for eg, build_requires);

* Ability to label variants, where the label (and not the variant requires) defines the installation subpath;

* Ability to explicitly select variants in a resolve, if required.

That last feature is needed in cases where variants are not mutually exclusive. For example, you might have DCC-based and standalone variants in a package - it then becomes unclear as to which specific variant may be chosen (that process is still deterministic, but not necessarily easy to get your head around).

A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Thomas Mansencal]

Hi,

Just jumping in to say that we would be happy consumers of that feature. Are there any workarounds people using currently?

Cheers,

Thomas

To unsubscribe from this group and stop receiving emails from it, send an email to rez-c...@googlegroups.com.

[thomas.m...@gmail.com]

I find myself needing that again and coming in the hope for good news! :]

[brian...@locksmithanimation.com]

We are encountering very similar issues, and hacking our way around them, though not in any very effective way. We would also certainly use these features!

It would also be great if the variant could be accessed by its label in a build script, @early functions, etc.

[Allan Johns]

See here: https://github.com/nerdvegas/rez/wiki/Package-Definition-Guide#available-objects

You should be able to include build-only requirements on a per-variant basis using this feature. Since they will only appear during a build, they would effectively act like private build requirements. It would be better if variants did have native support for build/private-build requirements, but that's a fairly substantial update.

Please let me know how you get on with this approach. If it works for you, it may be worth updating the wiki to make things clearer.

Hth

A

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/12c6fdd5-883a-4c00-9575-33fa99f3a8b1n%40googlegroups.com.

[Thomas Mansencal]

Thanks Allan!

Besides the fact it ended up producing gross and rather inelegant code, it worked great! :)

Certainly worth documenting as a good placeholder method until something beautiful happens!

Thomas Mansencal - colour-science.org - thomasmansencal.com

You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/5RuZOOFUcok/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAGMneOeskJMrxRo0w8__MTWt5fDV4q7zGAFGuHo8rp58b%2Bsf1A%40mail.gmail.com.

[Allan Johns]

Yeah it's a bit clunky and wasn't written specifically to solve this case, but it works.

Cheers

A

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CACBPDdeyb9PdmYb3Q62hWRpD0fyAiRPmZJxhv55xPjSq3tOpag%40mail.gmail.com.