--branch option

[Alexandra Lefève-Gourmelon]

Hi Allan,

To test our packages, we test them in a special package file tree where the test versions for each package are called dev.

For example:

localPackages

  |__myLib

          |__dev

                |__package.py

                |__...

 |__myPlug

         |__dev

               |__package.py

               |__...

We would like to be able to ask for a version name (dev) globally, for each package resolved by rez in a given command, if this version exists for this package.

So for example, if we run:

rez env packageA myPlug packageB --branch dev

with packageB last version requiring myPlug-3.4, we would have

REZ_USED_RESOLVE="packageA-<last> myPlug-dev myLib-dev packageB-<last>"

Is it possible to do so through the API ?

Cheers,

Alex

[allan.johns]

It is not. But first I'd like to understand more about what you're trying to do (it may overlap with some thoughts I've had).

Where do you use these 'dev' packages? Do they only ever appear in a local package directory, or are you actually releasing them? Are they intended to hang around forever, or are they temporary, more like the idea of a 'beta' package?

If they're only in the local packages dir, then why not just have a package there with a version you know will be picked up? In your example, why not just have a local myPlug-3.4?

In short, what problem specifically are you trying to solve? One area where Rez falls a little short right now is where people would like to release packages for testing purposes, but have that release not be 'official' or picked up by default, until ready to do so. This is different from a local package, because they would like multiple users to test the package, rather than just testing it themselves. Does this describe the problem you want to solve? If so, I'll expand on my own thoughts on this - there may be a relatively simple feature that would address this.

Thx
A

[Fabien Castan]

Where do you use these 'dev' packages? Do they only ever appear in a local package directory, or are you actually releasing them?

They are local packages.

Are they intended to hang around forever, or are they temporary, more like the idea of a 'beta' package?

With this feature, all developers are working directly in a REZ environnement. We just have local "dev" packages. That allows us to execute things on the renderfarm, etc. without any difference with installed packages.

If they're only in the local packages dir, then why not just have a package there with a version you know will be picked up? In your example, why not just have a local myPlug-3.4?

As it's our everyday workspace, we don't want to change the directory name (IDE configuration, etc.). And we would like to be able to force the use of all our "dev" packages easily.

In short, what problem specifically are you trying to solve? One area where Rez falls a little short right now is where people would like to release packages for testing purposes, but have that release not be 'official' or picked up by default, until ready to do so.

This is different from a local package, because they would like multiple users to test the package, rather than just testing it themselves. Does this describe the problem you want to solve? If so, I'll expand on my own thoughts on this - there may be a relatively simple feature that would address this.

We use some configuration files for that with "release", "beta" and "alpha" configurations to specify packages versions. Something like a small SOMA ;)

But we are interested to get your ideas on that.

[Mark Streatfield]

One area where Rez falls a little short right now is where people would like to release packages for testing purposes, but have that release not be 'official' or picked up by default, until ready to do so.

We have a manual solution for this.  "Proper" packages are released to the main central location <root>/packages.  However we modified rez release so we can release testing packages to <root>/testing/packages.  We are then able to promote a package from <root>/testing/packages to <root>/packages without rebuilding (thanks to some internal processes we already have).  Our production environment only has <root>/packages in the REZ_PACKAGES_PATH, but prerelease and beta environments have the option of including the <root>/testing/packages too.  It works quite well so far.

We also use this with our Jenkins builds.  Jenkins builds to <root>/testing/packages by default.  If the integration tests pass, the build is promoted to <root>/packages automagically.  If they fail it stays in testing for a developer to poke around with.  Jenkins automatically adds a build number to the package.yaml version.

With this feature, all developers are working directly in a REZ environnement. We just have local "dev" packages. That allows us to execute things on the renderfarm, etc. without any difference with installed packages.

We have talked about something similar to this.  Our approach was, if a package (of any version) exists in ~/packages then ignore a package with the same name on <root>/packages, almost like the opposite of the -nl option in rez env.  But we still allow packages to come from <root>/packages where they don't exist locally at all otherwise we'd get a broken environment or have to build everything locally.  

The idea is this would give developers a bit of grace in case they version incorrectly initially, and help when patching an old version (rolling back in development and environment is tricky to get right sometimes).  So far though we've not actually needed this.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[allan.johns]

Ok, 3 separate points I need to make!

1) RE 'beta' packages. The approach I had in mind is similar to what Mark describes below. But I was thinking of making it a little more flexible - rather than having just a single 'testing' packages path, you would have some base 'testing' path, then an arbitrary number of named paths under that. It's best illustrated with an example. Consider:

]$ export REZ_DEV_PACKAGES_PATH=/packages/testing
]$ rez-env foo bah --dev=beta

This would result in packages being pulled from the usual REZ_PACKAGES_PATH, but also from "/packages/testing/beta". The problem you might hit with a single dev path is that unrelated dev testing from different groups of developers will overlap - whilst group A are testing pkgA, and group B pkgB, they actually both get dev A and B, which may be problematic. It's the same approach as Mark fundfamentally, but with finer granularity.

2) Mark, the "opposite of --nl" approach is kind of interesting, although it still means you need some reasonably correct version number, which doesn't address Fabien's issue.

3) Ok, onto the meaty bit. Fabien to answer you directly, I don't think we should go with this approach, it would really be quite hacky thinking about the implementation. I don't like that it completely does away with version numbers - almost sidestepping the dependency resolution in a way. However I have another suggestion. If I understand you correctly, 'localPackages' in Alex's example is actually your working directory - you're actually updating the code in packages there directly right? I understand the desire to do this, but there are two problems - first it introduces the limitation that the structure of the package must exactly match the project itself, which sometimes is not realistic. And it happens to work for python, but what about compiled code? I don't like that this model breaks as soon as you hit other types of packages. Second, it causes the problem we're faced with right now.

Recently Mark added a PR that allows a cmake build to switch to installing symlinks to the original files/dirs when you do a local install. It strikes me that this gets you a long way towards what you want to be doing. I'm suggesting that you have a separate working directory rather than one that is also the local packages path, and you "rez-build -i" as usual when you need to. Which won't be often - only when a version is changed, or a project changes significantly (a new dir is added for eg). The local packages will all contain symlinks pointing back at your python source, so you will not have to keep rebuilding, and you can still update source on the fly. Let me know what you think.

Thx
A

ps - Mark just so you know - this upcoming update ('resources2' branch) gives you the ability to load a variant from one 'package repository' (ie, package path) and install that variant into another path. If the destination package already exists, the variant is correctly merged into it. This sounds like what you're already doing, so you might be able to get rid of some of your own code and be able to utilize Rez for this now.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Alexandra Lefève-Gourmelon]

Ok, 3 separate points I need to make!

1) RE 'beta' packages. The approach I had in mind is similar to what Mark describes below. But I was thinking of making it a little more flexible - rather than having just a single 'testing' packages path, you would have some base 'testing' path, then an arbitrary number of named paths under that. It's best illustrated with an example. Consider:

]$ export REZ_DEV_PACKAGES_PATH=/packages/testing
]$ rez-env foo bah --dev=beta

On our side, we create a group of profiles (our equivalent of SOMA profile) for each configuration (like release or beta, for example), to define shared versions to test.

This would result in packages being pulled from the usual REZ_PACKAGES_PATH, but also from "/packages/testing/beta". The problem you might hit with a single dev path is that unrelated dev testing from different groups of developers will overlap - whilst group A are testing pkgA, and group B pkgB, they actually both get dev A and B, which may be problematic. It's the same approach as Mark fundfamentally, but with finer granularity.

We deal with that inside our small SOMA. That allows us to specify different package versions for specific shots, steps, etc. in each configuration.

2) Mark, the "opposite of --nl" approach is kind of interesting, although it still means you need some reasonably correct version number, which doesn't address Fabien's issue.

On our side, local packages can override the official versions if we use the same version number. Or we use named versions for development. "dev" is the standard name, but if we work on multiple features simultaneously we could change this name:

localPackages

  |__myLib

          |__dev    <------ The everyday development

                |__package.py

                |__...

          |__1.5    <------- the version 1.5 which is already in production need a fix, we can test/modify it locally

                |__package.py

                |__...

          |__new_graph  <-------- a long term, development is on the road

                |__package.py

                |__...

 |__myPlug

         |__new_graph  <-------- a long term, development is on the road and it needs some changes in the required "myLib" package

               |__package.py

               |__...

With the --branch option I can switch all my repositories to the "new_graph" versions.

 

I don't like that it completely does away with version numbers - almost sidestepping the dependency resolution in a way.

The problem is that when we are developing a feature, we should not have to think about versions. That's the responsibility of the project/package maintainer.

A user ask a feature, one developer start coding it, then he submit a "pull request" and the maintainer is responsible to merge it or not in the next version. To use a concrete example, if 2 developers are starting 2 features at the same time... we have no idea which one will finish first...

So using an upper version just to make REZ happy is a bit hacky...

However I have another suggestion. If I understand you correctly, 'localPackages' in Alex's example is actually your working directory - you're actually updating the code in packages there directly right? I understand the desire to do this, but there are two problems - first it introduces the limitation that the structure of the package must exactly match the project itself, which sometimes is not realistic.

And it happens to work for python, but what about compiled code? I don't like that this model breaks as soon as you hit other types of packages.

3/4 of our internal developments are in Python, so it's an important use case. And the python projects have really fast iterations with a lot of versions, which make REZ so useful.

For compiled code, we declare the variants to an empty "platform" package with "linux" and "darwin" versions. As compiled code is always platform dependent, a compiled package looks like:

myCppLib

  |__dev

          |__platform-linux

                |__lib

                |__bin

          |__CMakeList.txt

          |__package.py

          |__src

                |__main.cpp

For some packages, we provide release and debug versions with an empty "build" package with "release" and "debug" versions.

myCppLib

  |__dev

          |__platform-linux

                |__build-debug

                      |__lib

                      |__bin

                |__build-release

                      |__lib

                      |__bin

          |__src

                |__main.cpp

And in that case, we even install the source code in production. That allows us to directly run gdb on a debug version in production condition.

Recently Mark added a PR that allows a cmake build to switch to installing symlinks to the original files/dirs when you do a local install.

It strikes me that this gets you a long way towards what you want to be doing. I'm suggesting that you have a separate working directory rather than one that is also the local packages path, and you "rez-build -i" as usual when you need to. Which won't be often - only when a version is changed, or a project changes significantly (a new dir is added for eg). The local packages will all contain symlinks pointing back at your python source, so you will not have to keep rebuilding, and you can still update source on the fly. Let me know what you think.

I don't see how using symlinks instead of doing it in place could help us with our --branch option. Could you explain, please?

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--

Alexandra Lefève-Gourmelon Pipeline TD g...@mikrosimage.eu Mikros Image Digital Post-Production for Video and Film  www.mikrosimage.eu

[allan.johns]

On Thursday, January 8, 2015 1:08:44 AM UTC-8, Alexandra Lefève-Gourmelon wrote:

Ok, 3 separate points I need to make!

1) RE 'beta' packages. The approach I had in mind is similar to what Mark describes below. But I was thinking of making it a little more flexible - rather than having just a single 'testing' packages path, you would have some base 'testing' path, then an arbitrary number of named paths under that. It's best illustrated with an example. Consider:

]$ export REZ_DEV_PACKAGES_PATH=/packages/testing
]$ rez-env foo bah --dev=beta

On our side, we create a group of profiles (our equivalent of SOMA profile) for each configuration (like release or beta, for example), to define shared versions to test.

This would result in packages being pulled from the usual REZ_PACKAGES_PATH, but also from "/packages/testing/beta". The problem you might hit with a single dev path is that unrelated dev testing from different groups of developers will overlap - whilst group A are testing pkgA, and group B pkgB, they actually both get dev A and B, which may be problematic. It's the same approach as Mark fundfamentally, but with finer granularity.

We deal with that inside our small SOMA. That allows us to specify different package versions for specific shots, steps, etc. in each configuration.

2) Mark, the "opposite of --nl" approach is kind of interesting, although it still means you need some reasonably correct version number, which doesn't address Fabien's issue.

On our side, local packages can override the official versions if we use the same version number. Or we use named versions for development. "dev" is the standard name, but if we work on multiple features simultaneously we could change this name:

localPackages

  |__myLib

          |__dev    <------ The everyday development

                |__package.py

                |__...

          |__1.5    <------- the version 1.5 which is already in production need a fix, we can test/modify it locally

                |__package.py

                |__...

          |__new_graph  <-------- a long term, development is on the road

                |__package.py

                |__...

 |__myPlug

         |__new_graph  <-------- a long term, development is on the road and it needs some changes in the required "myLib" package

               |__package.py

               |__...

With the --branch option I can switch all my repositories to the "new_graph" versions.

 

I don't like that it completely does away with version numbers - almost sidestepping the dependency resolution in a way.

The problem is that when we are developing a feature, we should not have to think about versions.

I actually strongly disagree with this. Consider two developers working on two features at once, then a maintainer has to merge the results. What if one developer made a patch change, and another made a minor version change? How does the maintainer know this? That information isn't there, because the version is just 'dev'. It doesn't matter if the versions don't match up to what the next package version is actually going to be - it matters that the version tells the maintainer something, so that they can then decide what the next package version should be. It also keeps developers in good practice - always think about what kind of change you're making, and reflect that in the version of your software.
 

That's the responsibility of the project/package maintainer.

A user ask a feature, one developer start coding it, then he submit a "pull request" and the maintainer is responsible to merge it or not in the next version. To use a concrete example, if 2 developers are starting 2 features at the same time... we have no idea which one will finish first...

So using an upper version just to make REZ happy is a bit hacky...

And this is actually an argument for Mark's suggestion, where we add a mode where local packages take total precedence and other package versions are ignored. I think this would be enabled with a '--local' flag.
 

Of course it is possible for in-place development to work. But my problems with this approach are:
- it forces you to have the same package source structure as installed structure. Maybe that's ok for you guys, it definitely is not for me.
- there's no guarantee that the installed version of your package (the one resulting from a release) actually matches the local version (perhaps some files or directories are in different places). That's not great - you should be testing the same thing that's going to be released.
- this isn't how software is normally deployed. I've never seen this use case with Rez, and other packaging systems all follow the develop/install/test cycle as well. Python even has a feature for egg development where symlinks are installed that point back to the original python sourcefiles, to allow for fast development - sound familiar? It just seems pretty ill-advised to me to go against a workflow that is so ingrained everywhere else.
 

Recently Mark added a PR that allows a cmake build to switch to installing symlinks to the original files/dirs when you do a local install.

It strikes me that this gets you a long way towards what you want to be doing. I'm suggesting that you have a separate working directory rather than one that is also the local packages path, and you "rez-build -i" as usual when you need to. Which won't be often - only when a version is changed, or a project changes significantly (a new dir is added for eg). The local packages will all contain symlinks pointing back at your python source, so you will not have to keep rebuilding, and you can still update source on the fly. Let me know what you think.

I don't see how using symlinks instead of doing it in place could help us with our --branch option. Could you explain, please?

My reasoning is like this:
- You can't change the package version (from 'dev' for eg) because you're working in-place.
- Why are you working in-place? I assume so that you can update code directly.
- With the symlinking feature, you don't need to work in place
- ...which means, you don't really need the --branch feature.

In short, I can't see the justification for this feature - a feature which actually partially disables what the dependency resolver is doing, and is open to abuse I might add (can't quite get the env you want? - just force it with the --branch option! Pretty tempting in production). You can get in-place code updates and local testing of feature development by following the normal development cycle - no extra feature is needed, and the development paradigm is more standard. And, more information is conveyed to the maintainer so they can better decide what the version of the next package release should actually be.

I do think that the '--local' option would help though.

Thx
A
 

[Fabien Castan]

The problem is that when we are developing a feature, we should not have to think about versions.

I actually strongly disagree with this. Consider two developers working on two features at once, then a maintainer has to merge the results. What if one developer made a patch change, and another made a minor version change? How does the maintainer know this? That information isn't there, because the version is just 'dev'.

It doesn't matter if the versions don't match up to what the next package version is actually going to be - it matters that the version tells the maintainer something, so that they can then decide what the next package version should be. It also keeps developers in good practice - always think about what kind of change you're making, and reflect that in the version of your software.

I'm really surprised by your position.

Sure, the developer has to think if it's a minor change or not, if it will conflict with long-term developments already started, if it breaks compatibility, etc.

Sure, there will be a discussion with the maintainer during the "Pull Request".

But a developer will never modify the version in the code to indicate the version to the maintainer. I can't imagine the hell when you navigate through the history with wrong versions inside...

For example, I can't imagine modifying the version of Qt when I submit a patch! I have absolutely no idea of the other developments, etc.

The version is determined when the feature is merged: http://nvie.com/posts/a-successful-git-branching-model

To take a non-ambiguous example... In SWIG, I'm waiting for a merge of a feature started in 2012. The feature has started to be mergeable 6 months ago, the merge has been planned for SWIG 3.0.0. Now SWIG 3.0.3 has been released and it's still not there...

Do you agree that using an upper version just to make REZ happy is a bit hacky?

Of course it is possible for in-place development to work. But my problems with this approach are:
- it forces you to have the same package source structure as installed structure. Maybe that's ok for you guys, it definitely is not for me.

We can do the installation in-place inside a subdirectory of the package, as we do for compiled projects.

It's useful to provide rez packages with release and debug builds with full source code to allow debugging from everywhere. What do you suggest to do that?

- there's no guarantee that the installed version of your package (the one resulting from a release) actually matches the local version (perhaps some files or directories are in different places). That's not great - you should be testing the same thing that's going to be released.

In our case, the structure is exactly the same locally and in the official packages. Even for compiled projects.

The only difference is that we do not always provide the source code of compiled projects in the official packages.

- this isn't how software is normally deployed. I've never seen this use case with Rez, and other packaging systems all follow the develop/install/test cycle as well. Python even has a feature for egg development where symlinks are installed that point back to the original python sourcefiles, to allow for fast development - sound familiar?

Yes, "python setup.py develop" allows to create a dedicated python installation with a development package inside... It is really different from REZ which allows to choose packages versions, check conflicts, etc.

REZ is a bit closer to homebrew, apt-get, etc. than from the python installation process. But in homebrew and apt-get you have only one global environment... and you can use it to develop and test. As REZ creates environments dynamically... it seems useful to have a mechanism for that.

It just seems pretty ill-advised to me to go against a workflow that is so ingrained everywhere else.

I'm not sure that using symlinks is a really standard workflow. Do you have some other examples?

I don't see how using symlinks instead of doing it in place could help us with our --branch option. Could you explain, please?

My reasoning is like this:
- You can't change the package version (from 'dev' for eg) because you're working in-place.
- Why are you working in-place? I assume so that you can update code directly.
- With the symlinking feature, you don't need to work in place
- ...which means, you don't really need the --branch feature.

I don't see how it solves the problem... except that it allows to use a static path (to keep IDE configuration for example). But that's not the most important point.

In short, I can't see the justification for this feature - a feature which actually partially disables what the dependency resolver is doing and is open to abuse I might add (can't quite get the env you want? - just force it with the --branch option! Pretty tempting in production).

I'm not convinced by the idea of keeping constraints in the tool to avoid abuse.

You can get in-place code updates and local testing of feature development by following the normal development cycle - no extra feature is needed, and the development paradigm is more standard. And, more information is conveyed to the maintainer so they can better decide what the version of the next package release should actually be.

So the 2 solutions you propose are:

1/ use an upper version to take the precedence

2/ use a custom version and add an option --local to force the precedence of local packages

And use symlinks to keep a static path.

Is that correct?

[allan.johns]

On Sunday, January 11, 2015 at 5:07:08 PM UTC-8, Fabien Castan wrote:

The problem is that when we are developing a feature, we should not have to think about versions.

I actually strongly disagree with this. Consider two developers working on two features at once, then a maintainer has to merge the results. What if one developer made a patch change, and another made a minor version change? How does the maintainer know this? That information isn't there, because the version is just 'dev'.

It doesn't matter if the versions don't match up to what the next package version is actually going to be - it matters that the version tells the maintainer something, so that they can then decide what the next package version should be. It also keeps developers in good practice - always think about what kind of change you're making, and reflect that in the version of your software.

I'm really surprised by your position.

Sure, the developer has to think if it's a minor change or not, if it will conflict with long-term developments already started, if it breaks compatibility, etc.

Sure, there will be a discussion with the maintainer during the "Pull Request".

But a developer will never modify the version in the code to indicate the version to the maintainer. I can't imagine the hell when you navigate through the history with wrong versions inside...

For example, I can't imagine modifying the version of Qt when I submit a patch! I have absolutely no idea of the other developments, etc.

The version is determined when the feature is merged: http://nvie.com/posts/a-successful-git-branching-model

To take a non-ambiguous example... In SWIG, I'm waiting for a merge of a feature started in 2012. The feature has started to be mergeable 6 months ago, the merge has been planned for SWIG 3.0.0. Now SWIG 3.0.3 has been released and it's still not there...

Ok, I should point out that this is something of a personal choice, one way or the other whether you decide to version up during branch development or not, this doesn't affect Rez per se. In the past I've worked somewhere where we actually had a different take again - branch development included a branch part of the package version, so you could see what the 'local branch version' was, as well as what the original version (at the root of the branch) was.
 

Do you agree that using an upper version just to make REZ happy is a bit hacky?

Here I just don't understand what you mean - I thought I'd pointed out that I'd accept a new --local flag so that non-local packages are ignored.
 

Of course it is possible for in-place development to work. But my problems with this approach are:
- it forces you to have the same package source structure as installed structure. Maybe that's ok for you guys, it definitely is not for me.

We can do the installation in-place inside a subdirectory of the package, as we do for compiled projects.

It's useful to provide rez packages with release and debug builds with full source code to allow debugging from everywhere. What do you suggest to do that?

You just deal with this in your build, I've developed C++ for many years that is debuggable, but was installed in the usual manner. Perhaps there's something about your development process that is not obvious (or perhaps you're talking about some case I haven't considered, a different language?). Given that actual released packages aren't in-place, how do you provide debugging with full source in this case?
 

- there's no guarantee that the installed version of your package (the one resulting from a release) actually matches the local version (perhaps some files or directories are in different places). That's not great - you should be testing the same thing that's going to be released.

In our case, the structure is exactly the same locally and in the official packages. Even for compiled projects.

The only difference is that we do not always provide the source code of compiled projects in the official packages.

- this isn't how software is normally deployed. I've never seen this use case with Rez, and other packaging systems all follow the develop/install/test cycle as well. Python even has a feature for egg development where symlinks are installed that point back to the original python sourcefiles, to allow for fast development - sound familiar?

Yes, "python setup.py develop" allows to create a dedicated python installation with a development package inside... It is really different from REZ which allows to choose packages versions, check conflicts, etc.

REZ is a bit closer to homebrew, apt-get, etc. than from the python installation process. But in homebrew and apt-get you have only one global environment... and you can use it to develop and test. As REZ creates environments dynamically... it seems useful to have a mechanism for that.

It just seems pretty ill-advised to me to go against a workflow that is so ingrained everywhere else.

I'm not sure that using symlinks is a really standard workflow. Do you have some other examples?

I don't see how using symlinks instead of doing it in place could help us with our --branch option. Could you explain, please?

My reasoning is like this:
- You can't change the package version (from 'dev' for eg) because you're working in-place.
- Why are you working in-place? I assume so that you can update code directly.
- With the symlinking feature, you don't need to work in place
- ...which means, you don't really need the --branch feature.

I don't see how it solves the problem... except that it allows to use a static path (to keep IDE configuration for example). But that's not the most important point.

In short, I can't see the justification for this feature - a feature which actually partially disables what the dependency resolver is doing and is open to abuse I might add (can't quite get the env you want? - just force it with the --branch option! Pretty tempting in production).

I'm not convinced by the idea of keeping constraints in the tool to avoid abuse.

You can get in-place code updates and local testing of feature development by following the normal development cycle - no extra feature is needed, and the development paradigm is more standard. And, more information is conveyed to the maintainer so they can better decide what the version of the next package release should actually be.

So the 2 solutions you propose are:

1/ use an upper version to take the precedence

2/ use a custom version and add an option --local to force the precedence of local packages

And use symlinks to keep a static path.

Is that correct?

I'm not sure because I don't quite understand what you mean by (1). Let me describe the usual Rez process, perhaps that will clear some things up:

1) You have your source in, say, ~/workplace/foo/feature_1 and ~/workplace/foo/feature_2. Let's assume these are git clones of two different branches of the 'foo' project, for feature development.
2) You 'rez-build -i' to locally install the feature package you're working on.
3) Because of the new symlinking feature (which rez-build takes care of for you), you can continue to develop python files etc in-place, so you won't need to rez-build very often.
4) You 'rez-env --local foo bah eek...' in another shell, to test your local package. In your case it sounds as though you're happy to leave the version unchanged, that's fine.
5) If you want to switch to a different feature to test, you juts delete the previous local foo install, cd to the other working dir, and follow the same process.

Let me explain my resistance to this feature some more. It's about guarantees that Rez makes. Rez has been around for a few years now, and it has always guaranteed the user that the requirements they give their package are respected. You are asking to add a feature that breaks that guarantee - I consider that a pretty big deal. Consider the package:

```
# bah.yaml
name: bah
version: 2.3.4
requires:
# fixme: foo-1.3 contains a bug that will corrupt the database!!!
- foo-1.2
```

Unfortunately, because of the --branch feature, I cannot guarantee this package author that their requirements will be respected. It's one thing to provide your users with features that are capable of breaking things but that you should use responsibly (you raised this earlier and for the most part I agree with you). But it is another to remove a guarantee that your project has made from the beginning. In this case I think there is real danger here - someone using the --branch feature may not even be aware that package 'bah' exists. But, when creating an environment to test in, they could easily pull a 'bah' package into the resolve, which is now in a broken and potentially dangerous state. Of course you could argue, "but you'd only ever branch to 'dev' or something, there won't be other packages out there with a 'dev' version!" But there's nothing in Rez that guarantees that - it's an assumption based on your workflow, not on other studios'.

In order to move forward I really need to understand why the standard rez workflow - the one I outlined above - will not work for you. This is the first case ever where I've been asked to add a feature that supports in-place local package development. I hope you understand my reservations, given that this might be an atypical use case, and that the feature being asked for breaks a guarantee that's always been there. There may be repercussions to forcing requirements like this that we have not even thought of.

For the record, while I'm not such a fan of in-place package development, I don't think I should stop Rez from letting you do that. It is first and foremost the forcing of the dependency resolver to accept an arbitrary package version that I have concerns about.

Thx
A



 

[Fabien Castan]

Do you agree that using an upper version just to make REZ happy is a bit hacky?

Here I just don't understand what you mean - I thought I'd pointed out that I'd accept a new --local flag so that non-local packages are ignored.

You're right, "--local" is a good answer for that point.

I'm not sure because I don't quite understand what you mean by (1). Let me describe the usual Rez process, perhaps that will clear some things up:

1) You have your source in, say, ~/workplace/foo/feature_1 and ~/workplace/foo/feature_2. Let's assume these are git clones of two different branches of the 'foo' project, for feature development.
2) You 'rez-build -i' to locally install the feature package you're working on.
3) Because of the new symlinking feature (which rez-build takes care of for you), you can continue to develop python files etc in-place, so you won't need to rez-build very often.
4) You 'rez-env --local foo bah eek...' in another shell, to test your local package. In your case it sounds as though you're happy to leave the version unchanged, that's fine.
5) If you want to switch to a different feature to test, you juts delete the previous local foo install, cd to the other working dir, and follow the same process.

Thanks for this clarification.

I will re-test that workflow when I come back at work (in 2 weeks) and give you a feedback.

It's useful to provide rez packages with release and debug builds with full source code to allow debugging from everywhere. What do you suggest to do that?

You just deal with this in your build, I've developed C++ for many years that is debuggable, but was installed in the usual manner. Perhaps there's something about your development process that is not obvious (or perhaps you're talking about some case I haven't considered, a different language?). Given that actual released packages aren't in-place, how do you provide debugging with full source in this case?

I'm taking about the C++ case. We would like to be able to run some projects into gdb directly from any computer without any setup. As gcc put absolute paths to the source code files in its binaries, the idea was to build it in-place and have the full source code directly on the network.

In that case, we need to avoid using a user specific path (like ~/workspaces) to avoid having to retrieve the source code locally and then remap paths to this clone...

Unfortunately, because of the --branch feature, I cannot guarantee this package author that their requirements will be respected. It's one thing to provide your users with features that are capable of breaking things but that you should use responsibly (you raised this earlier and for the most part I agree with you). But it is another to remove a guarantee that your project has made from the beginning. In this case I think there is real danger here - someone using the --branch feature may not even be aware that package 'bah' exists. But, when creating an environment to test in, they could easily pull a 'bah' package into the resolve, which is now in a broken and potentially dangerous state. Of course you could argue, "but you'd only ever branch to 'dev' or something, there won't be other packages out there with a 'dev' version!" But there's nothing in Rez that guarantees that - it's an assumption based on your workflow, not on other studios'.

Yes, it's a description of the workflow we use currently (in top of rez). It's quite efficient but we are really open to find a better workflow.

In order to move forward I really need to understand why the standard rez workflow - the one I outlined above - will not work for you. This is the first case ever where I've been asked to add a feature that supports in-place local package development. I hope you understand my reservations, given that this might be an atypical use case, and that the feature being asked for breaks a guarantee that's always been there. There may be repercussions to forcing requirements like this that we have not even thought of.

For the record, while I'm not such a fan of in-place package development, I don't think I should stop Rez from letting you do that. It is first and foremost the forcing of the dependency resolver to accept an arbitrary package version that I have concerns about.

Sure, I understand that.

Thanks a lot for your time and the interesting and constructive discussion.

I will come back to you soon.

Fabien

[Allan Johns]

On Mon, Jan 12, 2015 at 3:59 PM, Fabien Castan <fa...@mikrosimage.eu> wrote:

Do you agree that using an upper version just to make REZ happy is a bit hacky?

Here I just don't understand what you mean - I thought I'd pointed out that I'd accept a new --local flag so that non-local packages are ignored.

You're right, "--local" is a good answer for that point.

I'm not sure because I don't quite understand what you mean by (1). Let me describe the usual Rez process, perhaps that will clear some things up:

1) You have your source in, say, ~/workplace/foo/feature_1 and ~/workplace/foo/feature_2. Let's assume these are git clones of two different branches of the 'foo' project, for feature development.
2) You 'rez-build -i' to locally install the feature package you're working on.
3) Because of the new symlinking feature (which rez-build takes care of for you), you can continue to develop python files etc in-place, so you won't need to rez-build very often.
4) You 'rez-env --local foo bah eek...' in another shell, to test your local package. In your case it sounds as though you're happy to leave the version unchanged, that's fine.
5) If you want to switch to a different feature to test, you juts delete the previous local foo install, cd to the other working dir, and follow the same process.

Thanks for this clarification.

I will re-test that workflow when I come back at work (in 2 weeks) and give you a feedback.

It's useful to provide rez packages with release and debug builds with full source code to allow debugging from everywhere. What do you suggest to do that?

You just deal with this in your build, I've developed C++ for many years that is debuggable, but was installed in the usual manner. Perhaps there's something about your development process that is not obvious (or perhaps you're talking about some case I haven't considered, a different language?). Given that actual released packages aren't in-place, how do you provide debugging with full source in this case?

I'm taking about the C++ case. We would like to be able to run some projects into gdb directly from any computer without any setup. As gcc put absolute paths to the source code files in its binaries, the idea was to build it in-place and have the full source code directly on the network.

In that case, we need to avoid using a user specific path (like ~/workspaces) to avoid having to retrieve the source code locally and then remap paths to this clone...

Ok that's what I thought you meant. But I was confused because since we're only talking about local packages, why would you be debugging from another computer? Typically local packages are only visible to the local developer. And when you actually release packages (centrally via rez-release I mean), presumably you still want to be able to debug these packages via gdb? So don't you have to figure out how to do this in an install anyway?

 

Unfortunately, because of the --branch feature, I cannot guarantee this package author that their requirements will be respected. It's one thing to provide your users with features that are capable of breaking things but that you should use responsibly (you raised this earlier and for the most part I agree with you). But it is another to remove a guarantee that your project has made from the beginning. In this case I think there is real danger here - someone using the --branch feature may not even be aware that package 'bah' exists. But, when creating an environment to test in, they could easily pull a 'bah' package into the resolve, which is now in a broken and potentially dangerous state. Of course you could argue, "but you'd only ever branch to 'dev' or something, there won't be other packages out there with a 'dev' version!" But there's nothing in Rez that guarantees that - it's an assumption based on your workflow, not on other studios'.

Yes, it's a description of the workflow we use currently (in top of rez). It's quite efficient but we are really open to find a better workflow.

In order to move forward I really need to understand why the standard rez workflow - the one I outlined above - will not work for you. This is the first case ever where I've been asked to add a feature that supports in-place local package development. I hope you understand my reservations, given that this might be an atypical use case, and that the feature being asked for breaks a guarantee that's always been there. There may be repercussions to forcing requirements like this that we have not even thought of.

For the record, while I'm not such a fan of in-place package development, I don't think I should stop Rez from letting you do that. It is first and foremost the forcing of the dependency resolver to accept an arbitrary package version that I have concerns about.

Sure, I understand that.

Thanks a lot for your time and the interesting and constructive discussion.

I'll keep thinking on this one. If you really need in-place packages there may be other ways we could do that.

 

I will come back to you soon.

Fabien

Thx,

Allan