Rez releasing Python Packages

[Alexandra Lefève-Gourmelon]

Hi,

I've been using rez for some years now but its release features are brand new to me. I was wondering how other studios were dealing with the release of python packages.
If I understand correctly, this is how we're supposed to work:

sandbox  ⇒ build and install ⇒ local package folder ⇒ release ⇒ released package folder

The sandbox contains the source code.

The local package folder is used to test the rez package.

The release package folder contains the code in prod.

At Superprod, we're working with windows and linux.

Because aliases don't work on windows (unless I missed something), we need to use bat files to run commands easily.

So even our python packages have variants.

But rez release puts all the code under the variant of the current platform, which means that if I have this kind of structure in the source code:

package_a

   |_ package.py

   |_ rezbuild.py

   |_ platform-windows

        |_bin

            |_ do_it.bat

   |_ platform-linux

       |_ bin

           |_ do_it

   |_python

       |_ package_a

           |_ script.py

 

All the code in bold will be released under the platform-windows variant, like this:

package_a

   |_ 1.0.0

      |_ package.py

      |_ platform-windows

         |_ platform-windows

              |_bin

                  |_ do_it.bat

         |_ platform-linux

             |_ bin

                 |_ do_it

         |_python

             |_ package_a

                 |_ script.py

Which is clearly not what we want.

So I was wondering I missed something with rez release.

We can cheat in rezbuild.py to move all the code up the variant when it's released but it's not very elegant.

Any advice on this?

Cheers,

Alex

--

Alexandra Lefève-Gourmelon

Lead R&D

www.superprod.net

107 Avenue Parmentier 75011 Paris France 

[Marcus Ottosson]

Because aliases don’t work on windows (unless I missed something)

Aliases should work, what issues have you been experiencing? On Windows, “alias” means “doskey”.

• https://www.computerhope.com/doskeyhl.htm

So I was wondering I missed something with rez release.

I also had trouble with this command at first, the name is a little misleading.

• If to “release” a package means making it available to production
• And rez build --install means hosting it in your local development directory, e.g. ~/packages
• Then rez release is unrelated to that

In a nutshell, rez release is an integration with git and Git “tags” to automate the process of building a package and creating a Git tag. You can use that as a means of “releasing” packages, but unless you leverage Git and tags for your packages, then it doesn’t apply to you. Other options are e.g. rez build --install --release to “install” onto your REZ_RELEASE_PACKAGES_PATH like rez release does, and another is rez build --install --prefix /some/central/location to specify a destination yourself.

• https://github.com/nerdvegas/rez/wiki/Releasing-Packages

In either case, rez release isn’t going to help you with arranging your hierarchy. For that, it sounds like what you want is something like..

package_a
   |_ package.py
   |_ rezbuild.py
   |_ python
      |_ script.py
   |_bin
      |_ do_it.bat
      |_ do_it

And then on build, you pick the right bin/do_it file based on platform.

if os.name == "nt":
  shutil.copyfile("bin/do_it.bat", os.path.join(build_dir, "bin"))
else:
  shutil.copyfile("bin/do_it", os.path.join(build_dir, "bin"))

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAAm-xYZJsv%2B5PPzWahuVqhKAsLz51N-BPBWFyMYgik2FR9BEfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Alexandra Lefève-Gourmelon]

Hi Marcus,

You're right, I was referring to rez build -i and not really rez release.

If I create platform variants, he code will be built in the current variant, and I will have the same problem, something like this:

package_a

|_1.0.0 |_ package.py |_ platform-windows |_ python |_ script.py |_ bin |_ do_it.bat

In my local package folder, when what I want is:

package_a

|_1.0.0 |_ package.py

|_ python |_ script.py   |_ platform-windows |_ bin |_ do_it.bat

I know I can have what I want by making the rezbuild.py move the code from under the platform variant to the upper folder, but it doesn't seem very elegant to me.

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAFRtmOB0dAnaT5s2cwPo1OOf6eztfje-jgp7GiNzkU9ZqiVhug%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

Ah, I think I see what you mean. You want the Windows-specific parts (in this case, the do_it.bat and do_it executable) separate from the cross-platform parts (in this case script.py), so that each variant can share the same Python code, is that right?

I understand that the above example is a minimal replica of the actual problem, but if that’s all there is, then have you considered not using a variant?

For example, this package is similar to your example, and builds for both Windows, Linux and MacOS, without using variants.

• mottosso/rez-pipz
• install.py#L59

To Windows, the executable is install.bat whereas for Linux/MacOS the executable is the install file. Because Linux/MacOS doesn’t care about the .bat file, and because Windows doesn’t care about the non-.bat files, they can happily co-exist in the same package and same resolved environment.

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAAm-xYZikCxzm2bF1UdfGNk%3DCp3tgxtUUS6Npz_X4BDfm4gWsQ%40mail.gmail.com.

[Alexandra Lefève-Gourmelon]

I didn't know about the mutually ignoring of the other platform executable scripts!

Thanks Marcus, this is exactly what I needed!

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAFRtmOD_2aNaYiw%3D%3D8pzQKkOfWKpcnW2LcC5BbiW0zCkQqBNGQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

Happy to help! :)

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAAm-xYb1uOnk93aC0pfGZmsi5uUQQtf9JURMjZkJxGK4VNBw5Q%40mail.gmail.com.

[Alexandra Lefève-Gourmelon]

I have another related question: when using rez pip, the created rez package always has all the implicit packages as variants, even when there's only python code in it.

I suppose it's to be sure to have these variants when they're needed. But since the code is often cross-platform, I was wondering if there was a way to prevent rez pip from creating the variants when there's no need for them?

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAFRtmOAhVgmKAHPXUaFs0gqRBZzF5rrfuCGdku%2BnZBfpd0Efuw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

That was one of my gripes with rez pip as well, which is one of things solved in the rez-pipz project linked to above.

$ git clone https://github.com/mottosso/rez-pipz.git
$ cd rez-pipz
$ rez build --install
$ rez env pipz -- install six
# Cross-platform Rez package created

• https://github.com/mottosso/rez-pipz#faq

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAAm-xYbwyv9Fh%3D%2B8Frbv6T%2BEL27b2-T0kf5dbKoWmDir_im1BQ%40mail.gmail.com.

[Alexandra Lefève-Gourmelon]

That's perfect :)! I will try it.

I'll open another thread about the alias problem on Windows.

Thanks again, Marcus!

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAFRtmODwJONnuSOxGDXhm5SLaj1sngDHcg_rAG_7bJ-9F3Ce1Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Thorsten Kaufmann]

Note that there is also a PR actively being worked on to fix this in mainline: https://github.com/nerdvegas/rez/pull/628

(also worth mentioning is the PR on fixing a bunch of regressions: https://github.com/nerdvegas/rez/pull/656)

[Alexandra Lefève-Gourmelon]

Good to know!
Thanks Thorsten.

Le mercredi 3 juillet 2019, 'Thorsten Kaufmann' via rez-config <rez-c...@googlegroups.com> a écrit :
> Note that there is also a PR actively being worked on to fix this in mainline: https://github.com/nerdvegas/rez/pull/628
> (also worth mentioning is the PR on fixing a bunch of regressions: https://github.com/nerdvegas/rez/pull/656)
>

> --
> You received this message because you are subscribed to the Google Groups "rez-config" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
> To post to this group, send email to rez-c...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rez-config.

> To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/7f4bc0dd-2cf3-41e6-9d4a-fb00f0600dec%40googlegroups.com.

[Allan Johns]

Yes, so variants are an all-or-nothing deal. If a package has variants, then you can consider these as separate but parallel instantiations of the package, with differing requirements for each. It is not possible to share any of the package payload between variants (why that is is a long conversation!).

You can always simply detect the current platform during your build instead, and alter the build as you see fit. Whether or not it's appropriate to do this instead of using variants really depends on your situation, both approaches are equally valid (unless you're releasing packages in a multi-platform studio of course!).

Also as others have pointed out, rez-pip is currently being updated, and one feature being developed aims to detect when a pip package is pure python, in which case the superfluous varianting on platform/arch/os will be avoided.

Cheers

A

To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/CAAm-xYaXW%2BXRonKinrKdQce60mybsoFdHHnmhSAeakzk6uB7rg%40mail.gmail.com.