Packaging multiple wheels in the same archive
377 views
Skip to first unread message
Nir Cohen
Nov 22, 2016, 12:44:37 PM11/22/16
to distutils-sig
Hey all,
As part of working on Cloudify (http://github.com/cloudify-cosmo) we've had to provide a way for customers to install our plugins in an environment where PyPI isn't accessible. These plugins are sets of Python packages which necessarily depend on one another (i.e. a regular python package with dependencies).
We decided that we want to package sets of wheels together created or downloaded by `pip wheel`, add relevant metadata, package them together into a single archive (tar.gz or zip) and use the same tool which packs them up to install them later on, on the destination hosts.
We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do just that and that's what we currently use to create and install our plugins.
While wheel solves the problem of generating wheels, there is no single, standard method for taking an entire set of dependencies packaged in a single location and installing them in a different location.
We thought it would be a good idea to propose a PEP for that and wanted to get your feedback before we start writing the proposal. Our proposed implementation is not the issue here of course, but rather if you think there should be a PEP describing the way multiple wheels should be packaged together to create a standalone installable package.
We would greatly appreciate your feedback on this.
Thanks!
As part of working on Cloudify (http://github.com/cloudify-cosmo) we've had to provide a way for customers to install our plugins in an environment where PyPI isn't accessible. These plugins are sets of Python packages which necessarily depend on one another (i.e. a regular python package with dependencies).
We decided that we want to package sets of wheels together created or downloaded by `pip wheel`, add relevant metadata, package them together into a single archive (tar.gz or zip) and use the same tool which packs them up to install them later on, on the destination hosts.
We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do just that and that's what we currently use to create and install our plugins.
While wheel solves the problem of generating wheels, there is no single, standard method for taking an entire set of dependencies packaged in a single location and installing them in a different location.
We thought it would be a good idea to propose a PEP for that and wanted to get your feedback before we start writing the proposal. Our proposed implementation is not the issue here of course, but rather if you think there should be a PEP describing the way multiple wheels should be packaged together to create a standalone installable package.
We would greatly appreciate your feedback on this.
Thanks!
Nir Cohen
Nov 22, 2016, 2:31:15 PM11/22/16
to Michael Merickel, distutils-sig
My implementation actually does exactly that but that's really not the issue here. Yes, `pip wheel` takes care of that in the sense that it downloads the wheels into a single folder. I'm actually talking about the packaging/installation side. The package needs to be named, it needs to contain metadata to aid in installation on the relevant platform, and so on.
I'd like to be able to run something like "pip wagon tensorflow" and have a single archive containing all wheels of all dependencies required (e.g. tensorflow-0.11.0rc2-py27-none-manylinux1_x86_64.wgn). At this point, I will be able to run `pip install tensorflow-0.11.0rc2-py27-none-manylinux1_x86_64.wgn` which will install it in my environment.
On Tue, Nov 22, 2016 at 9:16 PM Michael Merickel <mmer...@gmail.com> wrote:
On Tue, Nov 22, 2016 at 11:44 AM, Nir Cohen <nir...@gmail.com> wrote:While wheel solves the problem of generating wheels, there is no single, standard method for taking an entire set of dependencies packaged in a single location and installing them in a different location.
I can't comment on wagon itself or your specific issues but wheels do solve this problem via a wheelhouse (a folder of wheels). You can build the wheels for an entire dependency tree via ``pip wheel -w wheeldir`` and then take that wheel folder to another system with the same runtime requirements and install the wheels offline.pip install --no-index --find-links=wheeldir mypkgI guess it would be helpful to know what part of this didn't work for you or if you simply weren't aware of it.
Nir Cohen
Nov 22, 2016, 5:12:58 PM11/22/16
to Wayne Werner, distutils-sig
As I said earlier.. This is exactly what wagon does 🙂
The point is that there needs to be a type of archive with a naming convention and metadata that would contain these wheels and be createable and installable. That's a wagon. Does that make sense?
On Nov 22, 2016 10:59 PM, "Wayne Werner" <waynej...@gmail.com> wrote:
FWIW you can already specify the `--find-links` option[1] to specify that you want to install wheels from a given directory. Add `--no-index` if you want to install from *only* that dir.An approach that certainly might be interesting is if pip supported a zipfile of wheels vs a directory of wheels, then this becomes as easy as zipping up all your wheels.===========================================================I welcome VSRE emails. Learn more at http://vsre.info/===========================================================
On Tue, Nov 22, 2016 at 11:44 AM, Nir Cohen <nir...@gmail.com> wrote:
_______________________________________________
Distutils-SIG maillist - Distut...@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Reply all
Reply to author
Forward
0 new messages