Guidance on where to take my companies' internal tools for packaging.

[Yusuke Tsutsumi]

Hi pypa-dev,

This is a rather involved post, so I appreciate the time. I've been shepparding a lot of the way we do Python at a company called Zillow. We use Python quite a bit, and have a huge need to re-use tools and keep them stable among the 200+ Python repositories we have.

Internally, we heavily use a project I wrote called uranium (https://uranium.readthedocs.io/) that is effectively a python-based version of Make that uses Pip and Virtualenv under the hood. I have two main open questions:

1. Does pypa have a plan for additional, arbitrary build steps for applications?

The reason uranium is so freeform is to handle aspects of building an application that are outside the scope of downloading and installing python packages:

* simplifying the configuration and startup of local dev builds of web servers

* packaging the final application as a tarball, in a propietary format

* bootstrapping test configuration (e.g. linting rules)

* pulling in the above using a common dependency

* picking and choosing what to install from source vs from a wheel (we sometimes need to link python packages to compiled c shared objects, but we use wheels when we can)

I know that pyproject.toml was designed to allow custom configuration values for various systems (e.g. black rules). Is there a story for how this type of stuff would be done?

Now that I'm typing this... I think I would probably write a python package that encapsulates a lot of the configuration we have today, and then expose ways to manipulate that through the pyproject.toml. What I would end up with is a mega-plugin that facilitates everything that was described above.

2. Support for "platform versioning"

This next one may be a little weird.. so bear with me :)

Internally, we have found a lot of value in maintaining a blessed version set for packages that do not have a local override. For example, this has allowed us to pin back backwards-incompatible dependencies that were not expressed properly from the package maintainer via semver.

In other words, we have a centralized system that keeps track of blessed versions, kind of like a giant requirements.txt:

requests == 2.21.0

aiohttp == 3.5.4

Locally, if you do not specify a specific version, then the blessed version will be inserted in it's place. So locally if you had something like:

requests == 2.7.0

And you wanted aiohttp, requests, the final result would be:

requests == 2.7.0

aiohttp = 3.5.4

regardless of what the latest version in the package repository actually is.

The pros and cons of a system like this are probably worth a debate on it's own :) But aside from that: is there any plans to support something like this? 

Currently Uranium extends the pip package resolution process to ensure this works for dependencies as well. constraints.txt is almost perfect but IIRC it can conflict with a requirements.txt or a setup.py specification, rather than omit itself.

Summary:

my hope is to migrate over to more common open source tooling, and I think the work done around pyproject.toml and other projects like poetry are a huge step. I'm just trying to get a read on whether there are thoughts around this I should focus effort on, or figure out a solution outside of that.

Thanks!

-Yusuke

[Sumana Harihareswara]

Yusuke, I know it's been some time since you posted this; it looks like
it was a complicated question and volunteers said "oh I need to think
about that and get to it later" and then no one did.

We're about to close down this mailing list to reduce the number of
places where discussion fragments; if you're still dealing with
difficulties using/building Python packages at Zillow, I encourage you
to forward your note to https://discuss.python.org/c/packaging or
https://mail.python.org/mailman3/lists/distutils-sig.python.org/ to
discuss further.

Sorry again!

-Sumana


--
Sumana Harihareswara
Changeset Consulting
https://changeset.nyc