Modern Python packaging and Review Board

[Christian Hammond]

Hey everyone,

Gregory Szorc just provided us with a patch to convert one of our modules to support the new Python Wheel format. Thanks for that!

There were some questions along with the patch that I wanted to address to a wider audience. I thought this would be a good time to get a discussion going on the future of Python packaging and how it may impact us.

This is a long e-mail, so I'll try to section things off.


== Wheels and Eggs ==

For those who aren't familiar with the world of Python packaging, the Wheel format is a newer binary package standard designed to provide a common way to ship modules and to replace the older egg format (which is what we use). All the new packaging standards for metadata and versioning improvements are designed to work with wheels.

Wheels and eggs differ in a few key ways. Wheels are an installation format, which gets unpacked on install, while eggs can either be unpacked or can remain as a single relocatable file. Content in eggs are importable and can be selectively extracted or read, using a set of APIs in the pkg_resources module. Wheels cannot do this. Eggs ship .pyc files, whereas Wheels generate them at install time (potentially problematic -- see below).

Wheels are also versioned differently, in an interesting way. Unlike eggs, they don't have to be tied to a Python version, so we could, for instance, ship a single Wheel for Review Board 2.0.x, and not have to ship separate ones for Python 2.6 and 2.7. Neat, but in practice, it's not a strong benefit, as we automate all our builds, which need to package and run tests on the various versions anyway.

Wheels are new, though, and not supported everywhere. One needs to have a modern version of pip/setuptools to install them or work with them. This can be a problem on older Linux distros, or companies with servers that are set up a very particular way, where upgrading components is hard/time-consuming.

Now, I'm not at all an expert on Wheels. There are things I will need to be educated on, or things I'm misunderstanding. I'll have some questions later in this e-mail.

Wheels are definitely the way to go moving forward, as the Python world has agreed on it and that's where the tools are moving. It's possible, though, that we may face some challenges in moving to them entirely.


== Impact on Review Board ==

There's some work that needs to be done before we can move to Wheels today. Review Board/Djblets explicitly support handling eggs today, in the following ways:

1) Our Review Board extension framework (which lives in Djblets) has the expectation that eggs are used for packaging. At the time it was written, there was no Wheel. Our code looks in the PKG-INFO file for metadata (currently required to exist) and it (optionally) looks in .egg files for content.

   The code will need to be updated to either support the METADATA file (replacement for PKG-INFO), or to just stop assuming anything exists there and to fully require the data to live on the extension's class. We have the latter, actually, and just need to make this code work a bit smoother.

   Should all be doable without breaking anything fully...

   We'll also need to document these changes and provide some deprecation timeline and a migration guide.

2) Wheels, last I read, never ship .pyc files. This is in order to support different builds/versions of Python in a single Wheel package. However, commercial extensions (such as Power Pack) do not ship source, and need to ship the compiled .pyc files instead. We'd need a solution for this long-term. Short-term, so long as eggs can be installed at all, we're still okay.


== Impact on users ==

One concern I do have is legacy support. Review Board is installed on a lot of older machines that don't have any sort of modern setuptools/pip. We can't make everyone upgrade this to get a new version of Review Board, so for a while, we're going to have to ship eggs alongside any wheels. I assume wheels will take precedence if both are available?

We'll also need to keep shipping eggs/allowing eggs to be built in order to avoid problems for companies that either pull down the eggs separately from easy_install/pip, or those who build eggs for installation themselves. There would need to be a sufficient amount of time allowed for people to update their processes.

We also know that, while legacy, ez_setup.py is still needed in our builds by some companies' setups. We had an old version pointing to a set of URLs that were no longer around, and had bug reports from customers who couldn't install because of it. 


== Some questions about Wheels ==

Some other things I want to make sure of, because I can't find a lot of reliable info:

1) Are Entrypoints supported in Wheel?

2) Does pkg_resources continue to work as before? We need this for quite a lot.

3) Is there a good comparison guide somewhere showing the egg features that do or do not carry over to Wheel?

4) Are there any issues with a Wheel package depending on an egg? I assume not, but want to be sure.


== Package hosting ==

Another topic that came up was the hosting of packages.

We don't use PyPI for a few reasons:

1) We very much like having the ability to perform analysis of the downloads (beyond the simplistic ones provided by PyPI). It helps us see what versions of which packages are being downloaded, which Python versions seem to be most common right now, what operating systems are being used to download them, etc. We use this data to help make decisions on deprecation and support.

2) As mentioned above, not everyone installs using easy_install/pip, and not everyone uses the latest released versions. We know of companies out there that manually wget specific builds from current or older versions from the build directories on downloads.reviewboard.org, and copy them to an internal machine or integrate it with an internal deployment system. I believe you can get older builds on PyPI, but the interface doesn't make it as easy.

3) We distribute alphas/betas/RCs of builds, or sometimes custom builds, that we don't want people to install accidentally. It's too easy to cause problems there with PyPI. I think I read that the newer stuff can be made to install stable builds by default and not install some new alpha (might be wrong), but again, a good portion of our user base are using older installs.

   By keeping it separate, we can safely give people a command to install an alpha/beta/RC. Or a custom build, without having to advertise it to the world on PyPI.

4) We sometimes also build private staging releases of several of our components (into a staging location on downloads.reviewboard.org), run test installs and deployments off of those in a sandboxed environment, and make sure all that works before making that release public. We can't do this nearly as easily with PyPI, and would have to special-case stuff.


I've read a few threads over the past year about other large projects that also have concerns about putting everything in PyPI. I know originally, there was a big push to forcing everyone to move everything into PyPI [1], in order to ensure uptime and validity of files. There was a large thread [2] with much discussion around this, which eventually led to PEP 470 [3]. If approved, this seems like a good way to go for us.

We've made efforts to ensure all builds uploaded are accompanied by a file listing sha256sums for the build, and that every link in our indexes contain a md5sum in the URL (which pip checks for). It's not yet behind HTTPS, though, which I know is a concern for you guys. That's fixable, and I'll look into doing that soon.

Christian


[1] http://www.mail-archive.com/catal...@python.org/msg04855.html
[2] http://thread.gmane.org/gmane.comp.python.distutils.devel/21281
[3] http://legacy.python.org/dev/peps/pep-0470/


--
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org
Beanbag, Inc. - https://www.beanbaginc.com

[Stephen Gallagher]

I'd want to keep an eye on this as well (and I'll be talking to our
Python experts at Fedora). This seems like a pretty fundamentally broken
approach to packaging. One of the primary purposes of providing a
package is that end-users are guaranteed to be getting the same code no
matter what. If the actual code that is run by the end users might have
been automatically generated by whatever version of python they happen
to have on their system, it makes it into a support nightmare. (For
example, if the version of python running on the end-user system happens
to have a .pyc generation bug that didn't exist in the build-system).

I suspect that what will likely happen in distro packaging is that we
will be forced to "explode" these wheels and pre-compile them and just
place the results onto the filesystem. That's the only way to guarantee
that the .pyc's are the same on all systems.

>
> == Impact on users ==
>
> One concern I do have is legacy support. Review Board is installed on a
> lot of older machines that don't have any sort of modern
> setuptools/pip. We can't make everyone upgrade this to get a new
> version of Review Board, so for a while, we're going to have to ship
> eggs alongside any wheels. I assume wheels will take precedence if both
> are available?
>

As noted above, since distribution packaging at least is likely to
require exploding the files anyway, this should relatively easily
translate back to older distributions. It *does* mean that packaging for
e.g. RHEL 6 or Ubuntu 10.10 will *need* to be distro-packaged (or
COPR/PPA) instead of being able to rely on setuptools.

As an aside: I'd highly welcome a comaintainer for Review Board in
Fedora/EPEL. I really want to get the EPEL 6 backport of 2.x working,
but my time is finite and keeping the Fedora 21 and EPEL 7 packages up
to date is often as much time as I can spare.

> We'll also need to keep shipping eggs/allowing eggs to be built in
> order to avoid problems for companies that either pull down the eggs
> separately from easy_install/pip, or those who build eggs for
> installation themselves. There would need to be a sufficient amount of
> time allowed for people to update their processes.
>

I hope you will always include the tarball releases as well, since many
distributions require that the packages they distribute are built from
source (no shipping pre-built eggs).

> We also know that, while legacy, ez_setup.py is still needed in our
> builds by some companies' setups. We had an old version pointing to a
> set of URLs that were no longer around, and had bug reports from
> customers who couldn't install because of it.
>

I'd like to make a strong recommendation that we start treating the EPEL
packages as part of the official release, rather than something I do
after the fact (and working together to get some automated COPR builds
of the nightlies that we can run automated tests against).

This way, for those people that want to run Review Board on a long-term
stable distribution, the go-to answer would be "RHEL/CentOS" with these
officially-sanctioned packages.

Of course, if the reason they're building it themselves is because
they're carrying private patches, I suppose that's not going to solve
the problem for them.

[Gregory Szorc]

On 1/6/15 7:19 PM, Christian Hammond wrote:
> Hey everyone,
>
> Gregory Szorc just provided us with a patch to convert one of our modules to support the new Python Wheel format. Thanks for that!
>
> There were some questions along with the patch that I wanted to address to a wider audience. I thought this would be a good time to get a discussion going on the future of Python packaging and how it may impact us.
>
> This is a long e-mail, so I'll try to section things off.

Yikes, I appear to have disturbed a hornets nest. Sorry about that.

For the record, the plaintext distribution and requirement of
easy_install are what bother me most. easy_install has poor verification
(non-existent?) verification and combined with the plaintext
distribution are a gaping security hole. pip has much saner defaults
around trust management. Unfortunately, you can't `pip install
ReviewBoard` today - at least not easily. Because pip can't find binary
packages (only source distributions), it must build
{django-evolution,Djblets,ReviewBoard} from source archives. And this
likely fails due to missing dependencies required to build the media
files. Installing those dependencies is a nightmare.

If `pip install ReviewBoard` "just worked" and installed from pre-built
binary packages, I would be happy. I understand parts of that (mainly
packaging) would be hard.

More comments inline.

>
> == Wheels and Eggs ==
>
> For those who aren't familiar with the world of Python packaging, the Wheel format is a newer binary package standard designed to provide a common way to ship modules and to replace the older egg format (which is what we use). All the new packaging standards for metadata and versioning improvements are designed to work with wheels.
>
> Wheels and eggs differ in a few key ways. Wheels are an installation format, which gets unpacked on install, while eggs can either be unpacked or can remain as a single relocatable file. Content in eggs are importable and can be selectively extracted or read, using a set of APIs in the pkg_resources module. Wheels cannot do this. Eggs ship .pyc files, whereas Wheels generate them at install time (potentially problematic -- see below).
>
> Wheels are also versioned differently, in an interesting way. Unlike eggs, they don't have to be tied to a Python version, so we could, for instance, ship a single Wheel for Review Board 2.0.x, and not have to ship separate ones for Python 2.6 and 2.7. Neat, but in practice, it's not a strong benefit, as we automate all our builds, which need to package and run tests on the various versions anyway.
>
> Wheels are new, though, and not supported everywhere. One needs to have a modern version of pip/setuptools to install them or work with them. This can be a problem on older Linux distros, or companies with servers that are set up a very particular way, where upgrading components is hard/time-consuming.
>
> Now, I'm not at all an expert on Wheels. There are things I will need to be educated on, or things I'm misunderstanding. I'll have some questions later in this e-mail.
>
> Wheels are definitely the way to go moving forward, as the Python world has agreed on it and that's where the tools are moving. It's possible, though, that we may face some challenges in moving to them entirely.

The existence of wheels does not exclude non-wheel distribution. On
PyPI, it is common to upload both a wheel and a classical "sdist" source
archive. If your Python packaging tool knows about wheels and a wheel
for your system is available, it fetches the wheel. Else, it falls back
to installing from the source archive.

I'd like to see various Review Board packages released as both source
archives and wheels so people with modern tools can take advantage of
their existence. i.e. supplement eggs with wheels.

> == Impact on Review Board ==

> 2) Wheels, last I read, never ship .pyc files. This is in order to support different builds/versions of Python in a single Wheel package. However, commercial extensions (such as Power Pack) do not ship source, and need to ship the compiled .pyc files instead. We'd need a solution for this long-term. Short-term, so long as eggs can be installed at all, we're still okay.

In case .pyc-only wheels are not supported, you may wish to raise your
use case to the distutils-sig list so support may be considered.
https://mail.python.org/mailman/listinfo/distutils-sig

>
> == Impact on users ==
>
> One concern I do have is legacy support. Review Board is installed on a lot of older machines that don't have any sort of modern setuptools/pip. We can't make everyone upgrade this to get a new version of Review Board, so for a while, we're going to have to ship eggs alongside any wheels. I assume wheels will take precedence if both are available?

Yes, tools will prefer wheels.

> We'll also need to keep shipping eggs/allowing eggs to be built in order to avoid problems for companies that either pull down the eggs separately from easy_install/pip, or those who build eggs for installation themselves. There would need to be a sufficient amount of time allowed for people to update their processes.

That makes sense to me.

> We also know that, while legacy, ez_setup.py is still needed in our builds by some companies' setups. We had an old version pointing to a set of URLs that were no longer around, and had bug reports from customers who couldn't install because of it.

The presence of ez_setup.py is encouraging legacy and insecure packaging
practices. I think Review Board should be a forcing function for these
companies to upgrade their deployment methodology. I don't think adding
a few lines to the docs to say "if easy_install isn't available do X" is
too high of a penalty for those not following best practices.

> == Some questions about Wheels ==
>
> Some other things I want to make sure of, because I can't find a lot of reliable info:
>
> 1) Are Entrypoints supported in Wheel?

Yes.

> 2) Does pkg_resources continue to work as before? We need this for quite a lot.

I believe the data parts of pkg_resources continue to work. What doesn't
work is the multi-version / parallel install foo. My understanding is
Python's take on mutli-versioning is "don't do it: use virtualenvs instead."

> 3) Is there a good comparison guide somewhere showing the egg features that do or do not carry over to Wheel?

https://www.python.org/dev/peps/pep-0427/
https://packaging.python.org.org/en/latest/wheel_egg.html

> 4) Are there any issues with a Wheel package depending on an egg? I assume not, but want to be sure.

I'm not sure. But I do know that pip does *not* support installing from
eggs.

>
> == Package hosting ==
>
> Another topic that came up was the hosting of packages.
>
> We don't use PyPI for a few reasons:

> 2) As mentioned above, not everyone installs using easy_install/pip, and not everyone uses the latest released versions. We know of companies out there that manually wget specific builds from current or older versions from the build directories on downloads.reviewboard.org, and copy them to an internal machine or integrate it with an internal deployment system. I believe you can get older builds on PyPI, but the interface doesn't make it as easy.

Indeed, the PyPI web interface does not make finding old versions easy.
I suspect that's by design.

You can use pip requirements files to pin package versions. Anyone who
cares about determinism and reproducibility should be doing this.

https://pip.pypa.io/en/latest/user_guide.html#requirements-files

> 3) We distribute alphas/betas/RCs of builds, or sometimes custom builds, that we don't want people to install accidentally. It's too easy to cause problems there with PyPI. I think I read that the newer stuff can be made to install stable builds by default and not install some new alpha (might be wrong), but again, a good portion of our user base are using older installs.
>
> By keeping it separate, we can safely give people a command to install an alpha/beta/RC. Or a custom build, without having to advertise it to the world on PyPI.

By default, pip only finds and installs stable versions.

https://packaging.python.org/en/latest/installing.html#installing-prereleases

If maintaining separate alphas/betas/RCs distributions is important, I
think the preferred way to do that is to maintain a separate packaging
index and direct users at that. e.g. `pip install --index-url
https://downloads.reviewboard.org/alpha ReviewBoard`

> 4) We sometimes also build private staging releases of several of our components (into a staging location on downloads.reviewboard.org), run test installs and deployments off of those in a sandboxed environment, and make sure all that works before making that release public. We can't do this nearly as easily with PyPI, and would have to special-case stuff.

Run a separate index.

[Gregory Szorc]

On 1/7/15 5:07 AM, Stephen Gallagher wrote:
>
> On Tue, 2015-01-06 at 19:19 -0800, Christian Hammond wrote:
>> Hey everyone,
>>

>> 2) Wheels, last I read, never ship .pyc files. This is in order to
>> support different builds/versions of Python in a single Wheel package.
>> However, commercial extensions (such as Power Pack) do not ship source,
>> and need to ship the compiled .pyc files instead. We'd need a solution
>> for this long-term. Short-term, so long as eggs can be installed at
>> all, we're still okay.
>>
>
> I'd want to keep an eye on this as well (and I'll be talking to our
> Python experts at Fedora). This seems like a pretty fundamentally broken
> approach to packaging. One of the primary purposes of providing a
> package is that end-users are guaranteed to be getting the same code no
> matter what. If the actual code that is run by the end users might have
> been automatically generated by whatever version of python they happen
> to have on their system, it makes it into a support nightmare. (For
> example, if the version of python running on the end-user system happens
> to have a .pyc generation bug that didn't exist in the build-system).
>
> I suspect that what will likely happen in distro packaging is that we
> will be forced to "explode" these wheels and pre-compile them and just
> place the results onto the filesystem. That's the only way to guarantee
> that the .pyc's are the same on all systems.

No, I don't think that's how distros will do it. Instead, I suspect
distros will do what they do for all packages today: build them from
source. Distros will distribute rpms, debs, etc of Python packages that
they've built on their build farms.

Of course, savvy Python users can and do bypass the distro packaging
ecosystem as much as possible. Virtualenvs are a much saner approach to
packaging than the distro/OS model of "everything goes in a shared
namespace" so different applications don't trample on each other with
conflicting packages and versions. Virtualenvs make the dependency hell
problem largely a thing of the past. Personally, I find it a shame that
distros still inflict this model on unsuspecting users. You may enjoy
reading
http://gregoryszorc.com/blog/2014/07/15/python-packaging-do%27s-and-don%27ts/

[Stephen Gallagher]

<snip>

> > 2) Does pkg_resources continue to work as before? We need this for
> quite a lot.
>
> I believe the data parts of pkg_resources continue to work. What
> doesn't work is the multi-version / parallel install foo. My
> understanding is Python's take on mutli-versioning is "don't do it: use
> virtualenvs instead."
>

Yikes. This is actually going to make life extremely difficult for
distribution packaging. There are numerous places where multi-versioning
is necessary (case in point: in Fedora 20 we were maintaining both
stable releases of Django at the same time in the distribution, because
some packages required one version and some another. There's no easy way
to do global deployment with virtualenvs.

Now, if you can solve *that* problem...

[Christian Hammond]

That's sort of what I expected would happen.

Since Wheels are a packaging format, rather than a bundle format like Eggs were, I imagine you'd take a Wheel as input, install to a path, and ship the extracted contents in the RPM.

>
>
> >
> > == Impact on users ==
> >
> > One concern I do have is legacy support. Review Board is installed on a
> > lot of older machines that don't have any sort of modern
> > setuptools/pip. We can't make everyone upgrade this to get a new
> > version of Review Board, so for a while, we're going to have to ship
> > eggs alongside any wheels. I assume wheels will take precedence if both
> > are available?
> >
>
> As noted above, since distribution packaging at least is likely to
> require exploding the files anyway, this should relatively easily
> translate back to older distributions. It *does* mean that packaging for
> e.g. RHEL 6 or Ubuntu 10.10 will *need* to be distro-packaged (or
> COPR/PPA) instead of being able to rely on setuptools.
>
> As an aside: I'd highly welcome a comaintainer for Review Board in
> Fedora/EPEL. I really want to get the EPEL 6 backport of 2.x working,
> but my time is finite and keeping the Fedora 21 and EPEL 7 packages up
> to date is often as much time as I can spare.

Might be worth opening a thread about this on reviewboard@ to get some more eyes.

>
> > We'll also need to keep shipping eggs/allowing eggs to be built in
> > order to avoid problems for companies that either pull down the eggs
> > separately from easy_install/pip, or those who build eggs for
> > installation themselves. There would need to be a sufficient amount of
> > time allowed for people to update their processes.
> >
>
> I hope you will always include the tarball releases as well, since many
> distributions require that the packages they distribute are built from
> source (no shipping pre-built eggs).

Absolutely. That's not changing.

I expect we'd ship tarballs, eggs, and Wheels.

>
> > We also know that, while legacy, ez_setup.py is still needed in our
> > builds by some companies' setups. We had an old version pointing to a
> > set of URLs that were no longer around, and had bug reports from
> > customers who couldn't install because of it.
> >
>
> I'd like to make a strong recommendation that we start treating the EPEL
> packages as part of the official release, rather than something I do
> after the fact (and working together to get some automated COPR builds
> of the nightlies that we can run automated tests against).

The trick will be coordinating schedules. We're not often able to do the releases at the same times when you're free. We can see what we can do though. (This may be better as its own thread.)

We will soon be linking to the EPEL RBTools packages on the site, and we link to the Review Board ones now on the Get Review Board page.

> This way, for those people that want to run Review Board on a long-term
> stable distribution, the go-to answer would be "RHEL/CentOS" with these
> officially-sanctioned packages.

In theory, sure, but in practice, a lot of companies out there run on all sorts of older and newer distros with their own package repositories. We can't dictate anything here, and have a lot of different environments to work with and support.

> Of course, if the reason they're building it themselves is because
> they're carrying private patches, I suppose that's not going to solve
> the problem for them.

That too. We have paying customers with custom builds like that.

Christian

[Christian Hammond]

--
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org
Beanbag, Inc. - https://www.beanbaginc.com

-----Original Message-----
From: Gregory Szorc <gregor...@gmail.com>
Reply: reviewb...@googlegroups.com <reviewb...@googlegroups.com>>
Date: January 7, 2015 at 10:27:34 AM
To: reviewb...@googlegroups.com <reviewb...@googlegroups.com>>
Subject:  Re: Modern Python packaging and Review Board

> On 1/6/15 7:19 PM, Christian Hammond wrote:
> > Hey everyone,
> >
> > Gregory Szorc just provided us with a patch to convert one of our modules to support the
> new Python Wheel format. Thanks for that!
> >
> > There were some questions along with the patch that I wanted to address to a wider audience.
> I thought this would be a good time to get a discussion going on the future of Python packaging
> and how it may impact us.
> >
> > This is a long e-mail, so I'll try to section things off.
>
> Yikes, I appear to have disturbed a hornets nest. Sorry about that.

Nah, it's a discussion we needed to have eventually.

>
> For the record, the plaintext distribution and requirement of
> easy_install are what bother me most. easy_install has poor verification
> (non-existent?) verification and combined with the plaintext
> distribution are a gaping security hole. pip has much saner defaults
> around trust management. Unfortunately, you can't `pip install
> ReviewBoard` today - at least not easily. Because pip can't find binary
> packages (only source distributions), it must build
> {django-evolution,Djblets,ReviewBoard} from source archives. And this
> likely fails due to missing dependencies required to build the media
> files. Installing those dependencies is a nightmare.
>
> If `pip install ReviewBoard` "just worked" and installed from pre-built
> binary packages, I would be happy. I understand parts of that (mainly
> packaging) would be hard.

I wish it did. This has been a source of frustration on our part for some time. It would have been nice if pip would have just supported eggs in the past. Unfortunately, due to the packaging requirements we have, we had to recommend easy_install. I know we're not the only project that has to deal with this nonsense.

If we can distribute both wheels and eggs without problems, and pip is happy to handle our wheels, then life is good. Of course, we'll have to document what to do based on the version of pip.

By the way, we now have downloads.reviewboard.org available on https, but optionally because, of course, easy_install totally fails with it. *sigh*

I've actually given thought to writing our own package tool (wrapper, really) for RB that verifies builds, lets you stay pinned to a stable version range (for those on, say, 1.7 who want the latest-and-greatest 1.7.x but not a 2.0.x), and to make extension fetching/installation/lookup easy. And, of course, to avoid the nonsense above.

> The existence of wheels does not exclude non-wheel distribution. On
> PyPI, it is common to upload both a wheel and a classical "sdist" source
> archive. If your Python packaging tool knows about wheels and a wheel
> for your system is available, it fetches the wheel. Else, it falls back
> to installing from the source archive.
>
> I'd like to see various Review Board packages released as both source
> archives and wheels so people with modern tools can take advantage of
> their existence. i.e. supplement eggs with wheels.

I'm fine distributing wheels alongside source dists and eggs, provided that we don't make things any more difficult for existing users of eggs.

>
>
> > == Impact on Review Board ==
>
> > 2) Wheels, last I read, never ship .pyc files. This is in order to support different builds/versions
> of Python in a single Wheel package. However, commercial extensions (such as Power Pack)
> do not ship source, and need to ship the compiled .pyc files instead. We'd need a solution
> for this long-term. Short-term, so long as eggs can be installed at all, we're still okay.
>
> In case .pyc-only wheels are not supported, you may wish to raise your
> use case to the distutils-sig list so support may be considered.
> https://mail.python.org/mailman/listinfo/distutils-sig

Yeah, we'll have to look into this.

> The presence of ez_setup.py is encouraging legacy and insecure packaging
> practices. I think Review Board should be a forcing function for these
> companies to upgrade their deployment methodology. I don't think adding
> a few lines to the docs to say "if easy_install isn't available do X" is
> too high of a penalty for those not following best practices.

I agree in principle, but this has been a support issue in the past. We can revisit it at some point, but it'd have to be done for a major release and not a point release.

>
> > == Some questions about Wheels ==
> >
> > Some other things I want to make sure of, because I can't find a lot of reliable info:
> >
> > 1) Are Entrypoints supported in Wheel?
>
> Yes.

Okay, good. I thought so, but found some mixed answers on that. That makes things a lot more reasonable.

>
> > 2) Does pkg_resources continue to work as before? We need this for quite a lot.
>
> I believe the data parts of pkg_resources continue to work. What doesn't
> work is the multi-version / parallel install foo. My understanding is
> Python's take on mutli-versioning is "don't do it: use virtualenvs instead."

Ouch. That'll be a problem.

Looks like Stephen already tackled part of why.

> > 2) As mentioned above, not everyone installs using easy_install/pip, and not everyone
> uses the latest released versions. We know of companies out there that manually wget
> specific builds from current or older versions from the build directories on downloads.reviewboard.org,
> and copy them to an internal machine or integrate it with an internal deployment system.
> I believe you can get older builds on PyPI, but the interface doesn't make it as easy.
>
> Indeed, the PyPI web interface does not make finding old versions easy.
> I suspect that's by design.
>
> You can use pip requirements files to pin package versions. Anyone who
> cares about determinism and reproducibility should be doing this.
>
> https://pip.pypa.io/en/latest/user_guide.html#requirements-files

These features exist, but not everyone who installs RB has knowledge of these capabilities or even general packaging/Linux/Python experience. We can't require these features for the admins out there. It'll turn into a support nightmare.

Some of the companies I was referring to don't even use Python packages in the way that you'd expect. They use more custom build methods to essentially re-package Review Board and dependencies in their own build stores, deploying them using custom scripts or Chef or what have you. For this, they need easy access to all of our builds through a browsable index like we have. Some of them pull from directories we set up for them on here, that we would not want to put up PyPI.

We'll wait until the Python packaging guys sort out how to allow projects to reasonably provide externally-verified hosting of builds, and then make use of that solution. We just can't put everything directly into PyPI without causing too much disruption and limiting our own capabilities.

> > 3) We distribute alphas/betas/RCs of builds, or sometimes custom builds, that we don't
> want people to install accidentally. It's too easy to cause problems there with PyPI.
> I think I read that the newer stuff can be made to install stable builds by default and not
> install some new alpha (might be wrong), but again, a good portion of our user base are
> using older installs.
> >
> > By keeping it separate, we can safely give people a command to install an alpha/beta/RC.
> Or a custom build, without having to advertise it to the world on PyPI.
>
> By default, pip only finds and installs stable versions.
>
> https://packaging.python.org/en/latest/installing.html#installing-prereleases
>
> If maintaining separate alphas/betas/RCs distributions is important, I
> think the preferred way to do that is to maintain a separate packaging
> index and direct users at that. e.g. `pip install --index-url
> https://downloads.reviewboard.org/alpha ReviewBoard`

We do something similar today. We can tell people to simply do, say:

    sudo easy_install \
        -f http://downloads.reviewboard.org/releases/ReviewBoard/2.5/ \
        -U ReviewBoard

So that wouldn't be too different.

(That doesn't actually exist yet, people, so no need to try it!)

Christian