automating Django releases

[Tim Graham]

After doing releases about once a month for a while, I'm thinking it would be nice if releasing Django could be a bit more automated. As far as I know, the process hasn't changed too much in 10 years, while the state of Python packaging has improved.

Currently doing a release requires bumping the VERSION tuple in django/__init__.py [0], creating tarball/wheel and checksum files, asking someone in IRC to verify the checks look fine, and upload files to various locations [1]. My idea would be to use setuptools_scm [2] (that choice because this is the only tool I know about) to eliminate the need to bump/set a VERSION tuple and instead fetch the version information based on git tags. Something like (based on what I found in djanog-hosts [3]):

__version__ = pkg_resources.get_distribution('django').version

One issue I'm not sure how to deal with is how to provides backwards-compatibility for django.VERSION, the tuple format that looks something like:

VERSION = (1, 11, 0, 'alpha', 0)

I think removing or deprecating it would be quite disruptive for the community as I've seen lots of third-party apps that do checks like "if django.VERSION < (1, 9): ...", for example. I suspect it's possible to parse the version string to create something similar, but I wondered if anyone has experience with that.

Finally, uploading to PyPI and djangoproject.com (release and checksum files) could be done automatically by Jenkins or something similar.

Thanks for your comments, concerns, and suggestions.

[0] https://github.com/django/django/blob/6252fd6314f66d2d303cc47c791ffefd27169b42/django/__init__.py#L5
[1] https://docs.djangoproject.com/en/dev/internals/howto-release-django/#actually-rolling-the-release
[2] https://github.com/pypa/setuptools_scm
[3] https://github.com/jazzband/django-hosts/blob/0eae77b31927ddb0267943d6f9bcfc22f14706f2/django_hosts/__init__.py#L11

[Raffaele Salmaso]

On Fri, Nov 25, 2016 at 8:49 PM, Tim Graham <timog...@gmail.com> wrote:

After doing releases about once a month for a while, I'm thinking it would be nice if releasing Django could be a bit more automated. As far as I know, the process hasn't changed too much in 10 years, while the state of Python packaging has improved.

Currently doing a release requires bumping the VERSION tuple in django/__init__.py [0], creating tarball/wheel and checksum files, asking someone in IRC to verify the checks look fine, and upload files to various locations [1]. My idea would be to use setuptools_scm [2] (that choice because this is the only tool I know about) to eliminate the need to bump/set a VERSION tuple and instead fetch the version information based on git tags. Something like (based on what I found in djanog-hosts [3]):

__version__ = pkg_resources.get_distribution('django').version

Oh, this will never work with my setup (*.dist-info are not saved).

It is always really needed the *.dist-info metadata?

Need to check this...

(just tried a sample and I can't use django-hosts for this...

pkg_resources.DistributionNotFound: The 'django-hosts' distribution was not found and is required by the application

not nice)

One issue I'm not sure how to deal with is how to provides backwards-compatibility for django.VERSION, the tuple format that looks something like:

VERSION = (1, 11, 0, 'alpha', 0)

I think removing or deprecating it would be quite disruptive for the community as I've seen lots of third-party apps that do checks like "if django.VERSION < (1, 9): ...", for example. I suspect it's possible to parse the version string to create something similar, but I wondered if anyone has experience with that.

if tuple(map(int, pkg_resources.get_distribution('django').version.split('.')[:2])) < (1, 11):

    ...

and this doesn't work for '1.10a0'

--

| Raffaele Salmaso
| https://salmaso.org
| https://bitbucket.org/rsalmaso
| https://github.com/rsalmaso

[Raphaël Barrois]

Hi,

The process I use goes the other way around: with an automated tool, we type ``make release-minor`` and get:
1. Update the VERSION tuple
2. Update the ChangeLog with a release date
3. Create the adequate git tag
4. Prepare the ChangeLog for the next release

The tool we use is not open-source yet, however it seems very similar in design to https://pypi.python.org/pypi/zest.releaser.

Moreover, this pattern does not change the way your package is installed or detects its version number: that number is always available in the source code,
no matter how the package was installed.

--
Raphaël

[Josh Smeaton]

I think automating as much of the release process is a fantastic idea. Regarding bumping the release numbers, have you seen the bumpversion project https://pypi.python.org/pypi/bumpversion?

After some trial and error, I seem to have a configuration that works with the existing version tuple in django/__init__.py. Bumping the release number will change the VERSION tuple. The existing get_current_version will continue to work as normal, as will the setup.py. You can experiment with the tag/commit options and other options - the hard part is getting the serialize and parse options correct. This configuration goes in the existing setup.cfg.

[bumpversion]

current_version = (1, 11, 0, 'alpha', 0)  # you have to set this the first time, then bumpversion updates it along with the VERSION tuple in django/__init__.py

commit = False

tag = False

serialize = ({major}, {minor}, {patch}, {prerel}, {prerelnum})

parse = \((?P<major>\d+),\s(?P<minor>\d+),\s(?P<patch>\d+),\s(?P<prerel>\'alpha\'|\'beta\'|\'rc\'),\s(?P<prerelnum>\d+)\)

[bumpversion:file:django/__init__.py]

[bumpversion:part:prerel]

values = 

'alpha'

'beta'

'rc'

And here's an example run:

$ bumpversion patch --list

current_version=(1, 11, 0, 'alpha', 1)

new_version=(1, 11, 1, 'alpha', 0)

$ bumpversion patch --list

current_version=(1, 11, 1, 'alpha', 0)

new_version=(1, 11, 2, 'alpha', 0)

$ bumpversion minor --list

current_version=(1, 11, 2, 'alpha', 0)

new_version=(1, 12, 0, 'alpha', 0)

$ bumpversion prerel --list

current_version=(1, 12, 0, 'alpha', 0)

new_version=(1, 12, 0, 'beta', 0)

$ bumpversion prerel --list

current_version=(1, 12, 0, 'beta', 0)

new_version=(1, 12, 0, 'rc', 0)

$ bumpversion prerelnum --list

current_version=(1, 12, 0, 'rc', 0)

new_version=(1, 12, 0, 'rc', 1)

$ bumpversion major --list

current_version=(1, 12, 0, 'rc', 1)

new_version=(2, 0, 0, 'alpha', 0)

With regards to Jenkins, you could have a release project with parameters of BRANCH and VERSIONPART and whatever else was needed. The VERSIONPART would feed into the bumpversion command.

I think this way reduces backward compatibility concerns.

[Marc Tamlyn]

I'm pretty confident we can just automate the process you currently do.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f36b9371-f292-47c5-a417-c6557e5a2557%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Florian Apolloner]

Hi,

+1 to automating as much as possible; one thing though: Personally I do not like uploading via Jenkins as a security issue there might compromise our PyPi account. This is especially important if a build slave where able to do something to the master… So I think as first step it would be nice to have a script which automates everything we have locally and then if we want to, we can still put it into Jenkins.

Cheers,
Florian

[Aymeric Augustin]

I agree that most of the workflow could be scripted.

The manual verification step for checksums exists because there was an error in checksums once in the distant past and we took some flak for that. Since you’re making releases regularly, you’re much less likely to make a mistake. A script (hopefully) wouldn’t make mistakes either. So that step can be dropped.

To expand on what Florian said, as far as I know, the ops team treats Jenkins as an insecure machine. Since it contains only open-source software, there’s nothing to protect. It runs on an isolated domain (djangoci.com) to prevent a compromise from affecting djangoproject.com services. Significant steps to increase its security would have to be taken before it could be trusted to generate and sign releases.

Best regards,

-- 

Aymeric.

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3a3a59e7-b4e4-4bc9-8538-67d040459840%40googlegroups.com.

[Josh Smeaton]

Appreciate the security concerns of the jenkins boxes, but Jenkins is the obvious choice as a task runner. My first thought would be to bring up a special slave that was particularly hardened and walled off from the regular PR runners. I'm unsure how far access to the slave from the master could be minimised via permissions etc. At that point though, it'd probably be easier to have a simple bash script on this make believe instance that ops could ssh to and run. Or a combination of both - using jenkins for the version bump and package build, pushing the artifact elsewhere, then manually running a sign + deploy step.

[Carl Meyer]

Hi Tim,

On 11/25/2016 11:49 AM, Tim Graham wrote:
> After doing releases about once a month for a while, I'm thinking it
> would be nice if releasing Django could be a bit more automated. As far
> as I know, the process hasn't changed too much in 10 years, while the
> state of Python packaging has improved.
>
> Currently doing a release requires bumping the VERSION tuple in
> django/__init__.py [0], creating tarball/wheel and checksum files,
> asking someone in IRC to verify the checks look fine, and upload files
> to various locations [1]. My idea would be to use setuptools_scm [2]
> (that choice because this is the only tool I know about) to eliminate
> the need to bump/set a VERSION tuple and instead fetch the version
> information based on git tags. Something like (based on what I found in
> djanog-hosts [3]):
>
> __version__ = pkg_resources.get_distribution('django').version

+1 for more automation, but I recommend simply automating the bumping of
the hardcoded VERSION in the source tree (should not be hard to do). I
recommend against relying on git for release versioning, and even more
strongly against depending on pkg_resources to get the right version.
It's just so much less robust and reliable than having a version
hardcoded into the source tree.

The above pkg_resources line only works if the current source tree is
also installed in the current Python environment. If you are working in
one source tree, but some other Django is installed, you will get the
wrong version. If you are working in a source tree you haven't
installed, you will get no version at all. We used to use a similar
pkg_resources-based scheme in pip, and it caused no end of trouble.

The git-based version fetching only works if you are working from a git
repo, not an exported source tree. An exported source tree, independent
of git, should still know at least what released version it represents.
I am sure if we break this, it will break somebody's workflow.

Carl