Post mortem on today's packaging error.

[Carlton Gibson]

Hi all. 

This morning I released four versions of Django. Three of which, for 2.1, 2.0 and 1.11. (i.e. all the actually supported versions) were broken. 

In the package were additional files from `master`/2.2 which shouldn't have been there. 

This afternoon I have released follow-ups to correct this issue. 

First of all, sorry about that, and for any inconvenience caused. 

Then, these are process issues so, how can we do better next time? 

I'm not 100% sure what occurred. 

* The history in Git is correct. 

* I must have had the right commits checked out, because the package metadata is correct. (Filenames, version numbers and so on.) 

My best guess is that I've failed to `git clean` correctly before building each release. 

I'm not certain here because switching between branches doesn't leave the repos in an unclean state, and I'm pretty sure it was clean, but this seems the most likely error.

Q: is there a nice git command to "assert I'm at exactly this tag"?  

Steps I've taken: 

* Moved the `git clean` step into the helper script used to build the packages. No chance of then missing it. 

* Added a `diff`-step after building just to make sure what's in the `django` module matches the checkout. 

 

The second of these, whilst just a visual check, would have worked with what ended up in the package vs what was in my working tree when I checked later, but I'm not sure it would have caught the issue when the package was created (because presumably my working tree was wrong at that point). 

A similar issue affects the checksums, which check that nothing changed since it was packaged, but not that the right things were packaged...

(Similarly, pip install worked without error.) 

On the new 1.11 package, I created a virtualenv with Python 2 and ran the test suite. This worked so it should be good. But it would be good to automate this. 

(I'll look into it.) We don't ship the tests in the wheel, so we have to use the src-dist for this. (We could then diff the two django modules to make sure they were the same.) 

On the other packages I visually looked for incorrect files, but, beyond specific migrations (etc) that were known bad from the previous release, this doesn't generalise. 

"zaytsev" on IRC suggested running `makemigrations --check`, so having a test project to install against might also be worth it. 

(I'd guess running the test suite would be sufficient...) 

Any other thoughts very welcome. 

Kind Regards,

Carlton

[Andrew Godwin]

I also ran into this several times while releasing Channels - several releases had the wrong files ship out in them due to Git weirdness.

In the end, my solution was to build the release artifacts over on Travis to guarantee a fresh build environment each time, but I doubt that would work for Django. Adding "git clean" to the script is probably good enough, but I might be tempted to make a script where you pass it a commit hash and it checks it out to a fresh temporary directory and packages from there?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jamesie Pic]

Hi Carlton !

Seems like you're having as much fun as I had when doing releases
manually :D Just sharing some food for thought here.

Nowadays I have it automated and rely on setupmeta to keep myself away
from touching setup.py, and just have to push git tags :
http://github.com/zsimic/setupmeta

Then a silly little script like that to publish to pypi :
https://yourlabs.io/oss/shyml/blob/master/shyml/sh.yml

All python modules on yourlabs.io/oss have this kind of setup ... of
course you're still stuck on jenkins so I can understand it's not as
flexible as say gitlab-ci or drone-ci, but there's probably also a
way.

Hope this helps

Keep up the good work, you'll make it !

Have a great day

--
∞

[Adam Johnson]

Thanks for the detailed post mortem Carlton.

Andrew’s suggested approach to do at least a checkout to a fresh directory makes sense to me. Even got checkout and clean aren’t enough to bring an exisiting checkout folder to the same state as git won’t touch files in the gitignore. Note you can do a checkout or a local repo to a different directory, you wouldn’t need to reclone from a different remote.

Jamesie’s suggestion to use CI is also valid but a bunch more work. I guess the main advantage is you get a blank slate container to work in, which a fresh checkout to a temp dir provides most of the gain for less work.

That’s my two cents

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1_rgn8BnHHS-0%2B05yMKFi5g%2Bq%2BKmHZVz1-SP-sgxY-%3D3Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Florian Apolloner]

On Monday, February 11, 2019 at 11:01:55 PM UTC+1, Adam Johnson wrote:

Jamesie’s suggestion to use CI is also valid but a bunch more work. I guess the main advantage is you get a blank slate container to work in, which a fresh checkout to a temp dir provides most of the gain for less work.

If I remember we have been hesitant in the past because that would require us to give credentials to PyPi etc to the CI service. That said I think that is a risk we could take.

Cheers,

Florian

[Aleksi Häkli]

Jazzband (https://jazzband.co/about/releases) uses an approach that builds and pushes the PyPI packages to an intermediate repository that is owned by the Jazzband organization.

The Jazzband intermediate repository then allows publishing them from the Jazzband organization to PyPI via a push-button deployment from the Travis build. 

This eliminates the need for having the public PyPI warehouse credentials published to Travis or other target, but requires setting up a private repository.

An approach like this (2) of course introduces two additional components to the trust chain compared to the current model (1):

1. With a simple PyPI upload, only the PyPI warehouse and the uploader has to be ultimately trusted, and package signatures are easy to check against known public PGP keys with 2 parties of trust, but

2. with an intermediate private PyPI upload from e.g. Travis. both Travis and the private intermediate server have to be trusted in addition to PyPI warehouse and the original author with 4 parties of trust.

[Luke Plant]

Hi Carlton,

I have had problems in the past with `setup.py sdist` and friends actually depending on existing artefacts in build directories from previous runs. A quick google turns up this bug report - https://github.com/pypa/setuptools/issues/436 - originally filed by .... oh, me :-)  and it is still open. It seems that `git clean -dfx` does clean out these files, so adding that to the release Makefile should fix the issue if this was the root problem. The release process document needs to be adjusted to mention that wanted build artefacts need to be moved somewhere else before preparing the next release, otherwise they are going to get deleted before they are uploaded.

Luke

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90c531fc-3d5b-4d9e-a55f-b88f2e04bb54%40googlegroups.com.

[Carlton Gibson]

Hi All, 

Thanks for all the comments. 

Cloning to a tmp folder and checking out a specific commit is both quick and easy. That's not a bad idea: it should eliminate the point of failure that occurred here. 

(The `git clean` step should do that too, we use the `fdx` flags to eliminate ignored files etc, and it does remove build and dist dirs, but... well... something went wrong.) 

I want to add automated running of the test suite against the generated src-dist. That would have caught the issues too. This seems to me to be the most reliable way to verify that what got packaged was in fact correct. 

I don't think we need to automate any further the final actual shipping of the package. `twine upload -s` is pretty easy, and I do worry about us putting the signing keys on even a private server really. 

Grrr. Definitely sub-par yesterday. Onwards! :)

C.