Heads up: "mach" Python virtualenv behaviour changes ("./mach create-mach-environment" must be run ...)

[Mitchell Hentges]

Hey, there's some modifications to our handling of Mach virtualenvs that are having a little more disruption than I anticipated, so I'd like to give a quick summary of the current state, issues, and workarounds.

Specifically, we recently started asserting that the Mach virtualenv is up-to-date, and if it isn't then we prompt that it should be re-created: "./mach create-mach-environment must be run when moving across repositories".

TL;DR

If you're not using multiple repositories, you should now only be affected when you update over a revision that changes the structure of the Mach virtualenvs such as this. There should only be another couple "structure-changing" revisions before automatic-virtualenv-recreation lands.

If you are using multiple repositories, you're probably being prompted to re-create the virtualenv whenever you change between them.

You can work around this by setting the `MACH_USE_SYSTEM_PYTHON=1` environment variable - more details here.

Upcoming improvements

1. The Mach virtualenv will be re-created automatically
2. Each local repository will have its own Mach virtualenv

Background

Mach stores some Python state in a "virtualenv" that usually sits underneath your home folder.
However, over time, this virtualenv could become out-of-date without being noticed, thereby causing confusing failures when Mach commands were run.

This was exacerbated by all the work happening to improve the Python dependency/import scope situation, which required that virtualenv state be up-to-date.

To address this, the Mach virtualenv is now asserted to be up-to-date whenever a Mach command is run.

Why not back out the "assert up-to-date" revision?

This would cause a catch-22, since:
1. Improvements to the virtualenv logic require that the Mach virtualenv is kept up-to-date with the changes.

2. The "assert up-to-date" issue requires the upcoming improvements to be resolved.

-----

Sorry for the disruption, and I'm looking forward to getting to the other side of this usability issue.

--

Mitchell Hentges

Engineering Workflow

Mozilla