virtualenv 2.0

[Donald Stufft]

I think we should talk about virtualenv 2.0!

So current status is virtualenv is a bunch of hacks to enable a virtualized
environment. In Python 3.3 the ability to have a virtualized environment was
added straight to the Python interpreter. After poking around with it, it feels
way less hacky in general and I think it should be preferred over our older
hacks.

So I believe we should make it so virtualenv will use the 3.3+ venv module to
build the virtual environment if it is available, and fall back to hacks if
it is not. It has an API we can use to customize the build process which I
think we should use in order to:

* Install our own scripts if we deem them better/more consistent than the
CPython ones, or we should ape the CPython ones in the hacky method.

* Install our own version of pip. Our virtualenv package will always be more
up to date with pip, etc than the stdlib ensurepip will be in 3.4+ (and there
is no ensurepip in 3.3) so I think it makes sense to add our own.

* Perhaps layer additional features ontop of the venv module so as to provide
improvements in general, such as the ability to install an arbitrary set of
packages into each fresh new environment.

Essentially my proposal is let's let the interpreter take care of the low level
handling of actually virtualizing an environment while we layer on additional
stuff.

I also think that we should rework our hacks so they work more like the stdlib's
venv module. There are a lot of the same basic ideas just slight variations that
I think would be useful to normalize such as sys.real_prefix vs sys.base_prefix
and the config file vs dynamically discovering it.

I think given the nature of this shift, it would make sense to call this a 2.0
and also use it as an opportunity to clean up some of the deprecated or rough
edges of virtualenv such as:

* --no-site-packages
* --never-download
* --setuptools
* --distribute
* --unzip-setuptools
* WORKING_ENV warning

I also think it makes sense to call it a 2.0 because it is likely to be a
decent shift in the code base and will probably break a thing or two in the
process.

Thoughts?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Marcus Smith]

from what I recall the idea did come up over a year ago to have virtualenv wrap venv, when it's present, but nobody acted on it.

so yea, I like it,  and if in the process virtualenv actually grew more than 7 tests, that would be great!!

"2.0" to do a clean drop of old params sounds fine with me.

[Paul Moore]

On 4 April 2014 06:14, Marcus Smith <qwc...@gmail.com> wrote:
> from what I recall the idea did come up over a year ago to have virtualenv
> wrap venv, when it's present, but nobody acted on it.
> so yea, I like it, and if in the process virtualenv actually grew more than
> 7 tests, that would be great!!
> "2.0" to do a clean drop of old params sounds fine with me.

+1. I had thought about this in the past, but never had time to do
anything real with it.

To what extent do we want to carry over the various system-specific
hacks in virtualenv? There's a lot of "if you're on system X, we need
to look in <obscure directory>". Does the core venv support those
types of system? Should we? I guess I'm asking to what extent we're
intending a complete rewrite (with the associated risk of missing code
that fixes bugs reported ages ago).

Paul

[Donald Stufft]

The way the core venv works is actually pretty ingenious, the only real
difference between the sys.path of a global Python and the sys.path of
a venv python is that the site-packages dir points to the venv one. So
all of those hacks we need for various lib dirs and such should be
completely unnecessary except for *maybe* the lib64 one.

I have a local proof of concept working on Python 2.6 that only relies on
symlinking one file, and adding one file in order to backport the 3.3+ venv
system into Python 2.6. There’s still some gotchas and stuff to sort out
but I more or less have it working.

[Ned Deily]

In article <39D0DDE8-09F0-4058...@stufft.io>,

Donald Stufft <don...@stufft.io> wrote:
> The way the core venv works is actually pretty ingenious, the only real
> difference between the sys.path of a global Python and the sys.path of
> a venv python is that the site-packages dir points to the venv one. So
> all of those hacks we need for various lib dirs and such should be
> completely unnecessary except for *maybe* the lib64 one.
>
> I have a local proof of concept working on Python 2.6 that only relies on
> symlinking one file, and adding one file in order to backport the 3.3+ venv

> system into Python 2.6. There零 still some gotchas and stuff to sort out

> but I more or less have it working.

IIRC, there were changes needed in core Python to fully support venv.
AFAIK, they would only present be in 3.3+, in particular not in 2.7.
One that I remember and was able to find quickly:

http://hg.python.org/cpython/rev/b79d276041a8

I don't recall if there were others.

--
Ned Deily,
n...@acm.org

[Ionel Cristian Mărieș]

Hello,

You've mentioned '--no-site-packages' - isn't that option removed for some time now ? Did you mean '--system-site-packages' ? If so, please don't remove it, I believe there are still valid usecases for it.

Also, does the py3's venv support access to the global site-packages ?

Thanks,
-- Ionel M.

[Donald Stufft]

On Apr 4, 2014, at 5:04 AM, Ionel Cristian Mărieș <con...@ionelmc.ro> wrote:

Hello,

You've mentioned '--no-site-packages' - isn't that option removed for some time now ? Did you mean '--system-site-packages' ? If so, please don't remove it, I believe there are still valid usecases for it.

No I meant —no-site-packages. The option is still present in the command line, it just does nothing when you use it.

Also, does the py3's venv support access to the global site-packages ?

Yes.

[Donald Stufft]

On Apr 4, 2014, at 4:19 AM, Ned Deily <n...@acm.org> wrote:

> In article <39D0DDE8-09F0-4058...@stufft.io>,
> Donald Stufft <don...@stufft.io> wrote:
>> The way the core venv works is actually pretty ingenious, the only real
>> difference between the sys.path of a global Python and the sys.path of
>> a venv python is that the site-packages dir points to the venv one. So
>> all of those hacks we need for various lib dirs and such should be
>> completely unnecessary except for *maybe* the lib64 one.
>>
>> I have a local proof of concept working on Python 2.6 that only relies on
>> symlinking one file, and adding one file in order to backport the 3.3+ venv

>> system into Python 2.6. There¹s still some gotchas and stuff to sort out

>> but I more or less have it working.
>
> IIRC, there were changes needed in core Python to fully support venv.
> AFAIK, they would only present be in 3.3+, in particular not in 2.7.
> One that I remember and was able to find quickly:
>
> http://hg.python.org/cpython/rev/b79d276041a8
>
> I don't recall if there were others.
>
> --
> Ned Deily,
> n...@acm.org
>

Yes, the 3.3+ venv module required interpreter changes to *cleanly* implement
virtual environments. However virtualenv has always relied on dirty hacks to
function so the main change there would be to change those dirty hacks to end
up in a state much more similar to that of 3.3+’s venv module.

For the record, the way it works is:

Python uses the os.py module as a sort of sentinel value to determine where the
stdlib is. It will look in a relative directory first. So if you have a file structure like…

.
├── bin
│ └── python
└── lib
└── python2.6
└── os.py

Then the os.py in that directory will trigger the Python interpreter to believe that the
stdlib should be in that lib directory. virtualenv exploits this and symlinks os.py into
that directory, and then also installs a custom site.py [1] in that directory which does all
the heavy lifting of actually setting up our sys.path in the virtual environment.There is
also a custom distutils/__init__.py file as well that is installed to do some monkey patching
there. In order for the site.py functionality to work, virtualenv symlinks a number of things
into the lib directory so that they are available for import prior to site.py having been executed
to finish setting up the sys.path so that the regular stdlib is visible in the virtual environment.

So prior to 3.3 virtualenv will still rely on this, just the hacks will be cleaned up to remove
things we don’t need to support Python 2.6+ and 3.2+ and to make the hacks create a
virtual environment that looks as much like a 3.3+ environment as possible. Then it’ll
just use the 3.3 venv module if it’s at all possible so that the hacks are treated as a legacy
fallback mode and not the primary and sole method of isolation.

[1] https://github.com/pypa/virtualenv/blob/develop/virtualenv_embedded/site.py
[2] https://github.com/pypa/virtualenv/blob/develop/virtualenv_embedded/distutils-init.py

P.S. Figuring this all out has been very.. “educational”.

[Paul Moore]

On 4 April 2014 04:28, Donald Stufft <don...@stufft.io> wrote:
> So I believe we should make it so virtualenv will use the 3.3+ venv module to
> build the virtual environment if it is available, and fall back to hacks if
> it is not.

One interesting exercise I just tried is to run the full Python test
suite under virtualenv and venv. More tests failed under virtualenv -
essentially venv successfully ran the whole test suite (test_site
failed, because of some extra startup imports, but that's it) whereas
virtualenv failed on a few others (compileall, venv (doh), trace, and
some of the tcl modules).

So it looks like venv is a more reliable virtualisation method (not
surprising as it has core support) and so we should definitely use it
if it's available.

Paul