Virtualenv shell

108 views
Skip to first unread message

anatoly techtonik

unread,
Jun 26, 2011, 1:15:16 PM6/26/11
to python-v...@googlegroups.com
What I dislike most about virtualenv is that it becomes a rather common way to install Python web apps and yet it works in a very uncommon way both on Linux and Windows confusing system administrators with its own activate/deactivate environment logic. Are there any reasons (besides 'nobody had done this yet') against making virtualenv much like 'shell' session (probably using system shell) which you run and then exit from (instead of deactivating)?

Tres Seaver

unread,
Jun 26, 2011, 1:57:11 PM6/26/11
to python-v...@googlegroups.com


'activate' and 'deactivate' are entirely unnecessary -- I never use
them, and in fact consider them "attractive nuisances". For a sysadmin,
the correct choice is to just use the full path to the python (or
application-installed scripts) when configuring automated startup /
shutdown: they work perfectly without any of the shell majyk.


Tres.
--
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

Sebastien Douche

unread,
Jun 26, 2011, 3:02:36 PM6/26/11
to python-v...@googlegroups.com
On Sun, Jun 26, 2011 at 19:57, Tres Seaver <tres....@gmail.com> wrote:
> 'activate' and 'deactivate' are entirely unnecessary -- I never use
> them, and in fact consider them "attractive nuisances".  For a sysadmin,
> the correct choice is to just use the full path to the python (or
> application-installed scripts) when configuring automated startup /
> shutdown:  they work perfectly without any of the shell majyk.

+1. Same here.

--
Sebastien Douche <sdo...@gmail.com>
Twitter : @sdouche

Jakub Vysoky

unread,
Jun 27, 2011, 3:15:18 AM6/27/11
to python-v...@googlegroups.com
my sysadmin me:
+1 for specifying full path to python binary

my dev me:
+1 for removing differences between windows and linux and porting
virtualenv activate/deactivate and virtualenv-wrapper more to the
python world (but I know it is pretty hard to do so...)

> --
> You received this message because you are subscribed to the Google Groups "virtualenv" group.
> To post to this group, send email to python-v...@googlegroups.com.
> To unsubscribe from this group, send email to python-virtual...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/python-virtualenv?hl=en.
>
>

--
Jakub Vysoky

mob: +420 605 852 377
jab: jakub....@gmail.com
twit: https://twitter.com/kvbik

anatoly techtonik

unread,
Jun 27, 2011, 6:26:43 AM6/27/11
to python-v...@googlegroups.com
It is not solely about startup and shutdown. There are quite many operations that you need to do inside virtualenv periodically. Like upgrading, installing or uninstalling packages (virtualenv specific Trac plugins in my case), executing python modules and tools so that changes are persistent to a given environment etc.

Can you guarantee that every script and executable installed using executables from virtualenv bin/ directory (or Scripts/ on Windows) will use this virtualenv and won't resort to globally installed Python? It doesn't seem realistic to me. Perhaps I don't know all the majyk.

Tres Seaver

unread,
Jun 27, 2011, 9:24:04 AM6/27/11
to python-v...@googlegroups.com
On 06/27/2011 06:26 AM, anatoly techtonik wrote:

> It is not solely about startup and shutdown. There are quite many operations
> that you need to do inside virtualenv periodically. Like upgrading,
> installing or uninstalling packages (virtualenv specific Trac plugins in my
> case), executing python modules and tools so that changes are persistent to
> a given environment etc.

I do those by spelling 'bin/python', 'bin/easy_install',
'bin/<app-provided-script>', etc. from within the envirohmnent.

> Can you guarantee that every script and executable installed using
> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
> this virtualenv and won't resort to globally installed Python? It doesn't
> seem realistic to me. Perhaps I don't know all the majyk.

Yes, that feature is part of the design of virtualenv. The hashbang
line of every installed script is modified to use the virtualenv's python.

As I said, I make heavy use of multiple virtualenv's daily without ever
once sourcing the 'activate' script.

Carl Karsten

unread,
Jun 27, 2011, 10:28:06 AM6/27/11
to python-v...@googlegroups.com
On Mon, Jun 27, 2011 at 8:24 AM, Tres Seaver <tres....@gmail.com> wrote:
> On 06/27/2011 06:26 AM, anatoly techtonik wrote:
>
>> It is not solely about startup and shutdown. There are quite many operations
>> that you need to do inside virtualenv periodically. Like upgrading,
>> installing or uninstalling packages (virtualenv specific Trac plugins in my
>> case), executing python modules and tools so that changes are persistent to
>> a given environment etc.
>
> I do those by spelling 'bin/python', 'bin/easy_install',
> 'bin/<app-provided-script>', etc. from within the envirohmnent.
>
>> Can you guarantee that every script and executable installed using
>> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
>> this virtualenv and won't resort to globally installed Python? It doesn't
>> seem realistic to me. Perhaps I don't know all the majyk.
>
> Yes, that feature is part of the design of virtualenv.  The hashbang
> line of every installed script is modified to use the virtualenv's python.

Define "installed script" - If it doesn't cover everything, this is a
land mine.

>
> As I said, I make heavy use of multiple virtualenv's daily without ever
> once sourcing the 'activate' script.

Just because you don't use activate doesn't mean everything is going
to 'just work' the way the OP is hoping for. Easy example: everything
installed in a VE will be ignored by apt-get upgrade, including the
python binary. For some, this is desirable, for others it isn't. I
was scolded by a sysadmin for this. I did talk him into letting me
slide given that we couldn't come up with anything better.

I seem to remember a post to this list within the last few months
where some other package was doing something (I think with sys.path)
that broke the VE sandbox.

"sudo easy_install dabo" works. "make/activate ve; pip install dabo"
does not. There is something funky about dabo's setup.py.

I am not sure how fair it is to blame VE for these last 2: to me they
are broken installers. But they show that VE isn't as rock solid as
the OP is hoping for.

--
Carl K

anatoly techtonik

unread,
Jun 27, 2011, 10:29:03 AM6/27/11
to python-v...@googlegroups.com
On Mon, Jun 27, 2011 at 4:24 PM, Tres Seaver <tres....@gmail.com> wrote:
> On 06/27/2011 06:26 AM, anatoly techtonik wrote:
>
>> It is not solely about startup and shutdown. There are quite many operations
>> that you need to do inside virtualenv periodically. Like upgrading,
>> installing or uninstalling packages (virtualenv specific Trac plugins in my
>> case), executing python modules and tools so that changes are persistent to
>> a given environment etc.
>
> I do those by spelling 'bin/python', 'bin/easy_install',
> 'bin/<app-provided-script>', etc. from within the envirohmnent.
>
>> Can you guarantee that every script and executable installed using
>> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
>> this virtualenv and won't resort to globally installed Python? It doesn't
>> seem realistic to me. Perhaps I don't know all the majyk.
>
> Yes, that feature is part of the design of virtualenv.  The hashbang
> line of every installed script is modified to use the virtualenv's python.

That means that scripts will never work on Windows and other platforms
that doesn't understand hashbangs.

> As I said, I make heavy use of multiple virtualenv's daily without ever
> once sourcing the 'activate' script.

I agree that it's convenient, but 'activated shell' is nevertheless
useful. With it I don't need to care to put 'bin/' prefix when it's
needed (or blame myself that I've forgot it). The command prompt
indication for virtualenv terminal is also very handy to locate the
necessary window when troubleshooting problems. The only problem with
it that standard Ctrl-D or 'logout' combination that you use to exit
from 'sudo su' and 'screen -x user/screen' or 'ssh remote' environment
doesn't work for virtualenv. On Windows you can't open virtualenv
console by merely launching activate.bat from run dialog - the window
closes immediately.
--
anatoly t.

Tres Seaver

unread,
Jun 27, 2011, 11:13:57 AM6/27/11
to python-v...@googlegroups.com
On 06/27/2011 10:28 AM, Carl Karsten wrote:
> On Mon, Jun 27, 2011 at 8:24 AM, Tres Seaver <tres....@gmail.com> wrote:
>> On 06/27/2011 06:26 AM, anatoly techtonik wrote:
>>
>>> It is not solely about startup and shutdown. There are quite many operations
>>> that you need to do inside virtualenv periodically. Like upgrading,
>>> installing or uninstalling packages (virtualenv specific Trac plugins in my
>>> case), executing python modules and tools so that changes are persistent to
>>> a given environment etc.
>>
>> I do those by spelling 'bin/python', 'bin/easy_install',
>> 'bin/<app-provided-script>', etc. from within the envirohmnent.
>>
>>> Can you guarantee that every script and executable installed using
>>> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
>>> this virtualenv and won't resort to globally installed Python? It doesn't
>>> seem realistic to me. Perhaps I don't know all the majyk.
>>
>> Yes, that feature is part of the design of virtualenv. The hashbang
>> line of every installed script is modified to use the virtualenv's python.
>
> Define "installed script" - If it doesn't cover everything, this is a
> land mine.
>
>>
>> As I said, I make heavy use of multiple virtualenv's daily without ever
>> once sourcing the 'activate' script.
>
> Just because you don't use activate doesn't mean everything is going
> to 'just work' the way the OP is hoping for. Easy example: everything
> installed in a VE will be ignored by apt-get upgrade, including the
> python binary.

That has nothing to do with using activate -- it is intrinsic to the
fact that virtualenv copies the executable.

> For some, this is desirable, for others it isn't. I
> was scolded by a sysadmin for this. I did talk him into letting me
> slide given that we couldn't come up with anything better.

Without a (relatively) large internal investment (building your own repo
/ PPA) 'apt-get' (or yum) is not a tool suited for managing custom
Python-based apps.

> I seem to remember a post to this list within the last few months
> where some other package was doing something (I think with sys.path)
> that broke the VE sandbox.
>
> "sudo easy_install dabo" works. "make/activate ve; pip install dabo"
> does not. There is something funky about dabo's setup.py.

The mere presence of 'sudo' in that first command line gives me the
willies: it says to me, "you're holding it wrong."


> I am not sure how fair it is to blame VE for these last 2: to me they
> are broken installers. But they show that VE isn't as rock solid as
> the OP is hoping for.

The setup.py for Dabo does do some questionable / unusual stuff:

- It imports from 'dabo.__version__'.

- It does a very complicated dance to compute 'package_data' and
'datafiles', instead of using 'include_package_data'.

- It doesn't declare any dependencies: I guess without looking that
the various UI toolkit and RDBMS packages are "optional" dependencies
of the framework.

Also, the released tarball was not made using 'setup.py sdist'.

Or there could be a bug in pip. OTOH, this "works" for me:

---------------------- %< -------------------------------
$ /opt/Python-2.6.5/bin/virtualenv --no-site-packages /tmp/dabotest
New python executable in /tmp/dabotest/bin/python
Installing setuptools............done.
$ /tmp/dabotest/bin/pip install dabo
Downloading/unpacking dabo
Downloading dabo-0.9.3-mac-nix.tar.gz (2.1Mb): 2.1Mb downloaded
In the tar file /tmp/pip-48tTro-unpack/dabo-0.9.3-mac-nix.tar.gz the
member dabo/LICENSE.TXT is invalid: 'NoneType' object has no attribute
'isreg'
Running setup.py egg_info for package dabo
Installing collected packages: dabo
Running setup.py install for dabo
Successfully installed dabo
Cleaning up...
---------------------- %< -------------------------------

Carl Karsten

unread,
Jun 27, 2011, 11:28:55 AM6/27/11
to python-v...@googlegroups.com

This thread is not "how to use VE" it is "does a sysadmin need to know
how to use VE?" and it seems the answer is yes, which is not ideal.

I will admit nothing is ideal, and VE reduces the total pain, but it
does result in more pain for the sysadmin who may not care how much
better it is for the developer. It might even mean less pain for the
admin if the alternative is for the admin to figure out how to do what
VE does. But I can understand the admin sayinig "no, use what is in
the stable package repo, and no, I am not going to point the web
server at a binary in your /home dir."

--
Carl K

Carl Meyer

unread,
Jun 27, 2011, 11:11:59 AM6/27/11
to python-v...@googlegroups.com
On 06/27/2011 09:29 AM, anatoly techtonik wrote:
>>> Can you guarantee that every script and executable installed using
>>> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
>>> this virtualenv and won't resort to globally installed Python? It doesn't
>>> seem realistic to me. Perhaps I don't know all the majyk.
>>
>> Yes, that feature is part of the design of virtualenv. The hashbang
>> line of every installed script is modified to use the virtualenv's python.
>
> That means that scripts will never work on Windows and other platforms
> that doesn't understand hashbangs.

Scripts must work on Windows somehow, because activate.bat (just like
activate.sh) does nothing effective except set your shell PATH to point
to the virtualenv's bin/ (or Scripts\) dir. That's all the magic there
is. So using "activate" is really exactly equivalent to running scripts
directly from the virtualenv's bin dir (except that, unfortunately, it
misleads people into thinking there's some other magic going on).

I have to admit that, given the lack of hashbang support, I have no idea
_how_ scripts in the Scripts/ dir of a Windows virtualenv ensure that
the virtualenv's Python binary is used. But clearly they do, or
virtualenv wouldn't work at all on Windows.

This also means that running scripts directly from the virtualenv's bin/
dir is, by definition, no less reliable than using activate. If a
script/binary gets installed into a virtualenv that's broken such that
it doesn't actually run within the virtualenv, it will be equally broken
under "activate".

Carl

Tres Seaver

unread,
Jun 27, 2011, 12:26:37 PM6/27/11
to python-v...@googlegroups.com

I don't develop on Windows, or use it, but I am confident that
setuptools distutils does the equivalent majyk to fix up generated
scripts on Windows, using the python interpreter used to run 'setup.py'.
E.g., from the setuptools docs[1]:

- Automatically generate wrapper scripts or Windows (console and GUI)
.exe files for any number of "main" functions in your project. (Note:
this is not a py2exe replacement; the .exe files rely on the local
Python installation.)

(This is for scripts / wrappers generated via the 'console_scripts' and
'gui_scripts' entry points).

I can't verify what happens on Windows with "old-fashined" (pure
distutils) scripts: by inspection of the code, it looks like the
distutils adjusts the hashbang line inside its 'build_scripts' commend
while copying the script into the build area, using the "target"
interpreter. virtualenv doesn't interfere with that at all, AFAICT.


[1] http://peak.telecommunity.com/DevCenter/setuptools

anatoly techtonik

unread,
Jun 27, 2011, 12:45:52 PM6/27/11
to python-v...@googlegroups.com
On Mon, Jun 27, 2011 at 6:11 PM, Carl Meyer <ca...@oddbird.net> wrote:
> On 06/27/2011 09:29 AM, anatoly techtonik wrote:
>>>> Can you guarantee that every script and executable installed using
>>>> executables from virtualenv bin/ directory (or Scripts/ on Windows) will use
>>>> this virtualenv and won't resort to globally installed Python? It doesn't
>>>> seem realistic to me. Perhaps I don't know all the majyk.
>>>
>>> Yes, that feature is part of the design of virtualenv.  The hashbang
>>> line of every installed script is modified to use the virtualenv's python.
>>
>> That means that scripts will never work on Windows and other platforms
>> that doesn't understand hashbangs.
>
> Scripts must work on Windows somehow, because activate.bat (just like
> activate.sh) does nothing effective except set your shell PATH to point
> to the virtualenv's bin/ (or Scripts\) dir. That's all the magic there
> is. So using "activate" is really exactly equivalent to running scripts
> directly from the virtualenv's bin dir (except that, unfortunately, it
> misleads people into thinking there's some other magic going on).
>
> I have to admit that, given the lack of hashbang support, I have no idea
> _how_ scripts in the Scripts/ dir of a Windows virtualenv ensure that
> the virtualenv's Python binary is used. But clearly they do, or
> virtualenv wouldn't work at all on Windows.

There is no magic - they don't work if executed directly unless you
have a correct file association in windows registry. But activate.bat
doesn't set this association, so the only way to use script with
interpreter supplied by virtualenv is to launch the script with
interpreter explicitly. Here you can either specify a full path to
interpreter or make sure that virtualenv dir is located before global
one in PATH.

> This also means that running scripts directly from the virtualenv's bin/
> dir is, by definition, no less reliable than using activate. If a
> script/binary gets installed into a virtualenv that's broken such that
> it doesn't actually run within the virtualenv, it will be equally broken
> under "activate".

Under Windows you can't run virtualenv scripts directly - you have to
invoke scripts like 'python script.py', because file associations are
global to the system, so if you change it - all other scripts will be
executed in wrong context. So, the only way to ensure that 'python
script.py' executes with a correct interpreter is to alter PATH. There
is no magic on Windows to do this based on file extension or its
hashbang line.

anatoly techtonik

unread,
Jun 27, 2011, 12:53:21 PM6/27/11
to python-v...@googlegroups.com

Let's avoid setuptools layer. There are still lot of packages that
(unlike Trac in my setup) don't use setuptools at all.

> I can't verify what happens on Windows with "old-fashined" (pure
> distutils) scripts:  by inspection of the code, it looks like the
> distutils adjusts the hashbang line inside its 'build_scripts' commend
> while copying the script into the build area, using the "target"
> interpreter.  virtualenv doesn't interfere with that at all, AFAICT.

It is made for Cygwin.
--
anatoly t.

Tres Seaver

unread,
Jun 27, 2011, 2:32:57 PM6/27/11
to python-v...@googlegroups.com
On 06/27/2011 12:53 PM, anatoly techtonik wrote:

>> I don't develop on Windows, or use it, but I am confident that
>> setuptools distutils does the equivalent majyk to fix up generated
>> scripts on Windows, using the python interpreter used to run 'setup.py'.
>> E.g., from the setuptools docs[1]:
>>
>> - Automatically generate wrapper scripts or Windows (console and GUI)
>> .exe files for any number of "main" functions in your project. (Note:
>> this is not a py2exe replacement; the .exe files rely on the local
>> Python installation.)
>>
>> (This is for scripts / wrappers generated via the 'console_scripts' and
>> 'gui_scripts' entry points).
>
> Let's avoid setuptools layer. There are still lot of packages that
> (unlike Trac in my setup) don't use setuptools at all.

This *is* the 'virtualenv' list: virtualenv itself uses / requires
setuptools. Projects which want to install their own scripts without
the entry point support were already broken under (non-Cygwin) Windows
with vanilla distutils, so there is nothing to discuss here about that.

>> I can't verify what happens on Windows with "old-fashined" (pure
>> distutils) scripts: by inspection of the code, it looks like the
>> distutils adjusts the hashbang line inside its 'build_scripts' commend
>> while copying the script into the build area, using the "target"
>> interpreter. virtualenv doesn't interfere with that at all, AFAICT.
>
> It is made for Cygwin.

Again, this is the distutils behvior, not anything specific to virtualenv.

anatoly techtonik

unread,
Jun 29, 2011, 1:59:54 PM6/29/11
to python-v...@googlegroups.com
On Mon, Jun 27, 2011 at 9:32 PM, Tres Seaver <tres....@gmail.com> wrote:
> On 06/27/2011 12:53 PM, anatoly techtonik wrote:
>
>>> I don't develop on Windows, or use it, but I am confident that
>>> setuptools distutils does the equivalent majyk to fix up generated
>>> scripts on Windows, using the python interpreter used to run 'setup.py'.
>>>  E.g., from the setuptools docs[1]:
>>>
>>>  - Automatically generate wrapper scripts or Windows (console and GUI)
>>>   .exe files for any number of "main" functions in your project. (Note:
>>>   this is not a py2exe replacement; the .exe files rely on the local
>>>   Python installation.)
>>>
>>> (This is for scripts / wrappers generated via the 'console_scripts' and
>>> 'gui_scripts' entry points).
>>
>> Let's avoid setuptools layer. There are still lot of packages that
>> (unlike Trac in my setup) don't use setuptools at all.
>
> This *is* the 'virtualenv' list:  virtualenv itself uses / requires
> setuptools.  Projects which want to install their own scripts without
> the entry point support were already broken under (non-Cygwin) Windows
> with vanilla distutils, so there is nothing to discuss here about that.

I don't understand. You say that projects without entry point support
were already broken on Windows. That's not true. If you say that they
are broken _inside_ virtualenv then it may be true. In this case
virtualenv is clearly broken on Windows.

>>> I can't verify what happens on Windows with "old-fashined" (pure
>>> distutils) scripts:  by inspection of the code, it looks like the
>>> distutils adjusts the hashbang line inside its 'build_scripts' commend
>>> while copying the script into the build area, using the "target"
>>> interpreter.  virtualenv doesn't interfere with that at all, AFAICT.
>>
>> It is made for Cygwin.
>
> Again, this is the distutils behvior, not anything specific to virtualenv.

This behavior results in a problem that on Windows scripts installed
into virtualenv are executed with default system interpreter and not
with the one from virtualenv.
--
anatoly t.

Reply all
Reply to author
Forward
0 new messages