How do I update a virtualenv?

[Skip Montanaro]

I have a virtualenv I'm using for some Django development. Today I
switched from MacPorts to HomeBrew on my Mac. I'm thus getting a
different version of gcc and its libs. How do I reinstall the
virtualenv? I've looked around and found a few descriptions of what to
do, but as I am little more than a monkey-see-monkey-do user of
virtualenv, I don't understand what appears to be a fairly complex set
of steps. Does virtualenv not have some sort of "upgrade" command that
just does what's necessary?

Thanks,

Skip

[Ned Batchelder]

Virtualenvs aren't built to be moved from one Python installation to
another. If you used pip to install your packages (you should), then
you can activate the virtualenv, and run: $ pip freeze > requirements.txt

Then you can create a new virtualenv using the new Python executable,
activate it, and: $ pip install -r requirements.txt

This will reinstall all the packages you had installed previously. Even
better is to maintain your own requirements.txt that has just the
packages you need. The "pip freeze" technique will also list packages
installed as dependencies.

--Ned.

[Ned Deily]

In article
<CANc-5Uy367Mu-zN30Z8xKfE_...@mail.gmail.com>,

Skip Montanaro <sk...@pobox.com> wrote:
> I have a virtualenv I'm using for some Django development. Today I
> switched from MacPorts to HomeBrew on my Mac.

Any particular reason for doing that? That seems like a step backwards,
especially for Python-related work.

--
Ned Deily,
n...@acm.org

[Skip Montanaro]

> Virtualenvs aren't built to be moved from one Python installation to
> another. If you used pip to install your packages (you should), then you
> can activate the virtualenv, and run: $ pip freeze > requirements.txt
>
> Then you can create a new virtualenv using the new Python executable,
> activate it, and: $ pip install -r requirements.txt
>
> This will reinstall all the packages you had installed previously. Even
> better is to maintain your own requirements.txt that has just the packages
> you need. The "pip freeze" technique will also list packages installed as
> dependencies.

Hmmm... And my git repo? I imagine I will eventually figure this out,
but updating an existing virtualenv in place to adapt to a new version
of Python (say, a new micro) or some of its libraries (contents of
requirements.txt) seems like it would be a very nice thing to have.

Skip

[Ned Batchelder]

On 10/28/13 7:53 PM, Skip Montanaro wrote:
>> Virtualenvs aren't built to be moved from one Python installation to
>> another. If you used pip to install your packages (you should), then you
>> can activate the virtualenv, and run: $ pip freeze > requirements.txt
>>
>> Then you can create a new virtualenv using the new Python executable,
>> activate it, and: $ pip install -r requirements.txt
>>
>> This will reinstall all the packages you had installed previously. Even
>> better is to maintain your own requirements.txt that has just the packages
>> you need. The "pip freeze" technique will also list packages installed as
>> dependencies.
> Hmmm... And my git repo?

Usually the virtualenv is outside the git repo (and vice-versa), but git
repos are also easy to recreate from the git server if you need to.
Maybe I don't understand what you mean?

> I imagine I will eventually figure this out,
> but updating an existing virtualenv in place to adapt to a new version
> of Python (say, a new micro) or some of its libraries (contents of
> requirements.txt) seems like it would be a very nice thing to have.

"pip install --upgrade" will upgrade your Python packages. "pip install
-r requirements.txt" will install new packages or versions named in the
requirements.txt file.

> Skip

--Ned.

[Skip Montanaro]

>> I have a virtualenv I'm using for some Django development. Today I
>> switched from MacPorts to HomeBrew on my Mac.
>
> Any particular reason for doing that? That seems like a step backwards,
> especially for Python-related work.

The guy who sits next to me at work recommended HomeBrew. (Someone to
ask for help. :-) I didn't have all that much stuff installed via
MacPorts, buncha libraries mostly, so it didn't seem like a huge
sacrifice. Also, when I'd tried installing gcc 4.8 a few weeks ago
things didn't go so well, so I figured I'd try HomeBrew.

Skip

[Ben Finney]

Skip Montanaro <sk...@pobox.com> writes:

> Hmmm... And my git repo?

You are keeping your virtualenv separate from your working tree, right?
I put mine in ‘/var/local/devel/$USER/virtualenvs/foobar/’ where
“foobar” is the specific virtualenv name.

This allows an arbitrary number of virtualenvs separate for each user
(‘/var/local/devel/$USER/’ is owned by each $USER), but all under one
directory tree where they can be treated specially for backup, cleanup,
permissions, or whatever.

> I imagine I will eventually figure this out, but updating an existing
> virtualenv in place to adapt to a new version of Python (say, a new
> micro) or some of its libraries (contents of requirements.txt) seems
> like it would be a very nice thing to have.

Yes, you simply ‘pip install -r ./requirements.txt’ again. It does a
simplistic “is the package already installed at this version in the
currently-active virtualenv? If not, install it in that virtualenv”.

The environment variables that determine where the virtualenv lives are
managed by ‘/var/local/devel/$USER/virtualenvs/foobar/bin/activate’. You
don't have the packages cluttering up your working tree.

--
\ “I don't know anything about music. In my line you don't have |
`\ to.” —Elvis Aaron Presley (1935–1977) |
_o__) |
Ben Finney

[Roy Smith]

In article <mailman.1731.1383006...@python.org>,

Ned Batchelder <n...@nedbatchelder.com> wrote:

> "pip install --upgrade" will upgrade your Python packages. "pip install
> -r requirements.txt" will install new packages or versions named in the
> requirements.txt file.

Yup, that's what we do. We've got a requirements.txt file that's
checked into our source repo. Part of our deployment process is to run
"pip install -r requirements.txt".

You want to start doing this from day one on any new project. One of
the problems we had was we started not even using virtualenv. When we
switched to do that, we needed to come up with a list of what should be
in it. We started with "pip freeze", and got a list of all the crap we
currently had installed. We then needed to start sorting through it to
figure out what we really needed and what was just excess baggage. It's
a lot cleaner if you start out doing it right at the beginning.

We build a fresh virtualenv on every code deployment. This makes sure
we know exactly what's in the virtualenv all the time. It's a little
extra work, but worth it.

We're looking at https://pypi.python.org/pypi/wheel to reduce deployment
time.

[Skip Montanaro]

>> Hmmm... And my git repo?
>

> Usually the virtualenv is outside the git repo (and vice-versa), but git
> repos are also easy to recreate from the git server if you need to. Maybe I
> don't understand what you mean?

I'm using Heroku, following their instructions. They have a "git init"
in the midst of things, so I wind up with a git repo that matches up
one-to-one for my Django project. ("git push" installs).

Part of the problem here is, of course, that I have done essentially
no web work since the days of editing HTML directly. I just have to
hunt around for something I can sort of understand and that has some
recommendations from someone I trust (in this case, some of the
denizens of chi...@python.org), and head off in that direction. I'm
sure the combinations of packaging systems, web application platforms,
and distribution tools are nearly limitless. I don't enough time left
to investigate and try out every combination, so for now, it's
HomeBrew+Django+Heroku. The decision to switch from MacPorts to
HomeBrew may well turn out to be problematic, but it is what it is.

>> I imagine I will eventually figure this out,
>> but updating an existing virtualenv in place to adapt to a new version
>> of Python (say, a new micro) or some of its libraries (contents of
>> requirements.txt) seems like it would be a very nice thing to have.
>

> "pip install --upgrade" will upgrade your Python packages. "pip install -r
> requirements.txt" will install new packages or versions named in the
> requirements.txt file.

Thanks. That's done.

So, in addition to the various virtualenv bits I needed to

* move .git and .gitignore from the old directory
* recreate the heroku stuff
* set up the remote heroku association again
* push everything
* sync the database

None of these things have anything to do with virtualenv, but they all
were things *in* the virtualenv. There are clearly various dot files
left laying around in my old virtualenv. I still think it would have
been easier overall if I was able to refresh the virtualenv in place.
It all boils down to a virtualenv only being one piece of a larger
puzzle. I'm not sure its constraints should drive the entire process.

Skip

[Tim Chase]

On 2013-10-29 11:42, Ben Finney wrote:
> You are keeping your virtualenv separate from your working tree,
> right?

This was one of the key bits I missed in most of the
virtualenv{,wrapper} documentation and only figured out after asking
several questions here on c.l.p Once I had that understanding, the
usefulness of virtualenvwrapper made more sense (previously, I'd
figured it was silly since I thought the code-base was supposed to be
contained in the virtualenv, not independent from it).

Now it's virtualenv{,wrapper} for just about all development.

-tkc

[Skip Montanaro]

>> Hmmm... And my git repo?
>

> You are keeping your virtualenv separate from your working tree, right?

> I put mine in ‘/var/local/devel/$USER/virtualenvs/foobar/’ where
> “foobar” is the specific virtualenv name.

No, I'm no expert in these things. I was just following the directions
on the Heroku "getting started" page. What you wrote is Greek to me.

Skip

[Ben Finney]

Skip Montanaro <sk...@pobox.com> writes:

> I'm using Heroku, following their instructions. They have a "git init"
> in the midst of things, so I wind up with a git repo that matches up
> one-to-one for my Django project. ("git push" installs).

That's not a good thing; you don't want loads of third-party files in
your VCS repository.

Instead, the deployment to the live system should be done from a wholly
separate build area, and the virtualenv can serve as that build area.
Then, your VCS need only contain the files you'll actually be changing
as source code.

Where specifically are these instructions that tell you to put the
virtualenv under VCS control? As you are a Heroku customer (I'm not),
would you be willing to suggest they alter them based on advice from
this forum?

--
\ “Pity the meek, for they shall inherit the earth.” —Donald |
`\ Robert Perry Marquis |
_o__) |
Ben Finney

[Ben Finney]

Fair enough; virtualenv is a baroque system with many knobs to tune. It
sucks that anything like this is needed.

I don't know of better official documentation for virtualenv than
<URL:https://pypi.python.org/pypi/virtualenv>.

--
\ “It is forbidden to steal hotel towels. Please if you are not |
`\ person to do such is please not to read notice.” —hotel, |
_o__) Kowloon, Hong Kong |
Ben Finney

[Skip Montanaro]

> Where specifically are these instructions that tell you to put the
> virtualenv under VCS control?

https://devcenter.heroku.com/articles/getting-started-with-python

> As you are a Heroku customer (I'm not), would you be willing to
> suggest they alter them based on advice from this forum?

It's not my job to pretend to be an expert about something I am
clearly not. All I would be doing is passing along your perspective on
the matter. I presume they have reasonable reasons for doing things
the way they do it. It would be better for people expert in VCS issues
to discuss this stuff with the Heroku people, who presumably know a
thing or two about application deployment, and can give you their
perspective on why they use "git push" as a deployment tool.

Skip

[alex23]

On 30/10/2013 12:08 AM, Skip Montanaro wrote:
>> Where specifically are these instructions that tell you to put the
>> virtualenv under VCS control?
>
> https://devcenter.heroku.com/articles/getting-started-with-python

I believe you may have misread the instructions slightly. You should
have a project structure like this:

my_project/
/venv
.gitignore

The instructions mention adding 'venv' to your .gitignore, so it will be
excluded from version control. If you have .git & .gitignore files
inside your venv folder, then you've changed directory into it when you
should be calling it from the project root instead.

[Skip Montanaro]

> I believe you may have misread the instructions slightly. You should have a
> project structure like this:
>
> my_project/
> /venv
> .gitignore
>
> The instructions mention adding 'venv' to your .gitignore, so it will be
> excluded from version control. If you have .git & .gitignore files inside
> your venv folder, then you've changed directory into it when you should be
> calling it from the project root instead.

I have .git and .gitignore at the top level, and venv is in
.gitignore. I did, indeed, misunderstand things though. That's the
problem with monkey-see-monkey-do. I completely glossed over the part
where venv was excluded from my git repo. That being the case then, to
update my virtualenv, I need to first create a new one in a fresh
directory (using my existing requirements.txt file). I should be able
to simply swap out the old venv directory for the new one.

Skip