Pip "dry run" mode - identifying required installs

[Paul Moore]

IPython 1.0 alpha has just been released, and it made me think about a scenario that I don't think we currently cover very well with pip.

The install instructions for IPython say basically

    pip install ipython[notebook]

However, a number of the dependencies include C extensions, and so need a compiler (and potentially a complex build process). As wheels are not currently available from PyPI, I need to make those wheels available manually, which I usually do by obtaining bdist_msi installers and doing "wheel convert" on them. However, there's no easy way of determining what I need to make wheels for, short of trying an install and spotting the failures in the raft of output. (Or hoping that IPython include the list of dependencies in the documentation somewhere).

We should probably have some sort of pip command that responds with a simple list of "this is what will be installed", so that the user can then make sure that those distributions are available.

Have I missed something that's already there? If not, does anyone have any ideas on how/if we could provide this? (Until we get Metadata 2.0 we may need to do a download of the sdist and something like setup.py egg_info to extract the metadata - I'm fine with that if it's all we've got at the moment).

Obviously IPython could provide a requirements file that specifies precisely what needs to be installed - or I could work one out form myself - but that's not the point, which is that the information is present in the dependency data and I think pip should provide a means to get it out...

Paul

[Jason R. Coombs]

One technique is to install to a clean environment on the target platform (with a compiler) and then pip freeze that env.

Sent from my comm

[Paul Moore]

On 22 July 2013 11:38, Jason R. Coombs <jar...@jaraco.com> wrote:

One technique is to install to a clean environment on the target platform (with a compiler) and then pip freeze that env.

The point is that the install will fail, because I can't build the binaries myself. And pip can't auto-download built ones, because the projects don't provide wheels. I have to manually make wheels available one by one, hence my need for a list of what's needed.

Paul

[Donald Stufft]

I know this is something that Chef and such would like.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Daniel Holth]

How about --no-install? It should just download everything and run egg_info...

[Paul Moore]

On 22 July 2013 14:46, Daniel Holth <dho...@gmail.com> wrote:

How about --no-install? It should just download everything and run egg_info...

Interesting. Yes, it does seem to do more or less what I need:

>pip install --no-install "ipython[notebook]"

Downloading/unpacking ipython[notebook]

  Using download cache from C:\Users\uk03306\pip\Cache\https%3A%2F%2Fpypi.python

%2Fipython%2Fipython-0.13.2.zip

  Running setup.py egg_info for package ipython

  Installing extra requirements: 'notebook'

Downloading/unpacking pyreadline>=1.7.1 (from ipython[notebook])

  Using download cache from C:\Users\uk03306\pip\Cache\https%3A%2F%2Fpypi.python

%2Fpyreadline%2Fpyreadline-2.0.zip

  Running setup.py egg_info for package pyreadline

    package init file 'pyreadline\configuration\__init__.py' not found (or not a

Downloading/unpacking tornado>=2.0 (from ipython[notebook])

  Using download cache from C:\Users\uk03306\pip\Cache\https%3A%2F%2Fpypi.python

%2Ftornado%2Ftornado-3.1.tar.gz

  Running setup.py egg_info for package tornado

    warning: no previously-included files matching '_auto2to3*' found anywhere i

Downloading/unpacking pyzmq>=2.1.11 (from ipython[notebook])

  Using download cache from C:\Users\uk03306\pip\Cache\https%3A%2F%2Fpypi.python

%2Fpyzmq%2Fpyzmq-13.1.0.zip

  Running setup.py egg_info for package pyzmq

    no previously-included directories found matching 'docs\build'

    no previously-included directories found matching 'docs\gh-pages'

    warning: no directories found matching 'bundled\uuid'

    warning: no previously-included files found matching 'bundled\uuid\Makefile*

    warning: no previously-included files found matching 'bundled\zeromq\src\Mak

    warning: no previously-included files found matching 'bundled\zeromq\src\pla

    warning: no previously-included files found matching 'setup.cfg'

    warning: no previously-included files found matching 'zmq\libzmq*'

    warning: no previously-included files matching '.deps\*' found anywhere in d

    warning: no previously-included files matching '*.so' found anywhere in dist

    warning: no previously-included files matching '*.pyd' found anywhere in dis

    warning: no previously-included files matching '*.pyc' found anywhere in dis

    warning: no previously-included files matching '.git*' found anywhere in dis

    warning: no previously-included files matching '.DS_Store' found anywhere in

    warning: no previously-included files matching '.mailmap' found anywhere in

Successfully downloaded ipython pyreadline tornado pyzmq

It's pretty verbose, and I'll need to run it in a temporary virtualenv (or make sure to clean up the build directory) and "pip install --no-install" is probably the least obvious command name in the universe :-) but that last line does look like exactly what I need..

So many thanks for that.

A properly supported command to do this might be worthwhile for other reasons, as Donald suggested, but I have enough for my needs.

Paul

[Marcus Smith]

A properly supported command to do this might be worthwhile for other reasons, as Donald suggested, but I have enough for my needs.

there's this refactor idea to better present some of pip's functionality

https://github.com/pypa/pip/issues/906

[Paul Moore]

Interesting. If we were offering a "list required dependencies" command, I'd expect it to do the equivalent of --no-install followed by deleting the build directories, and only listing the final line in the output. So it's a bit more than the proposed "pip unpack".

But it really only starts to make sense with metadata 2.0, when we can just get the metadata and not download the whole distribution and run the setup.py. So I'm happy with pip install --no-install plus some manual tidy-up till then.

Paul

[Marcus Smith]

On Mon, Jul 22, 2013 at 9:01 AM, Paul Moore <p.f....@gmail.com> wrote:

On 22 July 2013 16:50, Marcus Smith <qwc...@gmail.com> wrote:

A properly supported command to do this might be worthwhile for other reasons, as Donald suggested, but I have enough for my needs.

there's this refactor idea to better present some of pip's functionality

https://github.com/pypa/pip/issues/906

Interesting. If we were offering a "list required dependencies" command, I'd expect it to do the equivalent of --no-install followed by deleting the build directories, and only listing the final line in the output. So it's a bit more than the proposed "pip unpack".

yes, the intent of "pip unpack" is just to get rid of the absurdity of "pip install --no-install"