Installing from t.h.o mirror @ bitbucket WAS: [Trac] Re: t-h.o down, but community not so much

[Olemis Lang]

Hi !

The whole story ...

We have an issue now that t.h.o is down . The last few days Apache(TM)
Bloodhound 0.1.0-rc1 was released by the ASF . The fact is that it
contains an installer script . One of the steps consists of
pip-installing some plugins from (... guess what ...) t.h.o. svn repos
. Rules specified in requirements files [1]_ [2]_ .

On 8/14/12, Steffen Hoffmann <hof...@web.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Am 14.08.2012 21:10, schrieb Steffen Hoffmann:
>> Hello,
>>
>> I call for your response, if there is another repository clone out there
>> with a revision higher than my own - 11924.
>>
[...]
>>
>> We'll want to serve all of us as good as possible with such an
>> intermediate solution, at least until t-h.o is restored.
>
[...]
>
> Until someone else steps up with an even better plan, there it is for a
> start:
>
> https://bitbucket.org/hasienda/t-h.o_shadow
>

... so I'm wondering , is it possible to change requirements rules so
that pip will download plugins from t-h.o_shadow hg repository ?

I'm asking this in spite of providing a temporary solution in order to
work around existing issue [3]_ hoping that Bloodhound users will be
able to use installer script in the meantime , even with some
modifications . We will provide new requirements file if this is
possible , of course ;)

Thanks in advance !

.. [1] requirements-dev.txt
(http://svn.apache.org/viewvc/incubator/bloodhound/trunk/installer/requirements-dev.txt?revision=1371732&view=co)

.. [2] requirements.txt
(http://svn.apache.org/viewvc/incubator/bloodhound/trunk/installer/requirements.txt?revision=1371732&view=co)

.. [3] Install fails when trac-hacks is down
(https://issues.apache.org/bloodhound/ticket/161)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

[Ethan Jucovy]

Hi Olemis,

On Thu, Aug 16, 2012 at 1:30 PM, Olemis Lang <ole...@gmail.com> wrote:

>  https://bitbucket.org/hasienda/t-h.o_shadow
>

... so I'm wondering , is it possible to change requirements rules so that pip will download plugins from t-h.o_shadow hg repository ?

It's not clear what you're asking.  Of course you can edit your own requirements.txt file to refer to different URLs.  But this t-h.o_shadow repository is a single repository, and pip does not support installing from a subdirectory of a (non-SVN) repository: pip insists that the package you are installing (specifically the setup.py file) is located at the repository root.  This feature has been requested[1] but no one has implemented it in pip.

You should probably clone the t-h.o_shadow repository, copy the subdirectories with the plugins you need, create tar.gz files for those plugins yourself by running `python setup.py sdist` and host those tar.gz files on a server you control.  This has two advantages; in addition to making sure that your release doesn't depend on external circumstances, you will also be pinning your release to specific known versions of its dependencies.  Your release's requirements.txt file would then just point to the proper self-hosted tar.gz files for all of its dependencies, and your trunk's requirements-dev.txt file would point to the unpinned original codebases.  I've done this "cheaply" by [ab]using Github Pages to host my dependencies[2,3,4] -- but the tar.gz files could even be checked in to your repository tree under the release; pip supports installing from relative file paths.  I've taken this approach before[5]; it's by far the simplest way to ensure that your code ships with all its dependencies available.

If you're asking whether a pip requirements file supports "fallbacks" -- so that it tries a backup URL if the default one is not available -- the answer is "yes and no."  This is exactly what a Python Package Index is for: your requirements file would just specify the required name and version number for each dependency, and the installer (pip) would be responsible for finding an available package matching that spec, by searching on the package index(es) for an available download.  The Trac community doesn't always follow Python conventions here -- plugins are not always released on PyPI, and plugin version numbers often mirror their supported Trac version instead of being "independent" version numbers -- but this is not really an issue.  You can create your own Package Index easily, upload the necessary plugin releases there, and use pip's `find-links` option to point to it instead of (or in addition to) the canonical PyPI.  Then your requirements file could just specify "AccountManager==someversion" and pip would find an available copy satisfying the spec by searching through the package indexes you specify -- also in the requirements file[6].

Ultimately though this doesn't really add value beyond pointing directly to self-hosted copies of the releases; in either case the bulk of the work is the same (creating the copies of the releases, hosting them somewhere yourself, and making sure your release points to those copies of the code)

-Ethan

[1] https://groups.google.com/forum/#!msg/python-virtualenv/QpbUPWCPCs4/5XVYVpergesJ%5B1-25%5D 

[2] https://github.com/socialplanning/dist.socialplanning.org which is a gh-pages branch that includes requirements.txt files as well as a rarely-changing Git submodule...

[3] https://github.com/socialplanning/eggs containing lots of tar.gz files, which combine to generate...

[4] http://dist.socialplanning.org/ whose /eggs/ subpath can be used by ``find-links``, or individual code packages can be referred to directly by a requirements file

[5] http://ccnmtl.columbia.edu/compiled/sysadmin/deploying_django_and_deploying.html

[6] http://www.pip-installer.org/en/latest/requirements.html#indexes-find-links

[Sam Halliday]

another solution: why not do an `hg up` from the t-h.o snapshot and host it on some static webserver? That way all you should need to do is point your installer scripts at that location.