setuptools install problem

[Ralf Stephan]

This comes up frequently when running patchbot in SMC. Recently, in one run two tickets

built fine on top of 6.2.rc2, then suddenly three other failed due to this.

The tickets are unrelated, and at least one is known to pass in another run,

and it also builds fine on another system. It also happened with 6.2.rc1 as base.

Installed /scratch/sage/local/lib/python2.7/site-packages/setuptools-2.0.1-py2.

7.egg

Processing dependencies for setuptools==2.0.1

Traceback (most recent call last):

  File "setup.py", line 200, in <module>

    dist = setuptools.setup(**setup_params)

  File "/scratch/sage/local/lib/python2.7/distutils/core.py", line 152, in setu

p

    dist.run_commands()

  File "/scratch/sage/local/lib/python2.7/distutils/dist.py", line 953, in run_

commands

    self.run_command(cmd)

  File "/scratch/sage/local/lib/python2.7/distutils/dist.py", line 972, in run_

command

    cmd_obj.run()

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/install.py", line 73, in run

    self.do_egg_install()

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/install.py", line 101, in do_egg_install

    cmd.run()

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/easy_install.py", line 359, in run

    self.easy_install(spec, not self.no_deps)

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/easy_install.py", line 575, in easy_install

    return self.install_item(None, spec, tmpdir, deps, True)

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/easy_install.py", line 626, in install_item

    self.process_distribution(spec, dist, deps)

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/setuptools/command/easy_install.py", line 672, in process_distribut

ion

    [requirement], self.local_index, self.easy_install

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/pkg_resources.py", line 576, in resolve

    requirements.extend(dist.requires(req.extras)[::-1])

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2

.0.1.p0/src/pkg_resources.py", line 2224, in requires

    dm = self._dep_map

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 2209, in _dep_map

    for extra,reqs in split_sections(self._get_metadata(name)):

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 2650, in split_sections

    for line in yield_lines(s):

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 1910, in yield_lines

    for ss in strs:

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 2238, in _get_metadata

    for line in self.get_metadata_lines(name):

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 1300, in get_metadata_lines

    return yield_lines(self.get_metadata(name))

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 1292, in get_metadata

    return self._get(self._fn(self.egg_info,name))

  File "/tmp/tmpDH6Cyy-sage-git-temp-2516/local/var/tmp/sage/build/setuptools-2.0.1.p0/src/pkg_resources.py", line 1356, in _get

    return self.loader.get_data(path)

zipimport.ZipImportError: bad local file header in /scratch/sage/local/lib/python2.7/site-packages/setuptools-2.0.1-py2.7.egg

real    0m1.367s

user    0m0.670s

sys     0m0.179s

************************************************************************

Error installing package setuptools-2.0.1.p0

Regards,

[Ralf Stephan]

This is not SMC specific. Also the CentOS machine running patchbot has these build failures

occasionally.

http://patchbot.sagemath.org/log/8734/CentOS/6.5/x86_64/2.6.32-431.5.1.el6.x86_64/kepler.math.osu.edu/2014-04-30%2014:37:38%20-0400

[Ralf Stephan]

http://trac.sagemath.org/ticket/16300

People running patchbot with errors on setuptools install please try

version 2.2b1 of setuptools and report if this changes the behaviour

or not.

Regards,

[Peter Bruin]

To try this one has to apply #16300 and then run patchbot with the resulting branch as the base branch, is that correct?


Op woensdag 7 mei 2014 10:05:32 UTC+1 schreef Ralf Stephan:

[Ralf Stephan]

Just change builds/pkg.../package-version, download to upstream, run sage to fix checksums, maybe also sage -f setuptools, and then use that as base. I had no problems in two runs after that but more data is needed.

This goes back as far as the first commit after 6.1.1

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/7acENsAQeXo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Peter Bruin]

OK, I'll try that after the patchbot finishes the current ticket (which seems to be unaffected by the setuptools problem).


Op woensdag 7 mei 2014 11:27:19 UTC+1 schreef Ralf Stephan:

[Ralf Stephan]

Oh yes you're right. Making it the base branch is necessary too.

[Ralf Stephan]

Oh well, so base_branch means remote base branch on github.

So, let's push that ticket.

[Peter Bruin]

Oh yes you're right. Making it the base branch is necessary too.

Something that worked for me is making the changes without committing them; then the patchbot won't undo those changes.

[Ralf Stephan]

On Wed, May 7, 2014 at 5:08 PM, Peter Bruin <pjb...@gmail.com> wrote:

Oh yes you're right. Making it the base branch is necessary too.

Something that worked for me is making the changes without committing them; then the patchbot won't undo those changes.

​That will work until a ticket with "unsafe" changes​ is encountered, with full rebuild in /tmp.

Unfortunately, src/doc is deemed unsafe too by current patchbot:

​https://github.com/robertwb/sage-patchbot/issues/22​

[leif]

Well, just create an upstream tarball of setuptools 2.2.x with the same
md5sum as 2.0.1.p0 (and name it setuptools-2.0.1.tar.{bz2,gz}), or in
case that takes too long for you, create a wrapper script for 'md' /
'md5sum' which returns the "correct" md5sum for the setuptools upstream
tarball.


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Peter Bruin]

OK, I see, this has now happened to my patchbot on #15316.


Op woensdag 7 mei 2014 17:18:17 UTC+1 schreef Ralf Stephan:

[Ralf Stephan]

https://github.com/robertwb/sage-patchbot/pull/24

and if you change patchbot/trac.py:pull_from_trac() in line 309

such that after

        if not inplace_safe():

an exception is raised (e.g. just "raise") then patchbot will only

checkout and test tickets in-place, i.e. no reinstall. Unsafe tickets

will just have to wait until the upgrade of setuptools is official.

[Peter Bruin]

I started a patchbot based on 6.3.beta1, and the problem has unfortunately popped up again with the new setuptools version:

http://patchbot.sagemath.org/log/16348/debian/wheezy/sid/x86_64/3.2.0-57-generic/selmer/2014-05-13%2019:57:47%20+0100

Op woensdag 7 mei 2014 10:05:32 UTC+1 schreef Ralf Stephan:

http://trac.sagemath.org/ticket/16300

[Ralf Stephan]

http://trac.sagemath.org/ticket/16350

Please review.

Regards,

[Ralf Stephan]

It may be possible to test newer package versions in patchbot without

official ticket by forking Sage on github, installing the package in the local

Sage, publish the changed branch on github, and use that branch as

base_repo with patchbot. This may work also with unsafe tickets because

the cloning into /tmp copies upstream/ too. I'm testing that soon.

If this doesn't work then my fork of patchbot soon will have a version

with a --keep-tmp option that doesn't delete failed installs in /tmp so

that the setuptools problem can be postmortemed. I have stopped

making pull requests at robertb/patchbot.

https://github.com/rwst/patchbot

Regards,

On Tuesday, May 13, 2014 9:51:17 PM UTC+2, Peter Bruin wrote:

[Ralf Stephan]

Ah I finally found the relevant upstream ticket:

https://bitbucket.org/pypa/setuptools/issue/168/ez_setuppy-failing-sporadically-when

which appears resolved with setuptools-3.5.2

Next is testing of a version >= 3.5.2

Regards,

[Peter Bruin]

It would be great if that worked.  I disabled the tmp directory for the moment and am testing all tickets in the same directory, but that can leave the tree in a bad state.

Peter


Op vrijdag 16 mei 2014 10:03:01 UTC+1 schreef Ralf Stephan:

[Ralf Stephan]

It is possible that the /tmp installation is broken anyway. With setuptools-3.6

I get 4 out of 4 failures because some paths point back to the original sage/

directory, but some not and the underlying files are not there (numpy for example).

Moreover, none of the 6.2 TestsPassed runs (n=47) is an unsafe ticket and

thus a /tmp install.

The snapshot method proposed in another thread gets increasingly

attractive to me.

[Robert Bradshaw]

On Thu, May 15, 2014 at 12:53 AM, Ralf Stephan <gtr...@gmail.com> wrote:
> It may be possible to test newer package versions in patchbot without
> official ticket by forking Sage on github, installing the package in the
> local
> Sage, publish the changed branch on github, and use that branch as
> base_repo with patchbot. This may work also with unsafe tickets because
> the cloning into /tmp copies upstream/ too. I'm testing that soon.

Yes, that should work fine. Perhaps we should have a standard patchbot
branch that is usually master but also contains the rare fix like
this.

> If this doesn't work then my fork of patchbot soon will have a version
> with a --keep-tmp option that doesn't delete failed installs in /tmp so
> that the setuptools problem can be postmortemed. I have stopped
> making pull requests at robertb/patchbot.
>
> https://github.com/rwst/patchbot

I've been out of the country with extremely limited internet access
the last two weeks; please keep the pull requests coming and I will
get to them (along with all of my other backlog).

> On Tuesday, May 13, 2014 9:51:17 PM UTC+2, Peter Bruin wrote:
>>
>> I started a patchbot based on 6.3.beta1, and the problem has unfortunately
>> popped up again with the new setuptools version:
>>
>>
>> http://patchbot.sagemath.org/log/16348/debian/wheezy/sid/x86_64/3.2.0-57-generic/selmer/2014-05-13%2019:57:47%20+0100
>>
>>
>> Op woensdag 7 mei 2014 10:05:32 UTC+1 schreef Ralf Stephan:
>>>
>>> http://trac.sagemath.org/ticket/16300
>>>
>>> People running patchbot with errors on setuptools install please try
>>> version 2.2b1 of setuptools and report if this changes the behaviour
>>> or not.
>>>
>>> Regards,
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Ralf Stephan]

On Saturday, May 17, 2014 11:42:43 AM UTC+2, Ralf Stephan wrote:

It is possible that the /tmp installation is broken anyway. With setuptools-3.6

I get 4 out of 4 failures because some paths point back to the original sage/

directory, but some not and the underlying files are not there (numpy for example).

https://github.com/robertwb/sage-patchbot/issues/35

[Ralf Stephan]

On Saturday, May 17, 2014 11:42:43 AM UTC+2, Ralf Stephan wrote:

It is possible that the /tmp installation is broken anyway. With setuptools-3.6

I get 4 out of 4 failures because some paths point back to the original sage/

directory, but some not and the underlying files are not there (numpy for example).

I think I can now safely say this problem will be gone when both

http://trac.sagemath.org/ticket/16390

http://trac.sagemath.org/ticket/16413
are applied.

Please review.