recompilation with "sage -clone"

[vdelecroix]

Hi,

It seems that the behavior of "sage -clone" changes around the version 5.9. I freshly compiled a version of 5.10.beta2 and when I do a clone it recompiles the wholse sage sources (but hopefully not the auxilliary spkg). The first line in the log of "sage -clone" which differs from previous version is the following

Updating Cython code....
Compiling sage/algebras/quatalg/quaternion_algebra_element.pyx because it depends on ./sage/structure/sage_object.pxd
...

I was not able to find some information elsewhere.

Best,
Vincent

[Eric Gourgoulhon]

I confirm this behavior on a fresh install of version 5.9 from the sources: sage -clone triggers the recompilation of the Cython sources. In version 5.8, it did not. Don't know the reason for this...

Eric.

[leif]

Eric Gourgoulhon wrote:
> I confirm this behavior on a fresh install of version 5.9 from the
> sources: sage -clone triggers the recompilation of the Cython sources.
> In version 5.8, it did not. Don't know the reason for this...

FWIW, it did still work in 5.9.beta2.

Probably #13031 [1] (merged into 5.9.beta4), or one of its follow-ups(?)
is to blame...


-leif

[1] http://trac.sagemath.org/sage_trac/ticket/13031


> Le vendredi 10 mai 2013 14:53:27 UTC+2, vdelecroix a �crit :

>
> Hi,
>
> It seems that the behavior of "sage -clone" changes around the
> version 5.9. I freshly compiled a version of 5.10.beta2 and when I
> do a clone it recompiles the wholse sage sources (but hopefully not
> the auxilliary spkg). The first line in the log of "sage -clone"
> which differs from previous version is the following
>
> Updating Cython code....
> Compiling sage/algebras/quatalg/quaternion_algebra_element.pyx
> because it depends on ./sage/structure/sage_object.pxd
> ...
>
> I was not able to find some information elsewhere.
>
> Best,
> Vincent

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[leif]

leif wrote:
> Eric Gourgoulhon wrote:
>> I confirm this behavior on a fresh install of version 5.9 from the
>> sources: sage -clone triggers the recompilation of the Cython sources.
>> In version 5.8, it did not. Don't know the reason for this...
>
> FWIW, it did still work in 5.9.beta2.
>
> Probably #13031 [1] (merged into 5.9.beta4), or one of its follow-ups(?)
> is to blame...

For the impatient, the following at least avoids rebuilding of (most of
the) Cython-generated files:

diff --git a/sage-clone b/sage-clone
--- a/sage-clone
+++ b/sage-clone
@@ -53,6 +53,10 @@

cpdir(os.path.abspath('sage/sage'), os.path.abspath(branch + '/sage'))

+if os.path.isfile('sage/.cython_version'):
+ print "Copying over hidden Cython version file..."
+ os.link('sage/.cython_version', branch+'/.cython_version')
+
def copy_dtree(src_dir, dest_dir):
src_root = os.path.abspath(src_dir)
dest_root = os.path.abspath(dest_dir)


(To be applied to the Sage scripts repo, in $SAGE_ROOT/local/bin/.)


It seems also all(?) Python files get re-(byte-)compiled; don't know yet
why... (but the latter doesn't take as long as re-"cythonizing"* does)


-leif

______________
* What an ill-chosen name!

[leif]

leif wrote:
> leif wrote:
>> Eric Gourgoulhon wrote:
>>> I confirm this behavior on a fresh install of version 5.9 from the
>>> sources: sage -clone triggers the recompilation of the Cython sources.
>>> In version 5.8, it did not. Don't know the reason for this...
>

> For the impatient, the following at least avoids rebuilding of (most of
> the) Cython-generated files:
>
> diff --git a/sage-clone b/sage-clone
> --- a/sage-clone
> +++ b/sage-clone
> @@ -53,6 +53,10 @@
>
> cpdir(os.path.abspath('sage/sage'), os.path.abspath(branch + '/sage'))
>
> +if os.path.isfile('sage/.cython_version'):
> + print "Copying over hidden Cython version file..."
> + os.link('sage/.cython_version', branch+'/.cython_version')
> +
> def copy_dtree(src_dir, dest_dir):
> src_root = os.path.abspath(src_dir)
> dest_root = os.path.abspath(dest_dir)

It seems the interpreters for "fast_callable" get regenerated (and hence
also re-"cythonized") because sage-clone does not (or no longer)
properly copy devel/sage/sage/ext/interpreters/timestamp either:

Building interpreters for fast_callable
Updating Cython code....
Building sage/plot/plot3d/parametric_surface.pyx because it depends on
sage/ext/interpreters/wrapper_rdf.pxd.
Building modified file sage/ext/interpreters/wrapper_rdf.pyx.
Building modified file sage/ext/interpreters/wrapper_cdf.pyx.
Building modified file sage/ext/interpreters/wrapper_rr.pyx.
Building modified file sage/ext/interpreters/wrapper_py.pyx.
Building modified file sage/ext/interpreters/wrapper_el.pyx.
...

(This is what I meant by "avoids rebuilding of /most of/ the
Cython-generated code".)


-leif

[vdelecroix]

>> For the impatient, the following at least avoids rebuilding of (most of
>> the) Cython-generated files:

Thanks leif, it works quite well on sage-5.9. Do you have an idea for
avoiding documentation regeneration (see [1]) ?

Best,
Vincent

[1] http://groups.google.com/group/sage-devel/browse_thread/thread/3c0a371b78568d90

[leif]

Eric Gourgoulhon wrote:
> I confirm this behavior on a fresh install of version 5.9 from the
> sources: sage -clone triggers the recompilation of the Cython sources.
> In version 5.8, it did not. Don't know the reason for this...

Looks like reinstalling the Sage library spkg would meanwhile also
trigger the rebuild of *all* extension modules; not sure since when
though... (seen with Sage 5.10.beta5 at least)


-leif

> Le vendredi 10 mai 2013 14:53:27 UTC+2, vdelecroix a �crit :

>
> Hi,
>
> It seems that the behavior of "sage -clone" changes around the
> version 5.9. I freshly compiled a version of 5.10.beta2 and when I
> do a clone it recompiles the wholse sage sources (but hopefully not
> the auxilliary spkg). The first line in the log of "sage -clone"
> which differs from previous version is the following
>
> Updating Cython code....
> Compiling sage/algebras/quatalg/quaternion_algebra_element.pyx
> because it depends on ./sage/structure/sage_object.pxd
> ...
>
> I was not able to find some information elsewhere.
>
> Best,
> Vincent

[leif]

leif wrote:
> Looks like reinstalling the Sage library spkg would meanwhile also
> trigger the rebuild of *all* extension modules; not sure since when
> though... (seen with Sage 5.10.beta5 at least)

Yep. This is because #14570, merged into Sage 5.10.beta4, completely
breaks sage-sync-build, which is also used in the Sage library's
spkg-install. (sage -sync-build now removes all Cython-generated files,
hence *all* have to get rebuilt afterwards.)


IMHO both issues (complete rebuild upon cloning and after
sage-sync-build) should be resolved before the final 5.10 gets out.


-leif

[Simon King]

On 2013-05-31, leif <not.r...@online.de> wrote:
> IMHO both issues (complete rebuild upon cloning and after
> sage-sync-build) should be resolved before the final 5.10 gets out.

+1

It should be a blocker, IMHO.

Cheers, Simon

[John H Palmieri]

On Thursday, May 30, 2013 9:57:07 PM UTC-7, leif wrote:

leif wrote:
> Looks like reinstalling the Sage library spkg would meanwhile also
> trigger the rebuild of *all* extension modules; not sure since when
> though...  (seen with Sage 5.10.beta5 at least)

Yep.  This is because #14570, merged into Sage 5.10.beta4, completely
breaks sage-sync-build, which is also used in the Sage library's
spkg-install.  (sage -sync-build now removes all Cython-generated files,
hence *all* have to get rebuilt afterwards.)


IMHO both issues (complete rebuild upon cloning and after
sage-sync-build) should be resolved before the final 5.10 gets out.

The rebuild is dealt with at #13245 (needs partial review). Is there a ticket for the sage-sync-build problem?

--
John

[Travis Scrimshaw]

Hey,

   I noticed that after running sync-build that it was always taking a very long time. I'm moving through a couple of patches which create new cython files and having to watch sage rebuild because I need to run sync-build slows me down significantly. This definitely should be a blocker.

Best,

Travis

[leif]

I'll revisit #13245 now that you've changed the ticket's title... ;-)

[Avoiding also the rebuild of the fast_callable interpreters should be
feasable by just copying a few more files in sage-clone (I /think/ I
already did that), but I agree that's a minor issue, at least compared
to rebuilding the whole Sage library.]

> Is there a ticket for the sage-sync-build problem?

Not yet. There are a couple of ways to (quickly or temporarily) "fix" this:

1) Unmerge #14570, which doesn't fix a bug, but only introduces a new
(btw. non-configurable) feature. [A bit odd, especially since #14570
also patches the Cython spkg.]

2) Revert just #14570's change to setup.py, a one-liner (removing the
build_dir parameter to cythonize()). [Better than 1), although
effectively the same -- feature vanishes until we reenable it on some
follow-up ticket.]

3) Make the behaviour configurable (defaulting to the previous; little
change to setup.py). No breakage to ordinary users; people using the
new feature are expected to know what they're doing, i.e., at least
sage-sync-build [still] won't work for them. [IMHO preferable for a
temporary / quick fix; easy and safe.]

4) Minimally adapt sage-sync-build.py to reflect the (hard-coded) change
by #14570. [May be easy as well, but potentially unsafe.]


I'd personally prefer 2) or 3) for now, postponing a "proper" solution
to a follow-up ticket which presumably won't make it into 5.10 (but
hopefully 5.11.*), as sage-sync-build needs some rework* anyway, the
Cython build directory (kind of VPATH) should IMHO be configurable, and
#14570 might cause further issues we haven't noticed yet.


-leif

__________
* the usual "can of worms"

[leif]

leif wrote:

> John H Palmieri wrote:
>> Is there a ticket for the sage-sync-build problem?
>
> Not yet. There are a couple of ways to (quickly or temporarily) "fix"
> this:
>
> 1) Unmerge #14570, which doesn't fix a bug, but only introduces a new
> (btw. non-configurable) feature. [A bit odd, especially since #14570
> also patches the Cython spkg.]
>
> 2) Revert just #14570's change to setup.py, a one-liner (removing the
> build_dir parameter to cythonize()). [Better than 1), although
> effectively the same -- feature vanishes until we reenable it on some
> follow-up ticket.]

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -532,7 +532,7 @@
ext_modules = cythonize(
ext_modules,
nthreads = int(os.environ.get('SAGE_NUM_THREADS', 0)),
- build_dir = 'build/cythonized',
+ # build_dir = 'build/cythonized',
force=force)

open(version_file, 'w').write(Cython.__version__)


works for me (with Sage 5.10.rc0).

> 3) Make the behaviour configurable (defaulting to the previous; little
> change to setup.py). No breakage to ordinary users; people using the
> new feature are expected to know what they're doing, i.e., at least
> sage-sync-build [still] won't work for them. [IMHO preferable for a
> temporary / quick fix; easy and safe.]

Similar to the above change, but with (say)

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -532,7 +532,7 @@
ext_modules = cythonize(
ext_modules,
nthreads = int(os.environ.get('SAGE_NUM_THREADS', 0)),
- build_dir = 'build/cythonized',
+ build_dir = 'build/cythonized' if
os.environ.get("BREAK_SAGE_SYNC_BUILD", "no")=="yes" else None,
force=force)

open(version_file, 'w').write(Cython.__version__)


which *optionally* enables building extension modules "out of tree"
(they're in fact in the same tree, but one level deeper, still
hard-coded location) at the expense of breaking (at least)
sage-sync-build. (Works for me, too.)


-leif

[leif]

Even that turns out to be non-trivial, AFAICS, i.e., the whole strategy
would have to get changed, as opposed to just some file / directory
names in a few places.


-leif

[Robert Bradshaw]

+1 to (2) or (3), along with the cython_version file fix, until we get
build_dir working fully.

- Robert

[leif]

leif wrote:

>> leif wrote:
>>> 4) Minimally adapt sage-sync-build.py to reflect the (hard-coded) change
>>> by #14570. [May be easy as well, but potentially unsafe.]
>
> Even that turns out to be non-trivial, AFAICS, i.e., the whole strategy
> would have to get changed, as opposed to just some file / directory
> names in a few places.

Just for the record:

I've meanwhile managed to hack sage-sync-build.py such that it
/seems/^TM to work with a Cython build_dir (below $SAGE_SRC/build/) as
well, but the patch would need clean-up and thorough testing.

And as mentioned, the script needs further, not directly related, fixes
anyway, so I'd prefer to postpone changes to sage-sync-build to 5.11.


-leif

[Travis Scrimshaw]

Hey,

   Could we at least get the hacked sage-sync-build into 5.10? The combinat queue has a few patch which create new cython files that replace python files (i.e. cythonizes them) which requires the sync-build to be run when people are moving ahead of those patches. Rebuilding all cython files in sage seems like a high price to me for something we can put a quick patch (no pun intended) over.

Thanks,

Travis

[leif]

leif wrote:
> leif wrote:
>> Looks like reinstalling the Sage library spkg would meanwhile also
>> trigger the rebuild of *all* extension modules; not sure since when
>> though... (seen with Sage 5.10.beta5 at least)
>
> Yep. This is because #14570, merged into Sage 5.10.beta4, completely
> breaks sage-sync-build, which is also used in the Sage library's
> spkg-install. (sage -sync-build now removes all Cython-generated files,
> hence *all* have to get rebuilt afterwards.)
>
>
> IMHO both issues (complete rebuild upon cloning and after
> sage-sync-build) should be resolved before the final 5.10 gets out.

Note that "cythonizing out-of-tree" may also break sage-clone (not just
sage-sync-build), depending on the (rather random) order in which
subdirectories of $SAGE_SRC/build/ are coopied.

Cf. http://trac.sagemath.org/sage_trac/ticket/13245#comment:50 ff.

[leif]

leif wrote:
> leif wrote:
>> leif wrote:
>>> Looks like reinstalling the Sage library spkg would meanwhile also
>>> trigger the rebuild of *all* extension modules; not sure since when
>>> though... (seen with Sage 5.10.beta5 at least)
>>
>> Yep. This is because #14570, merged into Sage 5.10.beta4, completely
>> breaks sage-sync-build, which is also used in the Sage library's
>> spkg-install. (sage -sync-build now removes all Cython-generated files,
>> hence *all* have to get rebuilt afterwards.)
>>
>>
>> IMHO both issues (complete rebuild upon cloning and after
>> sage-sync-build) should be resolved before the final 5.10 gets out.
>
> Note that "cythonizing out-of-tree" may also break sage-clone (not just
> sage-sync-build), depending on the (rather random) order in which

> subdirectories of $SAGE_SRC/build/ are copied.

Disabling / postponing "cythonizing out-of-tree" is now

http://trac.sagemath.org/sage_trac/ticket/14721 ,

a 5.10 blocker ticket; trivial patch, needs review...