problems with sage -dev checkout --ticket (malformed dependencies)

[Martin R]

Hi there!

I tried to checkout a ticked using  sage -dev checkout --ticket 14347

This yields just an error message "malformed dependencies".

I then tried to follow http://wiki.sagemath.org/TentativeConventions, but it's not completely clear for me what I should do:

a) should I first "get the latest official development version" as follows

~/sage-git$ git fetch origin           # get the latest repository information from Trac

~/sage-git$ git reset --hard master    # make all your files correspond to the local master branch

~/sage-git$ git clean -d -f            # get rid of any untracked files or directories

~/sage-git$ git pull --ff-only         # move the local master branch forward to match the information from Trac

~/sage-git$ make start 

In particular, should I really do this in my local master branch?  I somehow would like to keep a copy of sage 6.0 (it took a night to compile!)

b) or should I just do git checkout origin/u/agd/gcis -b u/agd/gcis

which creates a local branch u/agd/gcis.  This looks fine at first, however, now issuing "make start" seems to take forever (roughly an hour).  And, worse, switching back to master and issuing "make start" again, seems to take just as long.  

See below how the output of "make start" looks like at the beginning...

Many thanks,

Martin

Updating Cython code....

Compiling sage/algebras/quatalg/quaternion_algebra_element.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/algebras/letterplace/free_algebra_letterplace.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/algebras/letterplace/free_algebra_element_letterplace.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/algebras/letterplace/letterplace_ideal.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/algebras/quatalg/quaternion_algebra_cython.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/calculus/riemann.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/categories/action.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/categories/category_singleton.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/categories/map.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/categories/morphism.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/categories/examples/semigroups_cython.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/coding/binary_code.pyx because it changed.

Compiling sage/combinat/expnums.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/matrices/dancing_links.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/structure/list_clone.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/structure/list_clone_demo.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/structure/list_clone_timings_cy.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/sets/finite_set_map_cy.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/partitions.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/permutation_cython.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/debruijn_sequence.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/degree_sequences.pyx because it changed.

Compiling sage/combinat/combinat_cython.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/enumeration_mod_permgroup.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/combinat/crystals/letters.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/crypto/boolean_function.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/ext/c_lib.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/ext/fast_callable.pyx because it depends on sage/libs/pari/decl.pxi.

Compiling sage/ext/fast_eval.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/ext/multi_modular.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/ext/pselect.pyx because it depends on sage/ext/signals.pxi.

Compiling sage/finance/time_series.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/functions/prime_pi.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/geometry/toric_lattice_element.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/geometry/integral_points.pyx because it changed.

Compiling sage/geometry/triangulation/base.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/chrompoly.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/cliquer.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/graph_decompositions/vertex_separation.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/convexity_properties.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/comparability.pyx because it depends on ./sage/graphs/distances_all_pairs.pxd.

Compiling sage/graphs/generic_graph_pyx.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/distances_all_pairs.pyx because it changed.

Compiling sage/graphs/base/static_sparse_graph.pyx because it changed.

Compiling sage/graphs/base/static_sparse_backend.pyx because it changed.

Compiling sage/graphs/modular_decomposition/modular_decomposition.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/weakly_chordal.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/matchpoly.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/graph_decompositions/rankwidth.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/spanning_tree.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/trees.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/genus.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/hyperbolicity.pyx because it depends on ./sage/graphs/distances_all_pairs.pxd.

Compiling sage/graphs/base/c_graph.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/base/sparse_graph.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/graphs/base/dense_graph.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/group.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/old.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/libgap_wrapper.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/permgroup_element.pyx because it changed.

Compiling sage/groups/semimonomial_transformations/semimonomial_transformation.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/double_coset.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/refinement_binary.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/refinement_graphs.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/refinement_lists.pyx because it depends on sage/ext/stdsage.pxi.

Compiling sage/groups/perm_gps/partn_ref/refinement_matrices.pyx because it depends on sage/ext/stdsage.pxi.

...

[Simon King]

Hi!

On 2013-12-29, Martin R <axio...@yahoo.de> wrote:
> I tried to checkout a ticked using sage -dev checkout --ticket 14347
>
> This yields just an error message "malformed dependencies".

I once had a similar problem, and I think I solved it *on the ticket*.
It is "comma separated" versus "blank space separated" list. IIRC, it
should be a comma separated list, so, the problem can likely be solved
by inserting a missing comma.

Best regards,
Simon

[Jeroen Demeyer]

On 2013-12-29 06:20, Martin R wrote:
> b) or should I just do git checkout origin/u/agd/gcis -b u/agd/gcis
>
> which creates a local branch u/agd/gcis. This looks fine at first,
> however, now issuing "make start" seems to take forever (roughly an
> hour). And, worse, switching back to master and issuing "make start"
> again, seems to take just as long.

This is normal since stdsage.pxi indeed changed since Sage 6.0 and
essentially all Cython modules depend on stdsage.pxi. Luckily, changes
to stdsage.pxi are relatively rare so this problem isn't going to happen
all the time. A recommended solution would be to not go back to Sage 6.0
but to the "develop" branch.

Jeroen.

[Martin R]

brilliant!  I modified the dependencies on the ticket into a comma-space separated list, now it works.

I think this should be mentioned in the error message, but I have no idea how to do this.

I'm left with the problem on how to keep various versions of sage around, without having to recompile several hours.  I just tried (after "make start") in u/agd/gcis

git checkout master

git checkout u/agd/gcis

./sage -b

and it starts compiling everything again :-(

Martin

[Martin R]

Hm, does "git checkout" touch stdsage.pxi?  Thus, even if I go to a different branch with "git checkout" by accident, I have to recompile?

Also, is there a way to create a "snapshot" of a build.  Namely (I was slightly imprecise), I tried

git checkout master

./sage

to see whether it would come up with the "clean" version of sage, i.e., without the ticket applied.  (It doesn't...)

Many many thanks,

Martin 

[P Purkayastha]

It won't because the git repo tracks only the source files, not the
compiled and installed files. So, once you checkout master you should
rebuild sage so that it compiles and installs the updated source files.

The following are my observations on working with the git dev process.
Ideally, if you are doing sage development

1. Avoid going to master. Because there will potentially be too many
changes between master and the ticket in the later stages of the release
process (rc's).

2. Try staying on develop branch. If you checkout a ticket, and the
ticket is not based off the latest develop branch, then merge the
develop branch into your local copy:

$ git fetch origin # get the latest changes from upstream
$ sage --dev checkout --ticket <number> # get the ticket patches
$ git merge origin/develop # merge the latest changes from develop
$ sage -b # assuming the only changes are in sage lib

There are two things I am not sure about:

i) how will this affect the ticket if you push some revision into the
ticket at some later stage? Till now I have not pushed anything to the
ticket if there is already a different author.

ii) what is the correct command to use instead of 'sage -b' if there
are updates to libraries other than sage? I think the correct command
should be 'sage -upgrade'. If so, the following command should be run
instead of "git fetch origin" as a first step:
$ sage -upgrade develop

3. Try to enable ccache, but note that this will take up a lot of disk
space (iirc, about 4G in ~/.ccache). To do so, install sage with
SAGE_INSTALL_CCACHE="yes". This way when you make major branch changes,
for instance between master and develop, the amount of compilation will
be less, and it will also be faster.

Two things I am not sure about here

i) how we should enable ccache retroactively?

ii) is ccache automatically used if it is installed? Or, should we
permanently set the environment variable SAGE_INSTALL_CCACHE in our
${SHELL}rc?

- basu.

[Martin R]

Am Sonntag, 29. Dezember 2013 10:51:50 UTC+1 schrieb P Purkayastha:

On 12/29/2013 05:23 PM, Martin R wrote:
> Hm, does "git checkout" touch stdsage.pxi?  Thus, even if I go to a
> different branch with "git checkout" by accident, I have to recompile?

 

It won't because the git repo tracks only the source files, not the

compiled and installed files. So, once you checkout master you should
rebuild sage so that it compiles and installs the updated source files.

I just checked: stdsage.pxi *is* touched if I go to a different branch and then back.  Thus, the recompilation time depends not only on the branch you are in and the branch you want to compile, but also on branches you "visit" (eg. by accident, or to check some source file) inbetween.

So, I guess I really need to learn how to take "snapshots".

Martin

[Jeroen Demeyer]

On 2013-12-29 10:51, P Purkayastha wrote:
> ii) what is the correct command to use instead of 'sage -b' if there
> are updates to libraries other than sage?

make

[Jeroen Demeyer]

On 2013-12-29 10:51, P Purkayastha wrote:

> i) how we should enable ccache retroactively?

I don't understand this question.

> ii) is ccache automatically used if it is installed?

Yes, either system ccache or Sage ccache.

[Simon King]

On 2013-12-29, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> This is normal since stdsage.pxi indeed changed since Sage 6.0 and
> essentially all Cython modules depend on stdsage.pxi. Luckily, changes
> to stdsage.pxi are relatively rare so this problem isn't going to happen
> all the time. A recommended solution would be to not go back to Sage 6.0
> but to the "develop" branch.

... or use ccache.

Cheers,
Simon

[Martin R]

But I do have ccache installed!!! (I followed precisely the TentativeConventions)

Martin

[Jeroen Demeyer]

On 2013-12-29 11:54, Martin R wrote:
> But I do have ccache installed!!! (I followed precisely the
> TentativeConventions)

What does "ccache -s" output (it will show whether it was used or not)

[Martin R]

martin@laptop:~/sage-git$ local/bin/ccache -s

cache directory                     /home/martin/.ccache

cache hit (direct)                  1004

cache hit (preprocessed)             928

cache miss                         14745

called for link                     2932

called for preprocessing             928

multiple source files                  9

compiler produced stdout               1

compile failed                       246

preprocessor error                   736

bad compiler arguments               162

unsupported source language          801

autoconf compile/link               1937

unsupported compiler option          399

no input file                       1024

files in cache                     33111

cache size                           3.2 Gbytes

max cache size                       4.0 Gbytes

[P Purkayastha]

On 12/29/2013 06:21 PM, Jeroen Demeyer wrote:
> On 2013-12-29 10:51, P Purkayastha wrote:
>> i) how we should enable ccache retroactively?
> I don't understand this question.

Suppose I install sage without setting SAGE_INSTALL_CCACHE. Now, I want
to use ccache. What should I do? Just run "make" with
SAGE_INSTALL_CCACHE="yes"?

[R. Andrew Ohana]

On Sun, Dec 29, 2013 at 4:02 AM, P Purkayastha <ppu...@gmail.com> wrote:

Suppose I install sage without setting SAGE_INSTALL_CCACHE. Now, I want to use ccache. What should I do? Just run "make" with SAGE_INSTALL_CCACHE="yes"?

The best way to use ccache is to prefix your path with ccache symlinks (which is what the ccache spkg does). Most distros automatically make these symlinks under /usr/libexec/ccache or something, but they don't add this to PATH. My suggestion is to lookup your distro's documentation on ccache.

One thing to note is that the default cache size is 1G, which is too small for the sage distribution. You will probably want to bump this up to ~4G with `ccache --max-size=4G`.

--

Andrew