reduce compilation times during development

[P Purkayastha]

Currently, what methods are there to reduce the compilation times during development?

I have a desktop which is not that fast and it takes ages to get sage ready if I change branches. As an example, I ran the following commands

$ sage -upgrade develop   # this upgraded to sage-6.3.beta0
$ sage --dev checkout --ticket 16325  # somehow this was not based off develop
$ git rebase develop      # this rebased the branch ticket/16325 on develop
$ # made changes to only one file: sage/plot/contour_plot.py
$ sage -btp --long src/sage/plot

The build took 15 min to complete - it is running the tests now.

I have both ccache and cycache enabled. The last command above identified a lot of files which do not need compilation, but then it also identified over 300 files which needed to be compiled. This is a recurring problem that I face when changing branches and is really time consuming.

[Frédéric Chapoton]

Hello,

to avoid recompliation, I am rather doing the following (not sure if this is a good thing to do) :

$ git fetch trac    # needs to be in /sage

$ git ch develop        # I switch to the develop branch 

$ git pull trac develop   # this updates to the latest developement version of sage

$ make build     # to compile the latest sage

$ git ch develop -b 16325   # creates a new branch, you can choose the name as you want

$ git pull trac public/ticket/16325    # applis the branch in trac on top of a copy of develop

$ sage -bt whatever

This is merging develop into the branch 16325, so that small changes give small compilation time

But one again, maybe this is not an orthodox way..

[Ralf Stephan]

On Sunday, May 11, 2014 12:41:24 PM UTC+2, P Purkayastha wrote:

I have both ccache and cycache enabled.

Is this set?

$ set|grep CCA

SAGE_INSTALL_CCACHE=yes

I occasionally look at what changes in the output of "local/bin/ccache -s"

before and after compilation.

Regards,

[P Purkayastha]

Yes, this is set. I tried it again just now. I changed to develop using

git checkout develop

and re-ran the upgrade process. It shouldn't have upgraded anything, but it still decided to compile a bunch of packages anyway. Thankfully, it found most in ccache, except for R. But for sage, the following message was spit out during compilation:

...
Found compiled sage/rings/polynomial/polynomial_ring_homomorphism.pyx in cache
Found compiled sage/rings/polynomial/polynomial_zmod_flint.pyx in cache
Found compiled sage/rings/polynomial/polynomial_zz_pex.pyx in cache
Finished Cythonizing, time: 24.08 seconds.
running install
running build
running build_py
copying sage/plot/contour_plot.py -> build/lib.linux-x86_64-2.7/sage/plot
running build_ext
building 'sage.algebras.quatalg.quaternion_algebra_element' extension
building 'sage.algebras.letterplace.free_algebra_letterplace' extension
...
building 'sage.ext.interpreters.wrapper_py' extension
building 'sage.ext.interpreters.wrapper_el' extension
Executing 321 commands (using 2 threads)

So, it decided to still recompile 321 files. This is far from optimal.

[Volker Braun]

With ccache and cycache the recompliation time of extension modules is negligible. Should work automatically for Sage >= 6.2.rc0

[P Purkayastha]

All of this was on 6.3.beta0

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/WpDnmLZJKDE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

Try "sage -ba". If it is not very fast, tell us why cycache is not working on your machine.

[P Purkayastha]

Here is some more information. I checked out ticket/16325 and it differs from develop by only two commits to a single file.

~/Installations/sage-git» git l -n 3
* 2ecec6b - (HEAD, refs/heads/ticket/16325) bordercolor must not be a list. Fix this error. 24 hours ago <Punarbasu Purkayastha>
* 2b1a65b - Enable fillcolor and bordercolor for implicit_plot 24 hours ago <Punarbasu Purkayastha>
* fa885d9 - (refs/remotes/origin/develop, refs/heads/develop) Updated Sage version to 6.3.beta0 2 days ago <Volker Braun>%

CCACHE is in the environment

~/Installations/sage-git» set|grep CCA
Binary file (standard input) matches
~/Installations/sage-git» env|grep CCA
SAGE_INSTALL_CCACHE=yes

I had just finished running sage -upgrade (and there was nothing to upgrade in reality - see my previous emails), so the commands below are run immediately after this upgrade command.

~/Installations/sage-git» local/bin/ccache -s
cache directory                     /home/punarbasu/.ccache
cache hit (direct)               4002297
cache hit (preprocessed)          166297
cache miss                        206048
called for link                  2019701
called for preprocessing           21331
multiple source files                226
compiler produced stdout              40
compile failed                      6261
preprocessor error                  3187
bad compiler arguments              3602
unsupported source language        12963
autoconf compile/link              46173
unsupported compiler option        12682
no input file                      19175
files in cache                     24529
cache size                           3.5 Gbytes
max cache size                       4.0 Gbytes

~/Installations/sage-git» git co ticket/16325
Switched to branch 'ticket/16325'
~/Installations/sage-git» time sage -ba >& ~/sage.log
y
sage -ba >&~/sage.log  101.00s user 25.62s system 50% cpu 4:13.18 total


~/Installations/sage-git» local/bin/ccache -s
cache directory                     /home/punarbasu/.ccache
cache hit (direct)               4002676
cache hit (preprocessed)          166331
cache miss                        206066
called for link                  2020083
called for preprocessing           21331
multiple source files                226
compiler produced stdout              40
compile failed                      6261
preprocessor error                  3187
bad compiler arguments              3602
unsupported source language        12963
autoconf compile/link              46173
unsupported compiler option        12682
no input file                      19180
files in cache                     24565
cache size                           3.5 Gbytes
max cache size                       4.0 Gbytes

The time required (4 min) is much better than yesterday, but still too long. Is this to be expected? The log file sage.log is attached.

A copy of all the files is kept in src/build/lib.* - can it not just check the diff between those and the files in src/sage and decide which ones need recompiling?

[P Purkayastha]

On Monday, May 12, 2014 6:54:23 PM UTC+8, P Purkayastha wrote:

The time required (4 min) is much better than yesterday, but still too long. Is this to be expected? The log file sage.log is attached.

A copy of all the files is kept in src/build/lib.* - can it not just check the diff between those and the files in src/sage and decide which ones need recompiling?

I need to clarify something here. I forgot to set MAKE='make -j2', so it is using single thread. So, perhaps the build time is only 2 min and not 4min. Also, it seems only pyx or compiled files are kept in src/build.

 

[Volker Braun]

Is the time spent in gcc or in Cython or both?

[P Purkayastha]

Most of this time is spent in gcc. This can be seen if you grep the log file for "Finished Cythonizing". The commands following it are all gcc/g++ commands.

[Ralf Stephan]

On Monday, May 12, 2014 11:36:20 AM UTC+2, P Purkayastha wrote:

... It shouldn't have upgraded anything, but it still decided to compile a bunch of packages anyway.

I have witnessed the same thing recently. Someone with knowledge of the code please correct me. I think what is probably happening is that $HOME/.cycache over time fills with old entries. Some get cleaned but the ones not getting cleaned fill up, until the cleanup process is cleaning a part of the previously cached files every time a major build happens.

The obvious workaround is to rm -rf $HOME/.cycache when this occurs.

Regards,

[Volker Braun]

There are some global dependencies, most notably setup.py. So if you change that then every *.pyx file is re-cythonized, and hence gcc called on every one of them. If there was no change to the Cython output then gcc should answer everything from ccache, so it would still be fast. Of course you really have to work if the Cython output *did* change.

[leif]

Volker Braun wrote:
> There are some global dependencies, most notably setup.py. So if you
> change that then every *.pyx file is re-cythonized, and hence gcc called
> on every one of them. If there was no change to the Cython output then
> gcc should answer everything from ccache, so it would still be fast. Of
> course you really have to work if the Cython output *did* change.

Is ccache agnostic to the comment on the first line of each
Cython-generated C/C++ file?


-leif

> On Saturday, June 7, 2014 8:08:10 AM UTC+1, Ralf Stephan wrote:
>
> On Monday, May 12, 2014 11:36:20 AM UTC+2, P Purkayastha wrote:
>
> ... It shouldn't have upgraded anything, but it still decided to
> compile a bunch of packages anyway.
>
>
> I have witnessed the same thing recently. Someone with knowledge of
> the code please correct me. I think what is probably happening is
> that $HOME/.cycache over time fills with old entries. Some get
> cleaned but the ones not getting cleaned fill up, until the cleanup
> process is cleaning a part of the previously cached files every time
> a major build happens.
>
> The obvious workaround is to rm -rf $HOME/.cycache when this occurs.

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Volker Braun]

On Saturday, June 7, 2014 2:50:41 PM UTC+1, leif wrote:

Is ccache agnostic to the comment on the first line of each
Cython-generated C/C++ file?

Jein ;-)

Leads to a direct mode cache miss. But the slightly slower preprocessor mode will ignore comments and hence match.

 

[P Purkayastha]

I put up a post in Google+ with my current observations regarding this:

https://plus.google.com/115114386497793444118/posts/2sUudfMLjmP

--

[Ralf Stephan]

On Saturday, June 7, 2014 9:08:10 AM UTC+2, Ralf Stephan wrote:

On Monday, May 12, 2014 11:36:20 AM UTC+2, P Purkayastha wrote:

... It shouldn't have upgraded anything, but it still decided to compile a bunch of packages anyway.

I have witnessed the same thing recently. Someone with knowledge of the code please correct me. I think what is probably happening is that $HOME/.cycache over time fills with old entries. Some get cleaned but the ones not getting cleaned fill up, until the cleanup process is cleaning a part of the previously cached files every time a major build happens.

I have now seen this again and did the obvious diagnostic. The reason for filling up

the .cycache directory or for cycache cache misses seems to be that the generated date

changes the hash:

ralf@ark:~/.cycache> diff -u stl_vector.cpp-*

--- stl_vector.cpp-61714d1d7a3a0da309ac72b02739d9f9     2014-06-29 09:46:04.937703446 +0200

+++ stl_vector.cpp-a83c93575bc69cb4cf4cc2ba89617672     2014-06-26 18:20:34.098244133 +0200

@@ -1,4 +1,4 @@

-/* Generated by Cython 0.20.1 on Fri Jun  6 16:59:00 2014 */

+/* Generated by Cython 0.20.1 on Fri Jun  6 17:14:20 2014 */

 

 #define PY_SSIZE_T_CLEAN

 #ifndef CYTHON_USE_PYLONG_INTERNALS 

In hindsight it's obvious, isn't it?

Regards,

[Volker Braun]

The timestamp is in the Cython output, not the input. Your example is one where different inputs yielded the same Cython output, possibly because of a changed C include header.