Linking time is too slow

[Hai Nguyen]

hi all, i have been using Cython for wrapping a C++ lib (for data analysis of bimolecular MD simulation). 

https://github.com/pytraj/pytraj (mostly written in Cython)

I think I am in love with Cython (still prefer Cython over Julia). The only problem is the compiling/linking time is too slow (about 4 times slower than compiling the original C++ lib). It takes about 15 minutes to compile. 

So my question is: is there any trick/flag to make this faster? 

Thanks for paying attention to my naïve question. 

Hai

[Jeroen Demeyer]

On 2015-02-26 04:39, Hai Nguyen wrote:
> hi all, i have been using Cython for wrapping a C++ lib (for data
> analysis of bimolecular MD simulation).
> https://github.com/pytraj/pytraj (mostly written in Cython)
>
> I think I am in love with Cython (still prefer Cython over Julia). The
> only problem is the compiling/linking time is too slow (about 4 times
> slower than compiling the original C++ lib). It takes about 15 minutes
> to compile.
>
> So my question is: is there any trick/flag to make this faster?

Get a new linker? On Linux, you can try "gold" which is supposed to be a
lot faster for C++.

[Hai Nguyen]

thanks. I will give it a try.

Hai

--

--- You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefan Behnel]

Hai Nguyen schrieb am 26.02.2015 um 04:39:
> hi all, i have been using Cython for wrapping a C++ lib (for data analysis
> of bimolecular MD simulation).
> https://github.com/pytraj/pytraj (mostly written in Cython)
>
> I think I am in love with Cython (still prefer Cython over Julia). The only
> problem is the compiling/linking time is too slow (about 4 times slower
> than compiling the original C++ lib). It takes about 15 minutes to compile.
>
> So my question is: is there any trick/flag to make this faster?

For testing, you can usually get away by setting CFLAGS="-O0 -ggdb" or so.
I build much more often for testing stuff than for actually using (or
benchmarking) it, and disabling all C/C++ compiler optimisations can reduce
the turnover times quite dramatically.

Stefan

[Nhai]

Great. Thanks.

Hai

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

[Hai Nguyen]

I just tried adding '-O0 -ggdb' and the compiling time reduces about 40% for my case. Nice.

Hai

[Hai Nguyen]

It seems that travis upgraded their system. Compiling my program (pytraj) is now as fast as compiling the C++ lib that pytraj links to. (about 5 minutes (vs 15 minutes before vs 2.5 minutes to install Cython itself). Great. save my day.

Hai

[Hai Nguyen]

ooops. I am really sorry to post short note. I just want to correct this info. The compiling time is dramatically reduced because I included cythonized file and used the same Cython version in travis. 

Hai

[Hai Nguyen]

Hi all,

I just added multiprocessing to my setup file to use multiple cores for compiling Cython extension. By adding "CFLAGS="-O0 -ggdb" as suggested from Stefan and combining with multiple cores, I am able to compile my program in less than 2 minutes (vs >15 minutes as before) if using 8 cores.

Here is the code at the end of my setup file (https://github.com/pytraj/pytraj/blob/master/scripts/setup_faster_build.py)

if __name__ == "__main__":
    from multiprocessing import cpu_count
    n_cpus = cpu_count()
    num_each = int(len(ext_modules)/n_cpus)
    
    sub_ext_modules_list = []
    for i in range(n_cpus):
        if i != n_cpus-1:
            sub_ext_modules_list.append(ext_modules[i*num_each:num_each*(i+1)])
        else:
            sub_ext_modules_list.append(ext_modules[num_each*i:])

    from multiprocessing import Pool
    pool = Pool(n_cpus)
    pool.map(build_func, sub_ext_modules_list)

Hope this help for those exciting about Cython but frustrated about building process.

Hai

On Fri, Feb 27, 2015 at 1:23 AM, Stefan Behnel <stef...@behnel.de> wrote:

[Jérôme Kieffer]

On Thu, 2 Apr 2015 14:47:35 -0400
Hai Nguyen <nha...@gmail.com> wrote:

> I just added multiprocessing to my setup file to use multiple cores for
> compiling Cython extension.

Hi,

This look appealing, especially that it looks portable.
Do you know if it is _actually_ portable on mac, win & linux ?

Cheers,

--
Jérôme Kieffer <goo...@terre-adelie.org>

[Joshua]

I also built a custom parallel setup.py for a project that was taking forever to compile, although I feel like I had to do some more sophisticated stuff to capture stdout so that I wouldn't get interspersed writes from all of the workers. I never solved the issue of capturing the numpy API warnings that still caused a mess of things. But parallelizing took the build process from 15 minutes down to a little more than 2 on 8-cores, which was huge. Also cythonize() has a nthreads keyword to do the translation from cython to c in parallel, but it doesn't cover the build part. On windows you would probably need to use multiprocessing.freeze_support. 

Josh 

[Hai Nguyen]

Hi,

unfortunately I don't know if this is portable to mac or win.

btw, to make the multiprocessing really works, I need to run the build process twice, like 'python setup.py build', then Ctrl-C to cancel and then 'python setup.py build' again. 

Hai

[Ian Bell]

On Thu, Apr 2, 2015 at 1:59 PM, Hai Nguyen <nha...@gmail.com> wrote:

Hi,

unfortunately I don't know if this is portable to mac or win.

btw, to make the multiprocessing really works, I need to run the build process twice, like 'python setup.py build', then Ctrl-C to cancel and then 'python setup.py build' again. 

Hai

On Thu, Apr 2, 2015 at 3:19 PM, Jérôme Kieffer <goo...@terre-adelie.org> wrote:

On Thu, 2 Apr 2015 14:47:35 -0400
Hai Nguyen <nha...@gmail.com> wrote:

> I just added multiprocessing to my setup file to use multiple cores for
> compiling Cython extension.

Hi,

This look appealing, especially that it looks portable.
Do you know if it is _actually_ portable on mac, win & linux ?

Cheers,

If your project is mostly c++ code, with a little bit of cython, I've found great happiness using cmake to generate a static library which is then linked into the cython code.  In this way, you can get back to incremental builds so you only have to do a full recompile once.  For me, that took compilation of my library down from a few minutes to a few seconds for a minimal change that doesn't change any headers.  This setup.py shows what I did: https://github.com/CoolProp/CoolProp/blob/master/wrappers/Python/setup.py

Depending on your needs, this might help.

Ian
 

[Robert Bradshaw]

Sage provides parallel execution, see
https://github.com/sagemath/sage/blob/419f5ae5280eb124e56af80bee8328eb64f9cdc5/src/setup.py#L287

Note that a tricky bit is that there may be races in trying to create
the directories or other common artifacts that we had to be careful
about.

BTW, you can just write

sub_ext_modules_list = [ext_modules[k::num_each] for k in range(n_cpus)]

[Hai Nguyen]

great. thanks. (I had hard time when googling about building in parallel (got Parallel programming in python instead))

Hai

[Hai Nguyen]

Hi,

thanks for your tips. 

Yes, my project is mostly C++ code (already compiled to dynamic lib so I don't bother about this). My python package has about ~20K lines of Cython code.

You're right that when updating header file (.pxd), I need to have fresh building from start (I am using incremental build too). 

Hai