Speed-up compilation time

[claudio...@gmail.com]

Hi.

I'm writing a Cython module and it becoming bigger; compilation time is now really long. There's something I could do to get a faster compilation? Something like splitting my module in multiple source files, recompile only what's needed to a .o file and let the linker assemble the final .so shared library, just like C with makefile?

Thanks for any suggestion.

[Stefan Behnel]

claudio.cilloni schrieb am 20.01.19 um 11:38:

Since this is a common question, I took the opportunity to write a FAQ
entry about it:

https://github.com/cython/cython/wiki/FAQ#how-can-i-speed-up-the-c-compilation

Stefan

[claudio...@gmail.com]

Down to 11 seconds from 41. That's a good compromise.

Thanks a lot.

Bye

[Jérôme Kieffer]

On Sun, 20 Jan 2019 04:51:43 -0800 (PST)
claudio...@gmail.com wrote:

> Down to 11 seconds from 41. That's a good compromise.

ccache works also (great) for cython & gcc !

Cheers,

Jérôme

[René Dudfield]

From python 3.5:

https://bugs.python.org/issue5309

* The ``build`` and ``build_ext`` commands now accept a ``-j``  +  option to enable parallel building of extension modules.

python3 setup.py build -j4 install

This improves compilation of pygame(with 36 modules) from 20 seconds to 8.3 seconds with optimization enabled on a 4 core laptop.

If you have 64 cores, and 64 separate modules... :)

There are some distutils hacks around for cythonize with multiprocessing as well.

[claudio...@gmail.com]

Unfortunately the the longest part of the process is the compilation of the single big .c file generated by Cython, and it cannot be parallelized. Unless I split my code into separated cython modules, I cannot take advantage of -j.

Thank you, anyway.

[René Dudfield]

Yeah, splitting your code up into separate modules will give you big improvements.

Because if you only change one part, then only that part is recompiled.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brock Mendel]

> ccache works also (great) for cython & gcc !

I've just learned about ccache and have installed it, but it isn't getting used.  Do I have to do anything cython-specific to tell cython to use it?

[René Dudfield]

CC='ccache gcc' python setup.py build

https://wiki.archlinux.org/index.php/Ccache

https://ccache.dev/manual/latest.html

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/98593db0-351f-49ca-aa29-2eb9334541e5%40googlegroups.com.

[Robert Bradshaw]

On Thu, Oct 31, 2019 at 6:16 PM Brock Mendel <jbrock...@gmail.com> wrote:
>
> > ccache works also (great) for cython & gcc !
>
> I've just learned about ccache and have installed it, but it isn't getting used. Do I have to do anything cython-specific to tell cython to use it?

It should just work, but if it doesn't compare the sources and file a
bug (we try to weed out any non-determinism in what we produce, but
it's possible something slipped through).

For pip installs, be aware of
https://ccache.dev/manual/latest.html#_compiling_in_different_directories
as it compiles things in ephemeral temporary directories.

> On Sunday, January 20, 2019 at 11:04:50 PM UTC-8, Jerome Kieffer wrote:
>>
>> On Sun, 20 Jan 2019 04:51:43 -0800 (PST)
>> claudio...@gmail.com wrote:
>>
>> > Down to 11 seconds from 41. That's a good compromise.
>>
>> ccache works also (great) for cython & gcc !
>>
>> Cheers,
>>
>> Jérôme
>