Lepl and Cython
Luca
I just tried to compile Lepl code using Cython, and it seems to
work :-)
Since I have already used Cython to speed up some of my critical code,
I thought that I could possibly do the same for Lepl and be able to
use it to handle some huge text files (> 1 GB). I could gain something
(around 30%) only by compiling the code, and much more (as indicated
in Cython tutorials) by adding a few cython decorators.
I would like to share the procedure, just in case someone is
interested.
First of all, one must install Cython (0.15.1) :
easy_install --upgrade cython
Then I added a section in Lepl setup.cfg to ask Cython to use a
temporary dir for generated .c and .so files :
[build_ext]
pyrex-c-in-temp=1
I also modified Lepl setup.py to allow a Cython compilation step :
from setuptools import setup, find_packages
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
import os
import pkg_resources
import glob
mods = []
for pkg in find_packages(where='src'):
pkgroot = os.path.join('src', pkg.replace('.', '/'))
for file in glob.glob(os.path.join(pkgroot, "*.py")):
if os.path.basename(file) not in [ '__init__.py', 'state.py',
'support.py', 'core.py'] :
base = os.path.splitext(os.path.basename(file))[0]
module = pkg + "." + base
extension = Extension(module, [file])
mods.append(extension)
setup(name='LEPL',
version='5.0.0',
[SNIP],
cmdclass={'build_ext': build_ext},
ext_modules=mods
)
As one can see, some files had to be skipped because of compilation
errors in Lepl code or of Cython limitations (i.e. @factory.setter
etc).
Now, to actually start the compilation, this command is needed :
python setup.py --verbose build_ext
To speed Lepl up, one could add decorators in critical parts, using
the "Pure Python Mode" or writing some .pxd files :
http://docs.cython.org/src/tutorial/pure.html
I hope this helps :-)
Best Regards,
Luca
andrew cooke
that's really cool, thanks.
so what was your final speedup?
has anyone tried pypy? i installed it last month, planning to test the speed,
but (as too often these days) ended up being distracted by somethinhg else
(the code i did try pypy on was actually slower than python by quite a bit,
which was a big surprise).
andrew
> --
> You received this message because you are subscribed to the Google Groups "lepl" group.
> To post to this group, send email to le...@googlegroups.com.
> To unsubscribe from this group, send email to lepl+uns...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/lepl?hl=en.
>
Luca
I did not take a real benchmark, but I could really see the
difference : the code I am personally working on had some tests taking
more than one minute and simply adding cython decorators in critical
parts made them complete in a couple of seconds, I think that, as
shown in cython tutorials, one can really expect a speedup of 100x or
150x when comparing to standard python code.
I would like to make some benchmarks in lepl, but I didn't understand
how to execute unit tests (I tried to configure "nose" but without
much success)...
I am also surprised about PyPy being slower, but since the approach is
so different from Cython, I think that the results can be different
too...
Please let me know if I can further contribute!
Best Regards,
Luca
On Sep 29, 12:08 pm, andrew cooke <and...@acooke.org> wrote:
> that's really cool, thanks.
>
> so what was your final speedup?
>
> has anyone tried pypy? i installed it last month, planning to test the speed,
> but (as too often these days) ended up being distracted by somethinhg else
> (the code i did try pypy on was actually slower than python by quite a bit,
> which was a big surprise).
>
andrew cooke
pytest is the only test runner i have had consistent succeess with - nose
doesn't like the "_" at the start of the _test directories.
performance test code is in lepl._performance and typically isn't unit tests,
but a "main" method that you can edit to print a timing to the command line,
or to generate profiler output. but because they are not tests they are not
run regularly and so may be broken (i just tried one and it was using "Float"
instead of "Real" for example).
but i'm explaining all that just in case it's any use for you. i don't think
it would help me much as the latest code has changed so much (and is still in
development). but this is something to add to the list of things to look at.
in general i'm not spending much time on lepl at the moment because (1) it
seems to be stable as it is (2) the changes i am doing are a lot of work and
(3) i'm side-tracked by a bunch of other projects.
cheers,
andrew