Sympy import time

[Richard Rondu]

Hi everyone.

I'm upgrading my Qt RPN calculator app for sailfish (https://github.com/lainwir3d/sailfish-rpn-calculator) to use sympy as an engine. 

Everything was well and happy when I was using the sailfish emulator to test the app, but then I decided to finally try it on my Jolla smartphone : the import time of sympy increase to a bit above ten seconds.

While I can manage to deal with ~5seconds, ~10seconds is a bit too much. I was using sympy 0.7.6, I moved yesterday to the master branch hoping it would be better but it's not.

Do any of you have any tips to decrease the import time ? I'm only using for the moment : 

- sympy.S / N

- sympy.cos / acos / sin / asin / tan / atan

- sympy.sqrt / root / log / factorials

I'm defining some custom constants (inheriting NumberSymbol) and some functions (inheriting Function) but I can import them later when needed. What I need to be quickly loaded at startup is mainly sympy.S, the others I can load in the background.

I tried to play a bit with __init__.py, here is the import time of each module on the master :

core=2373

logic=0

assumptions=78

polys=5542

series=0

functions=1

ntheory=2

concrete=1307

simplify=1

sets=0

solvers=0

matrices=0

geometry=349

utilities=0

integrals=0

tensor=0

parsing=2

calculus=21

plotting=177

printing=0

printing2=1

printing3=0

interactive=34

evalf=1

deprecated=8

Polys module is the one taking most of the time. Here is the import time of its submodules :

polytools=2009

polyfuncs=1736

rationaltools=0

polyerrors=1

numberfields=1190

monomials=0

orderings=0

rootoftools=0

polyroots=0

domains=0

constructor=0

specialpolys=1

orthopolys=0

partfrac=467

polyoptions=1

rings=0

fields=33

The import time is the same either with "import sympy" or "from sympy import S"

So, if anyone had any idea / pointers I could look at I would really appreciate it :-) Apart from that, sympy is quite awesome !

Thanks

Richard

[Ondřej Čertík]

Hi Richard,

On Thu, Jul 23, 2015 at 3:23 AM, Richard Rondu <rondu....@gmail.com> wrote:
> Hi everyone.
>
> I'm upgrading my Qt RPN calculator app for sailfish
> (https://github.com/lainwir3d/sailfish-rpn-calculator) to use sympy as an
> engine.
>
> Everything was well and happy when I was using the sailfish emulator to test
> the app, but then I decided to finally try it on my Jolla smartphone : the
> import time of sympy increase to a bit above ten seconds.
>
> While I can manage to deal with ~5seconds, ~10seconds is a bit too much. I
> was using sympy 0.7.6, I moved yesterday to the master branch hoping it
> would be better but it's not.

Even 5s is unnaceptable, at least for me.

I am really sorry it is taking so long to import, we need to fix that.
I hope it is possible to fix this in Python.

Thanks for posting the timings. To get to the bottom of the issue ----
is the import time slow because we are doing some (slow) calculations
at import time, or is the import time slow because Python needs to
read and parse ~200,000 lines of code?

Once we understand where the time is actually spent, then we need to
figure out a solution.

Ondrej

[Roberto Colistete Jr.]

On 23-07-2015 13:41, Ondřej Čertík wrote:
> Hi Richard,
>
> On Thu, Jul 23, 2015 at 3:23 AM, Richard Rondu <rondu....@gmail.com> wrote:
>> Hi everyone.
>>
>> I'm upgrading my Qt RPN calculator app for sailfish
>> (https://github.com/lainwir3d/sailfish-rpn-calculator) to use sympy as an
>> engine.
>>
>> Everything was well and happy when I was using the sailfish emulator to test
>> the app, but then I decided to finally try it on my Jolla smartphone : the
>> import time of sympy increase to a bit above ten seconds.
>>
>> While I can manage to deal with ~5seconds, ~10seconds is a bit too much. I
>> was using sympy 0.7.6, I moved yesterday to the master branch hoping it
>> would be better but it's not.
> Even 5s is unnaceptable, at least for me.

Hi,

SymPy 0.7.5 loads in 3.2-4.0s on Jolla smartphone
(https://jolla.com/jolla/) inside Integral
(https://openrepos.net/content/rcolistete/integral-sailfish-os),
Derivative
(https://openrepos.net/content/rcolistete/derivative-sailfish-os) and
Limit (https://openrepos.net/content/rcolistete/limit-sailfish-os) for
Sailfish OS.

Here are some old analysis (benchmark, etc) of SymPy (0.7.1) in
mobile/desktop OS :
http://www.robertocolistete.net/Python/SymPy/
I will later try to measure the SymPy import time in some smartphones
for diferente SymPy versions (0.7.1, etc).

Best regards,

Roberto

[Jason Moore]

Ondrej,

This is also a function of the processor speed, etc. On my laptop "from sympy import S" seems to take half a second. So this request may be similar to "can sympy run fast on my old laptop?"

Jason
moorepants.info
+01 530-601-9791

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBhEgRHww2Ehrwc554Hik4-rMmOaz4_ner%2B0%3D%3Dnx%3Dy_-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Rondu]

Hi Roberto.

Thanks for the info. That's quite weird, I'll do a test ASAP with 0.7.5 to see how long it takes to import. I'm deploying sympy inside of my rpm (in it's own import path) but I don't think that's relevant.

Thanks for the help.

Richard

[Richard Rondu]

Hi Ondrej,

Let's say I can manage to hide this a bit in the background before the user enter it's first operand and so on. It's more manageable than 10seconds :-/

I don't know where the time is actually spent, I can try to take a look at that though.

Thanks for answering :-)

Richard

[Aaron Meurer]

Try installing fastcache and see if that helps (you'll have to compile it).

Aaron Meurer

--

You received this message because you are subscribed to the Google Groups "sympy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/46c4b1dd-5071-400d-9d1c-155633823213%40googlegroups.com.

[Richard Rondu]

Hi,

Doesn't seems to help unfortunately.

I tried installing manually 0.7.5 but I still get 10seconds of import. Then I tried getting it from Roberto's rpm on openrepos, and indeed it imports correctly after ~4seconds, and that's weird.

Searching for differences between the rpm sympy files and my package, I noticed I had all the .pyo files and much more .pyc files ouside __pycache__ folders than the rpm. I removed them and it didn't change anything.

The resulting difference between the two folders is in the included file "files_diff.txt". TL;DR : It seems I have more modules that Roberto's rpm.

So I tried to copy roberto's sympy directory into my rpm import path in /usr/share/harbour-rpncalc/lib/python/, expecting to get 4seconds of import time. But no, it's back to 10seconds meaning something is wrong with the import path I guess.

I'll investigate further with verbose mode and keep you informed.

Cheers and thanks for the advices again.

Richard

[Richard Rondu]

I think I found the problem, which seems more my package / sailfish related than sympy.

For each module, during the import, I get : 

# bytecode is stale for 'sympy.interactive'

# code object from /usr/share/harbour-rpncalc/lib/python/sympy/interactive/__init__.py

# could not create '/usr/share/harbour-rpncalc/lib/python/sympy/interactive/__pycache__/__init__.cpython-34.pyc': PermissionError(13, 'Permission denied')

Full traces attached.

I guess this could slow down import "a bit"... ;-)

Now why is the permission denied, I don't know because the folders access right are good. I'll take a look at that, but I guess it doesn't concern sympy anymore. I'll keep posting updates just for reference if someone has an identical problem.

Richard

[Aaron Meurer]

You should definitely ship pre-compiled bytecode. That will speed up the import times a lot. Python determines if it needs to recompile the bytecode based on the file timestamps. If something changes those, it will recompile the bytecode and try to write new ones.  

Aaron Meurer