Shedskin - Compile Error

[Harnz]

Hi everyone,

I'm currently trying to compile a python project (5files @ total 1200 lines of code) with shedskin. I tried shedskin Version 0.9.3 and 0.9.2 both result in the same errors. This is the first error I encounter: mmain.cpp: In function ‘__shedskin__::list<__shedskin__::list<int>*>* __mmain__::list_comp_3(__shedskin__::__ss_int)’: mmain.cpp:133: error: no matching function for call to ‘__shedskin__::list<__shedskin__::list<int>*>::append(__shedskin__::list<double>*)’ Moreover, I after running shedskin (i.e. before typing "make") I receive many warnings - all related to dynamic types: *WARNING* mmain.py: expression has dynamic (sub)type: {float, int, list} However, shedskin seems to work flawlessly with the provided examples since I can compile and execute them without any errors. Do you have an idea where to look for the error or what the error is related to? Thank you.

[Mark Dufour]

hi harnz,

thanks for asking!

Moreover, I after running shedskin (i.e. before typing "make") I receive many warnings - all related to dynamic types: *WARNING* mmain.py: expression has dynamic (sub)type: {float, int, list}

there's no need to run 'make' after such warnings, because you'll have to resolve the dynamic types first. hopefully shedskin also provided you with a few warnings with line numbers and/or variable names, so you have some hints as to where exactly the mentioned types are being mixed. if you look at these lines the problem is usually easy to see. for example a piece of code such as:

for l in several_lists:
    do_something..
..
for l in some_numbers:
    do_something_else

in this case, the variable 'l' contains both lists and numbers at different times, and introducing a variable named 'l2' solves the problem.

However, shedskin seems to work flawlessly with the provided examples since I can compile and execute them without any errors.

Do you have an idea where to look for the error or what the error is related to?

if the problem remains unclear, you can always send your code (in private if necessary) and I will have a look..

thanks again for asking,
mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Harnz]

Hi Mark,

thanks for your advice. I'll try to look solve the problem for the next few hours and get back to you if I can't resolve it.

What I realized so far is the following:
1) All filles except for the main.py (inlcuding the main) compile flawlessly if I run:
1.1) shedskin include1.py
1.2) make (to make include1.o)
1.3) shedsking include2.py ...
2) I preallocate a list of some class A with integers (e.g.: a_list = [0] * num_items ... and use a_list to store Objects of class A later on). I guess that I have to preallocate it with some instance of calls A, right? (e.g.: a_list=[A()] *num_items)

Talk to you soon.
Best,
Harnz

[Harnz]

Hi Mark,

so I invested some time to comeup with a minimalistic code example that illustrats the problem:

def gamma2(x):
    gamma = 15. - 5.*x + 3.* x**2
    return gamma
 
def foo(hx, hy, hz, a):
    a2 = a ** 2
    hx2 = hx ** 2
    hy2 = hy ** 2
    hz2 = hz ** 2
    counter = 20

    xx= [0.]*(counter)
    some_floats = [0.]*(counter)
    indices = [[0]*3 for i in range(counter)]
    for i in range(counter):
        some_floats[i] = 1. / (a2 ** 3) * xx[counter] - 1. / (a2** 3) * x[counter]
        indices[i][0] = i-1
        indices[i][1] = i-1
        indices[i][2] = i-1
    return some_floats, indices, counter

if __name__ == '__main__':
    a=6.
    hx = 36./4
    hy = 36./4
    hz = 36./4

    x, xi, nmbneigh = foo(hx, hy, hz, a)

The shedskin feedback looks like:
*** SHED SKIN Python-to-C++ Compiler 0.9.2 ***
Copyright 2005-2011 Mark Dufour; License GNU GPL version 3 (See LICENSE)

[analyzing types..]
********************************100%
[generating c++ code..]
*WARNING* example.py: expression has dynamic (sub)type: {int, list}
*WARNING* example.py: variable 'nmbneigh' has dynamic (sub)type: {int, list}
*WARNING* example.py: variable 'x' has dynamic (sub)type: {int, list}
*WARNING* example.py: variable 'xi' has dynamic (sub)type: {int, list}
*WARNING* example.py:1: function gamma2 not called!
*WARNING* example.py:5: expression has dynamic (sub)type: {int, list}
*WARNING* example.py:16: class 'int' has no method '__getitem__'
*WARNING* example.py:16: expression has dynamic (sub)type: {float, list}
*WARNING* example.py:16: expression has dynamic (sub)type: {int, list}
*WARNING* example.py:20: expression has dynamic (sub)type: {int, list}
*WARNING* example.py:20: tuple with length > 2 and different types of elements
*WARNING* example.py:28: expression has dynamic (sub)type: {int, list}

How can I resolve the dynamic type of x,xi and nmbneigh and what causes this problem?

Thanks in andvance :)

Best,
Harnz

[Mark Dufour]

hi harnz,

thanks for the minimized snippet. the root of the problem seems to be the tuple that 'foo' returns. a three-tuple with different types of elements is not allowed at the moment. with something like this it works:

return (some_floats, indices), counter

so we simulate a three-tuple with two nested two-tuples.

another approach is to define a class with three attributes, which should work as well:

return myclass(some_floats, indices, counter)

perhaps later we can improve the warning system so it better points at the root a a dynamic typing problem.

let me know how this works for you, and/or if you run into further problems.

thanks again,
mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Harnz]

Hi Mark,

thanks for your quick reply. This work-around is working.

After resolving some more errors, I finally have only two Warnings left:
*WARNING* example.py:152: class 'float' has no method '__getitem__'
*WARNING* example.py:156: variable 'pot' has no type

These result in the following compiler error:
example.cpp: In function 'void* __example__::TestCompTopLevel(__shedskin__::list<__mmain__::tt*>*, __shedskin__::__ss_int)':
example.cpp:286: error: request for member '__getitem__' in 'tts->__shedskin__::list<T>::__getfast__ [with T = __mmain__::tt*]((nmbtts + -0x00000000000000001))->__mmain__::tt::stencilpot->__shedskin__::list<T>::__getfast__ [with T = double](i)', which is of non-class type 'double'
example.cpp:292: error: 'void*' is not a pointer-to-object type

Best,
Harnz

[Harnz]

Hi Mark,

nevermind... I it just was a minor problem and I finally fixed it.

So the code is fully converted by shedskin :)
I'll post speedups w.r.t. pure python and PyPy later on.

One more thing, can I use openmp to parallelize the FAST_FOR loops?
Where are these loops defined?

Best,
Harnz

[Harnz]

Hi Mark,

so I finally got the most compute-intensive loop running in parallel.

Here are some comments:
1) Would it be possible to generate a Makefile which generates .o files, such that one don't have to recompile the whole project if only one file gets changed?
2) Would it be possible to remove the if(s==0) condition (which checks the step of the loop) from FAST_FOR? Maybe by passing some option to shedskin.
3) Would it be possible to remove the check if(s>=0) from FAST_FOR? I believe this is something which can be checked when shedskin converts the python code to C++.
4) I realized that python lists translate to std::list. Is it possible to translate a list of floats to an contiguous float* array in C++?

Moreover, here are some timings (func1 + func2 = total):
Pure python: (872.88 + 100.1) sec
PyPy: (29.74 + 12.9) sec
Shedskin(23.9 + 2.25) sec
Shedskin@12Threads: (2.8 + 3.1) sec (((please note: I've only parallelized the first part)))

Hence, this yields a speedup of 165x over the original python version :)

Best,
Harnz

Am Dienstag, 8. Januar 2013 01:41:14 UTC+1 schrieb Harnz:

[Mark Dufour]

hiya harnz,

 

so I finally got the most compute-intensive loop running in parallel.

 

interesting! ^^

Here are some comments:
1) Would it be possible to generate a Makefile which generates .o files, such that one don't have to recompile the whole project if only one file gets changed?

well, I guess, but for larger programs the type analysis is really the slowest part, and this is a global analysis which isn't easily split across modules. fortunately there is another way to incrementally compile things, which is to remember analysis results between compile sessions. this would make analyzing the same or a slightly modified program dramatically faster.

I consider it a secondary optimization to then look at the Makefile, and perhaps only update .cpp files if they really change, so object files can also be updated incrementally.

I'm afraid shedskin is currently in maintenance mode though, so I probably won't start work on such a major new feature anytime soon.

2) Would it be possible to remove the if(s==0) condition (which checks the step of the loop) from FAST_FOR? Maybe by passing some option to shedskin.

would this help in any way? :P well, perhaps we could add a switch to disable this kind of 'debugging' checks that perhaps nobody sane should rely on in production code..
 

3) Would it be possible to remove the check if(s>=0) from FAST_FOR? I believe this is something which can be checked when shedskin converts the python code to C++.

yeah, I guess the step argument to (x)range is usually missing or a positive constant, so that would be possible. but again, would this really help in any way..?
 

4) I realized that python lists translate to std::list. Is it possible to translate a list of floats to an contiguous float* array in C++?

in fact they are translated to __shedskin__list, which uses a std::vector internally, so this is already the case.. :-)

as a sidenote, we probably want to get rid of the STL completely in the long term, to be able to match cpython even more closely in terms of performance characteristics. it was already dropped for the dict and set implementations, which are now based on their cpython counterparts.

Moreover, here are some timings (func1 + func2 = total):
Pure python: (872.88 + 100.1) sec
PyPy: (29.74 + 12.9) sec
Shedskin(23.9 + 2.25) sec

looks pretty good. did you use shedskin -b here? and did you profile the result, to see what the current bottleneck is? I'd be very interested to see for example the output of gprof2dot (see the shedskin documentation wiki for details about profiling and other performance tips).

you could also always send me the program in private, so I can have a look if I can get it to run faster with some minor changes.

Shedskin@12Threads: (2.8 + 3.1) sec (((please note: I've only parallelized the first part)))

please note that I'm quite sure the current implementation of the standard libs is not thread-safe in several places. the reason may lie in the fact that I'm not a big fan of thread-based parallellization (of course sometimes it's unavoidable). would it be possible to use process-based parallellization for you program? you could for example have a look at 'pylot' in the shedskin examples, which combines iirc pygame with multiprocessing and a shedskin-compiled extension module.

thanks!

mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk