Numpy and shedskin

[Mark Dewing]

I've been looking at using shedskin to translate some code using numpy.  In this particular example, there is set of particles (in a 2d array) and a simple computation related to the distance between particles.

There seem to be two issues with infering the type correctly:

 - the 'numpy.array' constructor can be used to create 1d or 2d arrays (or higher dimensions, but not in this example).  Shedskin can't determine the correct array type (even though it should have the type information it needs).

 - slices of 2d arrays can either be a 1d array (single index - a[1]) or a scalar (index is a tuple - a[1,2])

More generally, it seems there should be some way of conveying this decision information to shedskin (the stub files don't necessarily work for the case where the output type of a function requires a decision based on the input type (?)).

Or have I done something wrong with the numpy stub file?  Both the example and the numpy stub file are attached.

Mark

[srepmub]

hi mark,

indexing and slicing indeed seem the most difficult to support, as the result in fact depends on both array rank and length of the index tuple..

I'm afraid 'isinstance' and 'len' don't work as expected during the type inference that shedskin performs, so they can't be used to model this behaviour.. :P shedskin currently doesn't track the length of things in any way, which in general even seems impossible (comparable to the halting problem), and 'isinstance' is in fact not supported at all anymore. needless to say type inference is very nasty business.

but I guess the 'rank' of the data that goes into array() can be usually known by shedskin. but we'd have to track things when combining arrays further probably.. the index tuple forms an even more fundamental problem perhaps, since we can't know its length beforehand in all cases. restricting tuples to always be specified by its elements would make the problem easier, as it would allow shedskin to insert the correct types during analysis.

I'm sure we'd find other similar problems when trying to add direct support for numpy. in short, I don't think we want to try and support it directly. it's quite big and dynamic (in a way that guido himself dislikes - types being dependent on non-type data essentially). we obviously can't support every library out there, so perhaps it makes limit ourselves to the python standard library. other libraries can also often be combined with a shedskin-compiled extension module (as happens with for example pygame and pygtk in shedskin/examples).

an alternative approach to integrate with numpy without the performance loss of having to convert things, and that would probably also be useful in other areas (the regular 'array' module would be one example) would be to figure out a simple way to share memory between cpython and shedskin. I've thought about this a few times, but have not come upon a satisfactory solution yet..

thanks in any case for having me read through the numpy tutorial.. ;-)
mark.