shedskin 0.2.1 planned features

[srepmub]

hi all,

so here's the current plan for shedskin 0.2.1, which should be
released in about two months:

-support for itertools (thanks to the work of jeremie)
-support for csv (by eating our own dogfood, using something like
python-dsv - thanks chris)
-if we support itertools, then I guess 'map'/'filter'/'reduce' should
work too (yes, I give up my opposition :))
-as well as 'next'.. (any other useful (new) builtin functions
shedskin doesn't support at the moment?)
-and while we're passing function references around, it should be
possible to place these in containers and to pass method references
around as well
-support for the 'key' argument of 'list.sort' and 'sorted'
-passing keyword arguments to generated extension modules (done)
-boost type inference scalability by disallowing parametric
polymorphism (outside of lib/, as for data polymorphism it's just not
worth it.)


thanks,
mark.

[srepmub]

a status update:

> -support for itertools (thanks to the work of jeremie)

jeremie, I'm hoping to see another patch soon.. ^^ I'd be happy to
help out if you don't have time to fully complete it, now that most
other things on this list have been tackled. anyway, jeremie and I
sorted out most issues, and I think we can support practically
everything..

> -support for csv (by eating our own dogfood, using something like
> python-dsv - thanks chris)

I wrote a pure Python implementation based on _csv.c (and csv.py) from
CPython, and compiled it using shedskin (see csv15.py). I decided not
to support Sniffer for now, or custom Dialects, but most basic things
should work now.

> -if we support itertools, then I guess 'map'/'filter'/'reduce' should
> work too (yes, I give up my opposition :))

these have been added.. I also added support for passing around
builtin callables, which required a bit of surgery.. so the following
now works, for example:

print sorted([[2,3,4], [5,6], [7]], key=len)
print map(len, ['a','bc'])
print map(max, ['a','bc'], ['d'], ['e'])
print map(set, [[1]])

> -as well as 'next'.. (any other useful (new) builtin functions
> shedskin doesn't support at the moment?)

done.

> -and while we're passing function references around, it should be
> possible to place these in containers and to pass method references
> around as well

I decided to move this to a future release, because I've seen enough
function passing for now..

> -support for the 'key' argument of 'list.sort' and 'sorted'

done.

> -boost type inference scalability by disallowing parametric
> polymorphism (outside of lib/, as for data polymorphism it's just not
> worth it.)

this is the thing I will look into next. but if I get joris' original
blossoming algorithm to work with a simpler trick, I may also move
this one forward. there's already more than enough new stuff for a new
release.

some other things were improved in the meantime:

- forward referencing of variables and functions
- joris managed to improve basic indexing (without -b, by improving
inlining of __wrap)
- addition of 1-length lists and strings was optimized
- many, many bugfixes.

if I have time, I may look into buffers/memoryviews as suggested on my
blog (thanks!), to avoid copying overhead for extension modules, but I
will probably save this for the next release..

thanks,
mark.

[Jérémie Roquet]

Hello,

2009/12/20 srepmub <mark....@gmail.com>:

>> -support for itertools (thanks to the work of jeremie)
> jeremie, I'm hoping to see another patch soon.. ^^

^^

To sum up briefly what doesn't work yet for itertools:
- imap
- starmap
- product
- combinations_with_replacement (A 2.7 addition, btw)

The following have been completed since the first patch (and thanks to
Mark's improvements to shedskin):
- chain (accepts an arbitrary number of input iterables)
- tee (accepts an arbitrary number of output iterables)

The following have been added:
- izip (with some limitations: the iterables must be of the same type)
- izip_longest (with some limitations: the iterables must be of the
same type, and there is no way to distinguish None and 0 in the
output)
- permutations
- combinations

> I'd be happy to
> help out if you don't have time to fully complete it, now that most
> other things on this list have been tackled. anyway,

Depends on when you want the module to be complete, but I think
product and combinations_with_replacement will be available very soon
(there is no challenge there).
I haven't thought much about imap and starmap, but I'm expecting some
challenge there... so that's the last I'll work on.

Best regards,

--
Jérémie

[Mark Dufour]

Depends on when you want the module to be complete, but I think
product and combinations_with_replacement will be available very soon
(there is no challenge there).

thanks for the update, and nice to hear you are almost finished.. :) I am in no hurry to release, as I will probably need at least two more weeks to improve TI somewhat.. 

now that I look at starmap again, it looks like it may be impossible to support it fully at the moment.. the unpacking it does seems impossible to model, because the number of arguments it unpacks cannot be known during type inference (in contrast to (i)map, where you can count the arguments to (i)map..).

so maybe we can leave starmap out for now..? it doesn't look like a very popular function in any case.. 


thanks,
mark
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad taste'" - Linus Torvalds

[srepmub]

btw,

it may be interesting to also add support for 'heapq' before the next
release. it is quite small, and uses several itertools functions..

(see the networkx thread about compiling a graph algorithm that in
turn uses heapq..).

mark.

[Jérémie Roquet]

2009/12/26 srepmub <mark....@gmail.com>:

> it may be interesting to also add support for 'heapq' before the next
> release. it is quite small, and uses several itertools functions..
>
> (see the networkx thread about compiling a graph algorithm that in
> turn uses heapq..).

I'm looking at it this afternoon (unless you've already started to
work on it, of course :p).

--
Jérémie

[Mark Dufour]

I'm looking at it this afternoon (unless you've already started to
work on it, of course :p).

I'm still looking into type inference scalability at the moment, so go ahead.. :)


thanks,
mark.

[Jérémie Roquet]

2009/12/26 Mark Dufour <mark....@gmail.com>:

>> I'm looking at it this afternoon (unless you've already started to
>> work on it, of course :p).
> I'm still looking into type inference scalability at the moment, so go
> ahead.. :)

I attached a "preview" version, with the following functions :
- heappush
- heappop
- heappushpop
- heapify
- heapreplace

I still have to implement the following :
- merge
- nlargest
- nsmallest

I'm using the same algorithms as CPython.

BTW, concerning itertools, product and combinations_with_replacement
are implemented too.

Best regards,

--
Jérémie

[Mark Dufour]

haha, great, thanks :D I will try to have a look today or tomorrow (not much access to a computer at the moment..)

mark.

2009/12/26 Jérémie Roquet <arka...@gmail.com>

--
Jérémie

--

You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

[Mark Dufour]

the heapq implementation looks quite good, so I committed it to SVN. I only changed the type model a bit, to model the flow of 'item' arguments into the heap. I will update the LICENSE and 'thank you' section later, before the next release..

thanks again :D

[Jérémie Roquet]

2009/12/27 Mark Dufour <mark....@gmail.com>:

> the heapq implementation looks quite good, so I committed it to SVN. I only
> changed the type model a bit, to model the flow of 'item' arguments into the
> heap. I will update the LICENSE and 'thank you' section later, before the
> next release..
>
> thanks again :D

You're welcome :-)

--
Jérémie

[Mark Dufour]

BTW, concerning itertools, product and combinations_with_replacement
are implemented too.

great to hear! so iirc, that only leaves imap..? do you agree that starmap is too difficult or even impossible to support at the moment?

I'm making some good progress here as well. with a few minor improvements to the type inference engine, I already got the (almost) original matching algorithm to compile.. that means it probably won't be necessary to make any large changes, and we should be able to release 0.3 in a few weeks.

[Jérémie Roquet]

2009/12/27 Mark Dufour <mark....@gmail.com>:

>> BTW, concerning itertools, product and combinations_with_replacement
>> are implemented too.
> great to hear! so iirc, that only leaves imap..?

imap, plus :
- a small improvement to izip to (at least) support different
iterables types when there are only two iterables (since this is
already supported by tuples)
- some improvements to "lazyfiy" the combinatoric generators so that
they support infinite iterators as input

I think that's all ;-)
We also have full support for islice, thanks to the special case you
added to cpp.py and the last modifications I've done to it.

> do you agree that starmap
> is too difficult or even impossible to support at the moment?

I'm not sure...
The major challenge with both imap and starmap is to call a function
with parameters that are only known at runtime. I think it is at least
possible to support some hard-coded numbers of parameters ; then we'll
have to find a way to put the parameters on the stack "manually", but
I fear there is no portable way to do this... :s

> I'm making some good progress here as well. with a few minor improvements to
> the type inference engine, I already got the (almost) original matching
> algorithm to compile.. that means it probably won't be necessary to make any
> large changes, and we should be able to release 0.3 in a few weeks.

Great :-)

Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the
functions that are defined but not called, even if they are passed as
parameter to some other function.

Best regards,

--
Jérémie

[Jérémie Roquet]

2009/12/27 Jérémie Roquet <arka...@gmail.com>:

> Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the
> functions that are defined but not called, even if they are passed as
> parameter to some other function.

Attached trivial test case: shedskin generates code that does not compile.

Best regards,

--
Jérémie

[Mark Dufour]

I'm not sure...
The major challenge with both imap and starmap is to call a function
with parameters that are only known at runtime. I think it is at least
possible to support some hard-coded numbers of parameters ; then we'll
have to find a way to put the parameters on the stack "manually", but
I fear there is no portable way to do this... :s

I suggest we skip starmap for now, and hardcode imap up to several arguments. starmap is probably hardly ever used (I think?), and imap will usually receive only a few arguments, so practically all use cases of itertools should be covered already..

Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the
functions that are defined but not called, even if they are passed as
parameter to some other function.

I've never really considered it to be a very useful feature to be able to pass around functions, without ever using them, not even in theory.. :) but I guess we could support this relatively easily..

thanks!

[Jérémie Roquet]

2009/12/28 Mark Dufour <mark....@gmail.com>:

>> Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the
>> functions that are defined but not called, even if they are passed as
>> parameter to some other function.
> I've never really considered it to be a very useful feature to be able to
> pass around functions, without ever using them, not even in theory.. :) but
> I guess we could support this relatively easily..

Ok, I understand, I just forgot to call the predicates in the python
stubs. It works now ;-)

Thanks,

--
Jérémie

[Jérémie Roquet]

2009/12/27 Jérémie Roquet <arka...@gmail.com>:

>  - some improvements to "lazyfiy" the combinatoric generators so that
> they support infinite iterators as input

Well, actually this in not possible, and CPython does not do that either.

The following code in CPython causes an infinite loop, eating all the
available memory :

for i in itertools.combinations(itertools.count(), r = 2):
break

Conclusion: Let's code a Haskell compiler, now ;-)

--
Jérémie

[Mark Dufour]

Conclusion: Let's code a Haskell compiler, now ;-)

good idea, that would be much easier than shed skin.. :-)