hi all,

so here's the current plan for shedskin 0.2.1, which should be
released in about two months:

-support for itertools (thanks to the work of jeremie)
-support for csv (by eating our own dogfood, using something like
python-dsv - thanks chris)
-if we support itertools, then I guess 'map'/'filter'/'reduce' should
work too (yes, I give up my opposition :))
-as well as 'next'.. (any other useful (new) builtin functions
shedskin doesn't support at the moment?)
-and while we're passing function references around, it should be
possible to place these in containers and to pass method references
around as well
-support for the 'key' argument of 'list.sort' and 'sorted'
-passing keyword arguments to generated extension modules (done)
-boost type inference scalability by disallowing parametric
polymorphism (outside of lib/, as for data polymorphism it's just not
worth it.)

thanks,
mark.

a status update:

jeremie, I'm hoping to see another patch soon.. ^^ I'd be happy to
help out if you don't have time to fully complete it, now that most
other things on this list have been tackled. anyway, jeremie and I
sorted out most issues, and I think we can support practically
everything..

I wrote a pure Python implementation based on _csv.c (and csv.py) from
CPython, and compiled it using shedskin (see csv15.py). I decided not
to support Sniffer for now, or custom Dialects, but most basic things
should work now.

these have been added.. I also added support for passing around
builtin callables, which required a bit of surgery.. so the following
now works, for example:

print sorted([[2,3,4], [5,6], [7]], key=len)
print map(len, ['a','bc'])
print map(max, ['a','bc'], ['d'], ['e'])
print map(set, [[1]])

done.

I decided to move this to a future release, because I've seen enough
function passing for now..

done.

this is the thing I will look into next. but if I get joris' original
blossoming algorithm to work with a simpler trick, I may also move
this one forward. there's already more than enough new stuff for a new
release.

some other things were improved in the meantime:

- forward referencing of variables and functions
- joris managed to improve basic indexing (without -b, by improving
inlining of __wrap)
- addition of 1-length lists and strings was optimized
- many, many bugfixes.

if I have time, I may look into buffers/memoryviews as suggested on my
blog (thanks!), to avoid copying overhead for extension modules, but I
will probably save this for the next release..

thanks,
mark.

Hello,

2009/12/20 srepmub <mark.duf...@gmail.com>:

^^

To sum up briefly what doesn't work yet for itertools:
 - imap
 - starmap
 - product
 - combinations_with_replacement (A 2.7 addition, btw)

The following have been completed since the first patch (and thanks to
Mark's improvements to shedskin):
 - chain (accepts an arbitrary number of input iterables)
 - tee (accepts an arbitrary number of output iterables)

The following have been added:
 - izip (with some limitations: the iterables must be of the same type)
 - izip_longest (with some limitations:  the iterables must be of the
same type, and there is no way to distinguish None and 0 in the
output)
 - permutations
 - combinations

Depends on when you want the module to be complete, but I think
product and combinations_with_replacement will be available very soon
(there is no challenge there).
I haven't thought much about imap and starmap, but I'm expecting some
challenge there... so that's the last I'll work on.

Best regards,

--
Jérémie

thanks for the update, and nice to hear you are almost finished.. :) I am in
no hurry to release, as I will probably need at least two more weeks to
improve TI somewhat..

now that I look at starmap again, it looks like it may be impossible to
support it fully at the moment.. the unpacking it does seems impossible to
model, because the number of arguments it unpacks cannot be known during
type inference (in contrast to (i)map, where you can count the arguments to
(i)map..).

so maybe we can leave starmap out for now..? it doesn't look like a very
popular function in any case..

thanks,
mark
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

btw,

it may be interesting to also add support for 'heapq' before the next
release. it is quite small, and uses several itertools functions..

(see the networkx thread about compiling a graph algorithm that in
turn uses heapq..).

mark.

On 21 dec, 17:23, Mark Dufour <mark.duf...@gmail.com> wrote:

2009/12/26 srepmub <mark.duf...@gmail.com>:

I'm looking at it this afternoon (unless you've already started to
work on it, of course :p).

--
Jérémie

I'm still looking into type inference scalability at the moment, so go
ahead.. :)

thanks,
mark.
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

2009/12/26 Mark Dufour <mark.duf...@gmail.com>:

I attached a "preview" version, with the following functions :
 - heappush
 - heappop
 - heappushpop
 - heapify
 - heapreplace

I still have to implement the following :
 - merge
 - nlargest
 - nsmallest

I'm using the same algorithms as CPython.

BTW, concerning itertools, product and combinations_with_replacement
are implemented too.

Best regards,

--
Jérémie

heapq_tests.py 1K Download

heapq.cpp < 1K Download

heapq.hpp 3K Download

heapq.py < 1K Download

haha, great, thanks :D I will try to have a look today or tomorrow (not much
access to a computer at the moment..)

mark.

2009/12/26 Jérémie Roquet <arkano...@gmail.com>

--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

the heapq implementation looks quite good, so I committed it to SVN. I only
changed the type model a bit, to model the flow of 'item' arguments into the
heap. I will update the LICENSE and 'thank you' section later, before the
next release..

thanks again :D

--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

2009/12/27 Mark Dufour <mark.duf...@gmail.com>:

You're welcome :-)

--
Jérémie

great to hear! so iirc, that only leaves imap..? do you agree that starmap
is too difficult or even impossible to support at the moment?

I'm making some good progress here as well. with a few minor improvements to
the type inference engine, I already got the (almost) original matching
algorithm to compile.. that means it probably won't be necessary to make any
large changes, and we should be able to release 0.3 in a few weeks.

thanks,
mark.
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

2009/12/27 Mark Dufour <mark.duf...@gmail.com>:

imap, plus :
 - a small improvement to izip to (at least) support different
iterables types when there are only two iterables (since this is
already supported by tuples)
 - some improvements to "lazyfiy" the combinatoric generators so that
they support infinite iterators as input

I think that's all ;-)
We also have full support for islice, thanks to the special case you
added to cpp.py and the last modifications I've done to it.

I'm not sure...
The major challenge with both imap and starmap is to call a function
with parameters that are only known at runtime. I think it is at least
possible to support some hard-coded numbers of parameters ; then we'll
have to find a way to put the parameters on the stack "manually", but
I fear there is no portable way to do this... :s

Great :-)

Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the
functions that are defined but not called, even if they are passed as
parameter to some other function.

Best regards,

--
Jérémie

2009/12/27 Jérémie Roquet <arkano...@gmail.com>:

Attached trivial test case: shedskin generates code that does not compile.

Best regards,

--
Jérémie

prune.py < 1K Download

I suggest we skip starmap for now, and hardcode imap up to several
arguments. starmap is probably hardly ever used (I think?), and imap will
usually receive only a few arguments, so practically all use cases of
itertools should be covered already..

Oh, BTW, I've noticed a small bug: it looks like shedskin prunes the

I've never really considered it to be a very useful feature to be able to
pass around functions, without ever using them, not even in theory.. :) but
I guess we could support this relatively easily..

thanks!
mark.
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds

2009/12/28 Mark Dufour <mark.duf...@gmail.com>:

Ok, I understand, I just forgot to call the predicates in the python
stubs. It works now ;-)

Thanks,

--
Jérémie

2009/12/27 Jérémie Roquet <arkano...@gmail.com>:

Well, actually this in not possible, and CPython does not do that either.

The following code in CPython causes an infinite loop, eating all the
available memory :

for i in itertools.combinations(itertools.count(), r = 2):
    break

Conclusion: Let's code a Haskell compiler, now ;-)

--
Jérémie

good idea, that would be much easier than shed skin.. :-)

mark.
--
"Overdesigning is a SIN. It's the archetypal example of what I call 'bad
taste'" - Linus Torvalds