Possible merge with PyCSP
snim2
to outline what we are blocked on with respect to a possible merge
with PyCSP. Hopefully Rune and others from that group can contribute
to this discussion. In the medium term I do think a merge would be a
big boost to both projects, and being able to share the effort of
development makes a huge amount of sense. So, here is the current
state of play as I see it:
* License -- python-csp is GPL2 and PyCSP is under the MIT license. We
need to decide which license the merged project should take, and
relicense some code before merging.
* How channels work. In python-csp a Channel object is a direct
reference to a channel and Alt.select() returns the result of a
channel read. In PyCSP this works more like the Java and other
libraries; usually processes have access to an "end" of a channel and
select() returns a channel end or a reference to a one (I think).
Personally I think this is less "Pythonic" but YMMV. Either way some
considerable work will need to be done to rewrite example code and so
on in one of the libraries.
* Channel retirement (a version of poisoning) and ALTing on channel
writes are two features from PyCSP that we don't have. There are no
doubt others. Most of these I think we should just keep, where the two
libraries do not intersect.
* Channel and Barrier synchronisation works completely differently in
the two libraries. PyCSP synchronisation is based on JCSP which has
been "proven correct" by the Kent group and the python-csp
synchronisation has been model checked by Mohammad. We need some way
to resolve the differences between the approaches because they are
very different and any changes will create very large changesets in
the respositoriies. This is probably the most difficult issue to deal
with.
* python-csp uses Mercurial, PyCSP uses SVN.
* PyCSP is Windows compatible. python-csp should be (because it's
built on multiprocessing) but this hasn't been well tested. We now
know, thanks to Russel and others at the Europython sprint, that MacOS
compatibility is difficult.
* Branches. I'd like to keep Sam's work on the Jython port, but it
needs some major work to port all the more recent changes in the
python version back to the Jython code. The Jython port is useful for
two reasons, one is that some libraries are written in Java and have
no Python bindings and we can still use them with CSP! The second is
that Guido is keen that anything that goes into the standard library
(which is one thing we might aim for) should not be CPython only, but
be portable to other VMs. It's good to have a proof of concept for
this.
* Separate work -- both projects are research projects, I would guess
we want a structure for branches that allows us to work on separate
projects, like this:
/-Main dev branch, gets merged with stable when new features are added
- stable dev branch for releases and bugfixes
- Jython port
... separate branches for new features / research projects
* Longer term things...
+ It would be nice to add some support for Actors and dataflow
programming
+ It would be nice to add support for CUDA programming and some cloud
architectures
+ It would be nice to move channels to a much faster C implementation.
Sam has started some work on this, but it needs finishing. Also, it
would be good to think a bit about the best model for this; Richard
Taylor had some good ideas for this.
There are probably other issues I should have considered, but
hopefully we can at least start the discussion going...
Thanks,
Sarah
Rune M. Friborg
Kudos to Sarah for the initiative to bring this discussion out in the
open.
I have added a few comments, and mainly on the functionality of
PyCSP.
When doing a merge of pycsp and python-csp, I think we should focus on
creating a minimal engine
that handles all the nasty details of process creation, communication,
poisoning and
retiring.
Many other functionalities can be built using this minimal engine and
thus provide
different interfaces and functionalities. This might be, tracing,
CUDA,
dataflow programming, framework stuff, pre-made processes, Grid
interfaces and so on.
> * Separate work -- both projects are research projects, I would guess
> we want a structure for branches that allows us to work on separate
> projects
This is all a very good reason to sit down over a beer.
When we arrive at a more detailed plan,
we should probably meet, unfortunately CPA is just too far away.
Remember to read the comments below...
Thanks,
Rune
On Jul 29, 12:50 pm, snim2 <mount.sa...@gmail.com> wrote:
> * How channels work. In python-csp a Channel object is a direct
> reference to a channel and Alt.select() returns the result of a
> channel read. In PyCSP this works more like the Java and other
> libraries; usually processes have access to an "end" of a channel and
> select() returns a channel end or a reference to a one (I think).
> Personally I think this is less "Pythonic" but YMMV. Either way some
> considerable work will need to be done to rewrite example code and so
> on in one of the libraries.
AltSelect(..).
So, no difference there.
>
> * Channel retirement (a version of poisoning) and ALTing on channel
> writes are two features from PyCSP that we don't have. There are no
> doubt others. Most of these I think we should just keep, where the two
> libraries do not intersect.
ends. This was our
main reason for introducing the channel ends.
>
> * Channel and Barrier synchronisation works completely differently in
> the two libraries. PyCSP synchronisation is based on JCSP which has
> been "proven correct" by the Kent group and the python-csp
> synchronisation has been model checked by Mohammad. We need some way
> to resolve the differences between the approaches because they are
> very different and any changes will create very large changesets in
> the respositoriies. This is probably the most difficult issue to deal
> with.
own
synchronisation model that has proven very robust.
It is partially documented in http://pycsp.googlecode.com/files/paper-02.pdf
We have not yet done a formal verification of the synchronisation
model.
I do not believe that the synchronization models can be merged, but
instead
we will have to decide on one or the other, which will be the basis
for everything.
>
> * python-csp uses Mercurial, PyCSP uses SVN.
>
> * PyCSP is Windows compatible. python-csp should be (because it's
> built on multiprocessing) but this hasn't been well tested. We now
> know, thanks to Russel and others at the Europython sprint, that MacOS
> compatibility is difficult.
always run on both linux and MacOSX without any differences. This is
correct for
all 4 implementations (threads, processes, greenlets and net)
PyCSP is "windows" compatible.. it works, for some cases.
This week I got an interesting result from PyCSP and jython. Until
know PyCSP has
not been jython compatible, since we took advantage of the fact that
the GIL made list
operations atomic. Having protected these with a channel lock, we do
now support jython.
The mixture Commstime.py, pycsp.threads, Jython and sun-java has a
long startup delay,
but when running it delivers ~50 us results on a 2.4 Ghz Intel Core 2
duo (MacOSX)
Notice that pycsp.threads supports both CPython and Jython, thus no
extra Jython port is needed.
>
> * Branches. I'd like to keep Sam's work on the Jython port, but it
> needs some major work to port all the more recent changes in the
> python version back to the Jython code. The Jython port is useful for
> two reasons, one is that some libraries are written in Java and have
> no Python bindings and we can still use them with CSP! The second is
> that Guido is keen that anything that goes into the standard library
> (which is one thing we might aim for) should not be CPython only, but
> be portable to other VMs. It's good to have a proof of concept for
> this.
>
> * Separate work -- both projects are research projects, I would guess
> we want a structure for branches that allows us to work on separate
> projects, like this:
>
> /-Main dev branch, gets merged with stable when new features are added
> - stable dev branch for releases and bugfixes
> - Jython port
> ... separate branches for new features / research projects
>
Sarah Mount
>
> Kudos to Sarah for the initiative to bring this discussion out in the
> open.
:)
> When doing a merge of pycsp and python-csp, I think we should focus on
> creating a minimal engine
> that handles all the nasty details of process creation, communication,
> poisoning and
> retiring.
>
> Many other functionalities can be built using this minimal engine and
> thus provide
> different interfaces and functionalities. This might be, tracing,
> CUDA,
> dataflow programming, framework stuff, pre-made processes, Grid
> interfaces and so on.
Agreed.
> This is all a very good reason to sit down over a beer.
> When we arrive at a more detailed plan,
> we should probably meet, unfortunately CPA is just too far away.
Yes, I'm not sure quite how best to make this happen. It would be good
to have Mohammad there to talk about the model checking and Russel,
maybe Sam if he's interested any any others. I notice the PSF now have
sponsored sprints, I'm not sure if this is an appropriate avenue for
us.
> On Jul 29, 12:50 pm, snim2 <mount.sa...@gmail.com> wrote:
>> * How channels work. In python-csp a Channel object is a direct
>> reference to a channel and Alt.select() returns the result of a
>> channel read. In PyCSP this works more like the Java and other
>> libraries; usually processes have access to an "end" of a channel and
>> select() returns a channel end or a reference to a one (I think).
>> Personally I think this is less "Pythonic" but YMMV. Either way some
>> considerable work will need to be done to rewrite example code and so
>> on in one of the libraries.
>
> Since 2009 PyCSP has performed the communication inside the call to
> AltSelect(..).
> So, no difference there.
Even better.
>> * Channel retirement (a version of poisoning) and ALTing on channel
>> writes are two features from PyCSP that we don't have. There are no
>> doubt others. Most of these I think we should just keep, where the two
>> libraries do not intersect.
> It will be difficult to implement channel retirement without channel
> ends. This was our
> main reason for introducing the channel ends.
>
OK, this does need some careful thought then. Is the CPA32 paper the
best description of channel retirement?
>> * Channel and Barrier synchronisation works completely differently in
>> the two libraries. PyCSP synchronisation is based on JCSP which has
>> been "proven correct" by the Kent group and the python-csp
>> synchronisation has been model checked by Mohammad. We need some way
>> to resolve the differences between the approaches because they are
>> very different and any changes will create very large changesets in
>> the respositoriies. This is probably the most difficult issue to deal
>> with.
> PyCSP was originally based on JCSP, but has since 2009 used it's very
> own
> synchronisation model that has proven very robust.
> It is partially documented in http://pycsp.googlecode.com/files/paper-02.pdf
>
> We have not yet done a formal verification of the synchronisation
> model.
OK, this is a good thing to do!
> I do not believe that the synchronization models can be merged, but
> instead
> we will have to decide on one or the other, which will be the basis
> for everything.
>
No, I agree, there's no chance at all of merging this code. That said,
I'm not sure what the best way to compare the two would be.
Benchmarking both very carefully would be a start (in my timings our
code came out slightly quicker, but I'd be hesitant to take that as
definitive). That said, correctness is of course more important than
speed.
>> * python-csp uses Mercurial, PyCSP uses SVN.
You haven't mentioned this; personally I do still think Mercurial is a
big win over SVN.
>> * PyCSP is Windows compatible. python-csp should be (because it's
>> built on multiprocessing) but this hasn't been well tested. We now
>> know, thanks to Russel and others at the Europython sprint, that MacOS
>> compatibility is difficult.
> I am not sure on what you mean by MacOS compatibility, since PyCSP has
> always run on both linux and MacOSX without any differences. This is
> correct for
> all 4 implementations (threads, processes, greenlets and net)
Interesting, we had some problems with multiprocessing on MacOSX.
> PyCSP is "windows" compatible.. it works, for some cases.
!!
> This week I got an interesting result from PyCSP and jython. Until
> know PyCSP has
> not been jython compatible, since we took advantage of the fact that
> the GIL made list
> operations atomic. Having protected these with a channel lock, we do
> now support jython.
>
> The mixture Commstime.py, pycsp.threads, Jython and sun-java has a
> long startup delay,
> but when running it delivers ~50 us results on a 2.4 Ghz Intel Core 2
> duo (MacOSX)
Interesting, we got roughly the same performance on CPython and Jython
for Commstime.
> Notice that pycsp.threads supports both CPython and Jython, thus no
> extra Jython port is needed.
Interesting. We have a single script called csp/csp.py which imports
the "best" implementation of python-csp it can, there's been some
debate on this, but I think it makes some sense to have a single
sensible way to import the core library then let people import
specific implementations if they wish to. We've had some debate on how
best to go about that though :)
>> * Branches. I'd like to keep Sam's work on the Jython port, but it
>> needs some major work to port all the more recent changes in the
>> python version back to the Jython code. The Jython port is useful for
>> two reasons, one is that some libraries are written in Java and have
>> no Python bindings and we can still use them with CSP! The second is
>> that Guido is keen that anything that goes into the standard library
>> (which is one thing we might aim for) should not be CPython only, but
>> be portable to other VMs. It's good to have a proof of concept for
>> this.
>>
>> * Separate work -- both projects are research projects, I would guess
>> we want a structure for branches that allows us to work on separate
>> projects, like this:
>>
>> /-Main dev branch, gets merged with stable when new features are added
>> - stable dev branch for releases and bugfixes
>> - Jython port
>> ... separate branches for new features / research projects
>>
>
> What are your reason for having a separate Jython port?
>
We used Java threads rather than CPython threads, expecting the former
to be faster (they weren't as far as we could tell). Sam's C channels
look to be about as fast as JCSP channels when we benchmarked them,
but the current implementation still has some bugs so we haven't
merged that work into the core library just yet, and it would of
course need a specific Windows port since it uses the POSIX standard
(and everyone knows that the good thing about standards is that
there's so many to choose from!).
Cheers,
Sarah
--
Sarah Mount, Senior Lecturer, University of Wolverhampton
website: http://www.snim2.org/
twitter: @snim2
This email, together with any attachment, is for the exclusive and
confidential use of the addressee(s) and may contain legally
privileged information. Any use, disclosure or reproduction without
the sender's explicit consent is unauthorised and may be unlawful.
Any e-mail including its content and any attachments may be monitored
and used by The University of Wolverhampton for reasons of security
and for monitoring internal compliance with the University's policy on
internet use. E-mail blocking software may also be used. The
University cannot guarantee that this message or any attachment is
virus free or has not been intercepted and amended.
If you believe you have received this message in error please notify
the sender by email, telephone or fax and destroy the message and any
copies.
Stefan Schwarzer
On 2010-07-29 17:45, Sarah Mount wrote:
> On 07/29/2010, Rune M. Friborg <runed...@gmail.com> wrote:
>> On Jul 29, 12:50 pm, snim2 <mount.sa...@gmail.com> wrote:
>> This is all a very good reason to sit down over a beer.
>> When we arrive at a more detailed plan,
>> we should probably meet, unfortunately CPA is just too far away.
>
> Yes, I'm not sure quite how best to make this happen. It would be good
> to have Mohammad there to talk about the model checking and Russel,
> maybe Sam if he's interested any any others. I notice the PSF now have
> sponsored sprints, I'm not sure if this is an appropriate avenue for
> us.
If you want to have a rather "internal" discussion, Skype
may be an option. You can have kind of a phone conference
and at the same time interchange URLs or code snippets via
the chat window. There's also a function "screen sharing",
which sometimes can be useful, over a DSL line it's slow,
though. All this works on Linux, too. At least I've used
Skype successfully that way on Ubuntu.
I'd like it more if there was a comparable open source
project and protocol, but as far as I know, there isn't. I'd
be thankful for pointers to alternative projects if you know
any.
>> I do not believe that the synchronization models can be merged, but
>> instead
>> we will have to decide on one or the other, which will be the basis
>> for everything.
>
> No, I agree, there's no chance at all of merging this code. That said,
> I'm not sure what the best way to compare the two would be.
> Benchmarking both very carefully would be a start (in my timings our
> code came out slightly quicker, but I'd be hesitant to take that as
> definitive). That said, correctness is of course more important than
> speed.
My recommendation is to first make really sure that the code
works as intended. It's no use to find an application is
slow only to find out later it was because of a bug. Or the
opposite: an application that's very fast - because some
important part of the algorithm is never executed. ;-)
There's this great motto: "Make it work, make it right, make
it fast."
Moreover, in my opinion you can't really compare the speed
of two approaches until you profiled and seriously tried to
optimize them. Of course you might say, that by optimizing
you changed the approach. :)
By the way, here are some nice talk slides :-)
http://sschwarzer.com/download/optimization_europython2006.pdf
>>> * python-csp uses Mercurial, PyCSP uses SVN.
>
> You haven't mentioned this; personally I do still think Mercurial is a
> big win over SVN.
I agree. I've been using both for several years, and I like
Mercurial much more because it's so flexible. A distributed
system like Mercurial makes it simple to clone a project,
try out some things (including local commits) and later
offer the changesets to inclusion in the original project.
This is encouraging for to-be contributors.
>> Notice that pycsp.threads supports both CPython and Jython, thus no
>> extra Jython port is needed.
>
> Interesting. We have a single script called csp/csp.py which imports
> the "best" implementation of python-csp it can, there's been some
> debate on this, but I think it makes some sense to have a single
> sensible way to import the core library then let people import
> specific implementations if they wish to. We've had some debate on how
> best to go about that though :)
I know. :-)
>> What are your reason for having a separate Jython port?
>
> We used Java threads rather than CPython threads, expecting the former
> to be faster (they weren't as far as we could tell). Sam's C channels
> look to be about as fast as JCSP channels when we benchmarked them,
> but the current implementation still has some bugs so we haven't
> merged that work into the core library just yet, and it would of
> course need a specific Windows port since it uses the POSIX standard
> (and everyone knows that the good thing about standards is that
> there's so many to choose from!).
At this stage, I'd rather care for features, not raw speed.
If (Python) threads work on Java as well, I'd favor them.
Stefan
Rune M. Friborg
On Jul 29, 5:45 pm, Sarah Mount <mount.sa...@gmail.com> wrote:
>
> >> * Channel retirement (a version of poisoning) and ALTing on channel
> >> writes are two features from PyCSP that we don't have. There are no
> >> doubt others. Most of these I think we should just keep, where the two
> >> libraries do not intersect.
> > It will be difficult to implement channel retirement without channel
> > ends. This was our
> > main reason for introducing the channel ends.
>
> OK, this does need some careful thought then. Is the CPA32 paper the
> best description of channel retirement?
retirement.
>
> > We have not yet done a formal verification of the synchronisation
> > model.
>
> OK, this is a good thing to do!
>
> > I do not believe that the synchronization models can be merged, but
> > instead
> > we will have to decide on one or the other, which will be the basis
> > for everything.
>
> No, I agree, there's no chance at all of merging this code. That said,
> I'm not sure what the best way to compare the two would be.
> Benchmarking both very carefully would be a start (in my timings our
> code came out slightly quicker, but I'd be hesitant to take that as
> definitive). That said, correctness is of course more important than
> speed.
subjective view when comparing on simplicity and features, but I think
a
common ground could be found.
>
> >> * python-csp uses Mercurial, PyCSP uses SVN.
>
> You haven't mentioned this; personally I do still think Mercurial is a
> big win over SVN.
waste more lines on this matter.
I don't mind using Mercurial for a joint project.
> > PyCSP is "windows" compatible.. it works, for some cases.
>
> !!
has a hard time coping with the lacking support of fork() in Windows.
Most things
work, but a limitation is that one can't use the decorators for
defining processes.
>
> > Notice that pycsp.threads supports both CPython and Jython, thus no
> > extra Jython port is needed.
>
> Interesting. We have a single script called csp/csp.py which imports
> the "best" implementation of python-csp it can, there's been some
> debate on this, but I think it makes some sense to have a single
> sensible way to import the core library then let people import
> specific implementations if they wish to. We've had some debate on how
> best to go about that though :)
>
the
underlying implementation might change depending on the weather.
> > What are your reason for having a separate Jython port?
>
> We used Java threads rather than CPython threads, expecting the former
> to be faster (they weren't as far as we could tell). Sam's C channels
> look to be about as fast as JCSP channels when we benchmarked them,
> but the current implementation still has some bugs so we haven't
> merged that work into the core library just yet, and it would of
> course need a specific Windows port since it uses the POSIX standard
> (and everyone knows that the good thing about standards is that
> there's so many to choose from!).
Is the C channel implementation available in any repository?
Thanks,
Rune
snim2
On Aug 1, 12:01 pm, "Rune M. Friborg" <runedren...@gmail.com> wrote:
> On Jul 29, 5:45 pm, Sarah Mount <mount.sa...@gmail.com> wrote:
>
> > On 07/29/2010, Rune M. Friborg <runedren...@gmail.com> wrote:
>
> > > I do not believe that the synchronization models can be merged, but
> > > instead
> > > we will have to decide on one or the other, which will be the basis
> > > for everything.
>
> > No, I agree, there's no chance at all of merging this code. That said,
> > I'm not sure what the best way to compare the two would be.
> > Benchmarking both very carefully would be a start (in my timings our
> > code came out slightly quicker, but I'd be hesitant to take that as
> > definitive). That said, correctness is of course more important than
> > speed.
>
> I can add two more factors, simplicity and features. It is of course a
> subjective view when comparing on simplicity and features, but I think
> a
> common ground could be found.
>
sample programs written with both libraries and compare.
> > > PyCSP is "windows" compatible.. it works, for some cases.
>
> > !!
>
> pycsp.processes which is the multiprocessing implementation of pycsp
> has a hard time coping with the lacking support of fork() in Windows.
> Most things
> work, but a limitation is that one can't use the decorators for
> defining processes.
>
multiprocessing, if it's possible.
> > > Notice that pycsp.threads supports both CPython and Jython, thus no
> > > extra Jython port is needed.
>
> > Interesting. We have a single script called csp/csp.py which imports
> > the "best" implementation of python-csp it can, there's been some
> > debate on this, but I think it makes some sense to have a single
> > sensible way to import the core library then let people import
> > specific implementations if they wish to. We've had some debate on how
> > best to go about that though :)
>
> I would then prefer a specific, like pycsp.best, to make it clear that
> the
> underlying implementation might change depending on the weather.
PyCSP. I'll try to resolve this next week (it's a small change but one
that affects most files in our repository). I think the more small
changes we can make to eliminate differences now, the easier it will
be to make the bigger decisions later.
> > > What are your reason for having a separate Jython port?
>
> > We used Java threads rather than CPython threads, expecting the former
> > to be faster (they weren't as far as we could tell). Sam's C channels
> > look to be about as fast as JCSP channels when we benchmarked them,
> > but the current implementation still has some bugs so we haven't
> > merged that work into the core library just yet, and it would of
> > course need a specific Windows port since it uses the POSIX standard
> > (and everyone knows that the good thing about standards is that
> > there's so many to choose from!).
>
> Nice!
>
> Is the C channel implementation available in any repository?
>
http://code.google.com/p/python-csp/source/browse/#hg/csp/cchannels
I'll give you commit access to the project, also, in case you want to
play around. I think there were two major issues with the
implementation as it stands. One was that Sam found there was an issue
with the way his C channels interacted with the Python GC and that
caused a bug. Once that's fixed, an enhancement could be made; I think
Sam used one POSIX semaphore group for each semaphore (or something
similar) and this could be improved by holding several semaphores in
one group (Richard Taylor thought this would be possible, although Sam
wasn't sure about the way acquire / release works with groups). If we
can get that working it would be a big boost in speed, and probably
give us a way forward for other optimisations in the future.
Cheers,
Sarah
Rune M. Friborg
> > I can add two more factors, simplicity and features. It is of course a
> > subjective view when comparing on simplicity and features, but I think
> > a
> > common ground could be found.
>
> Agreed. I think a reasonable way to do this might be to have some
> sample programs written with both libraries and compare.
synchronization model and not the libraries.
>
> > > > PyCSP is "windows" compatible.. it works, for some cases.
>
> > > !!
>
> > pycsp.processes which is the multiprocessing implementation of pycsp
> > has a hard time coping with the lacking support of fork() in Windows.
> > Most things
> > work, but a limitation is that one can't use the decorators for
> > defining processes.
>
> Ugh. That's a pain. We should probably look to offer a patch for
> multiprocessing, if it's possible.
limitations and few advantages (one being no GIL).
> > Is the C channel implementation available in any repository?
>
> Yes, it's here:
>
> http://code.google.com/p/python-csp/source/browse/#hg/csp/cchannels
Thanks, Rune
Richard Taylor
On 07/29/2010, Rune M. Friborg <runed...@gmail.com> wrote:
>
> This is all a very good reason to sit down over a beer.Yes, I'm not sure quite how best to make this happen. It would be good
> When we arrive at a more detailed plan,
> we should probably meet, unfortunately CPA is just too far away.
to have Mohammad there to talk about the model checking and Russel,
maybe Sam if he's interested any any others. I notice the PSF now have
Fabian
I'm also opposed to the GPL and pro-MIT.
The GPL does more harm than good, and for a library it should anyway
be the L-GPL.
snim2
I've kicked off a wiki page to try and detail all of our decisions:
http://code.google.com/p/python-csp/wiki/MergerWithPyCSP
Hopefully if we keep this up to date it'll make things a little
clearer for dev's and users.
Sarah