[Python-Dev] PEP 371 Discussion (pyProcessing Module)
Jesse Noller
http://www.python.org/dev/peps/pep-0371/
This PEP includes some of the previous discussion on the processing
module's inclusion, and I hope clears up/clarifies some of the
goals/non goals and issues. I also included benchmark data and a link
to the code used for said benchmarks.
I would like to renew the discussion now that "there is a PEP" to see
if there are any outstanding things people would like to get resolved.
I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
both here and elsewhere that people would rather see this in sooner in
some form, rather than later (i.e.: 2.7/3.1).
-Jesse
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-dev2-garchive-22421%40googlegroups.com
Nick Coghlan
> Georg kindly published the PEP I submitted last night to the PEP site:
>
> http://www.python.org/dev/peps/pep-0371/
>
> This PEP includes some of the previous discussion on the processing
> module's inclusion, and I hope clears up/clarifies some of the
> goals/non goals and issues. I also included benchmark data and a link
> to the code used for said benchmarks.
>
> I would like to renew the discussion now that "there is a PEP" to see
> if there are any outstanding things people would like to get resolved.
> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
> both here and elsewhere that people would rather see this in sooner in
> some form, rather than later (i.e.: 2.7/3.1).
+1 from me (under the 'multiprocessing' name, with the understanding
that some code duplication with other parts of the standard library may
still remain in 2.6/3.0).
From a non-web programmer's point of view, it seems like even more of a
gain than standard library support for JSON ;)
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
http://www.boredomandlaziness.org
Michael Foord
> Jesse Noller wrote:
> > Georg kindly published the PEP I submitted last night to the PEP site:
> >
> > http://www.python.org/dev/peps/pep-0371/
> >
> > This PEP includes some of the previous discussion on the processing
> > module's inclusion, and I hope clears up/clarifies some of the
> > goals/non goals and issues. I also included benchmark data and a link
> > to the code used for said benchmarks.
> >
> > I would like to renew the discussion now that "there is a PEP" to see
> > if there are any outstanding things people would like to get resolved.
> > I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
> > both here and elsewhere that people would rather see this in sooner in
> > some form, rather than later (i.e.: 2.7/3.1).
>
> +1 from me (under the 'multiprocessing' name, with the understanding
> that some code duplication with other parts of the standard library
> may still remain in 2.6/3.0).
+1 from me as well.
I think multiple-processes is over played as a concurrency solution in
Python (where you need to marshal lots of data in and out, the overheads
of multiple processes can be very expensive) - but it is a very good
solution for some problems.
Michael Foord
>
> From a non-web programmer's point of view, it seems like even more of
> a gain than standard library support for JSON ;)
>
> Cheers,
> Nick.
>
--
http://www.ironpythoninaction.com/
http://www.theotherdelia.co.uk/
http://www.voidspace.org.uk/
http://www.ironpython.info/
http://www.resolverhacks.net/
Jesse Noller
<fuzz...@voidspace.org.uk> wrote:
> Nick Coghlan wrote:
>>
>> Jesse Noller wrote:
>> > Georg kindly published the PEP I submitted last night to the PEP site:
>> >
>> > http://www.python.org/dev/peps/pep-0371/
>> >
>> > This PEP includes some of the previous discussion on the processing
>> > module's inclusion, and I hope clears up/clarifies some of the
>> > goals/non goals and issues. I also included benchmark data and a link
>> > to the code used for said benchmarks.
>> >
>> > I would like to renew the discussion now that "there is a PEP" to see
>> > if there are any outstanding things people would like to get resolved.
>> > I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>> > both here and elsewhere that people would rather see this in sooner in
>> > some form, rather than later (i.e.: 2.7/3.1).
>>
>> +1 from me (under the 'multiprocessing' name, with the understanding that
>> some code duplication with other parts of the standard library may still
>> remain in 2.6/3.0).
>
> +1 from me as well.
>
> I think multiple-processes is over played as a concurrency solution in
> Python (where you need to marshal lots of data in and out, the overheads of
> multiple processes can be very expensive) - but it is a very good solution
> for some problems.
>
> Michael Foord
Agreed - this is a "step" rather than the final solution. As I pointed
out in the PEP this is a method to side-step GIL limitations rather
than to address the larger "GIL issue", I am implicitly assuming that
no movement will be made on that front until the bulk of Adam Olsen's
safethreading work is rolled into future versions.
-jesse
Jesus Cea
Hash: SHA1
Could you possibly extend the PEP to also document performance when, for
instance, passing "big" objects via a queue, or sending "Events" back,
testing "thread.isAlive()", and stuff like that?. What about mutexes?
(not to protect shared objects, but files, for example).
A share-nothing without data-passing doesn't need a new module :). I'm
interesting in an almost direct conversion from thread module, and so
I'm interested in knowing performance data outside "pyprocessing" sweet
point (that is, "fire and forget" code, with little communication).
How is implemented "thread.setDaemon()"?.
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jc...@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jc...@jabber.org _/_/ _/_/ _/_/_/_/_/
~ _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSD6x9plgi5GaxT1NAQI4DQP/dJWp2Mfj8e5M1zMM7JNJFFyBMRAxwagd
8EAS1Kbbl4sH8mhaucOwBntj6uXi55HOXo/+bcPD8OuMGq1gouIo7ZJvn3+SNoRM
OKsL9Judks8GAZmSF40COAziqQ/Y9M73tOoRBHtqbSTFvexbeKbfY4DvfE+/erss
g+oCjUi76FY=
=erA0
-----END PGP SIGNATURE-----
Jesse Noller
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Could you possibly extend the PEP to also document performance when, for
> instance, passing "big" objects via a queue, or sending "Events" back,
> testing "thread.isAlive()", and stuff like that?. What about mutexes?
> (not to protect shared objects, but files, for example).
>
> A share-nothing without data-passing doesn't need a new module :). I'm
> interesting in an almost direct conversion from thread module, and so
> I'm interested in knowing performance data outside "pyprocessing" sweet
> point (that is, "fire and forget" code, with little communication).
>
> How is implemented "thread.setDaemon()"?.
Alec Thomas sent me a bit of code to benchmark Queue-based object
passing performance which I will incorporate when I get a chance. As
for the provided examples/benchmarks - I can work on adding more, or
if you want - as linked in the PEP, Oudkerk already has some of those
outlined in a benchmark script here:
http://pyprocessing.berlios.de/examples/benchmarks.py
I chose not to recreate his tests directly, rather I chose to link to
them. I will work on adding Queue-based numbers. I also wouldn't say I
picked the "sweet spot" for the module - rather I picked the poor-spot
for the threading module (parallel, python-based crunching).
I do again want to point out that the goal is not to pick on
threading, but to offer an API which mimics the existing threading API
that allows for actual multi-processor/core usage.
-jesse
Guido van Rossum
> Jesse Noller wrote:
>> Georg kindly published the PEP I submitted last night to the PEP site:
>>
>> http://www.python.org/dev/peps/pep-0371/
>>
>> This PEP includes some of the previous discussion on the processing
>> module's inclusion, and I hope clears up/clarifies some of the
>> goals/non goals and issues. I also included benchmark data and a link
>> to the code used for said benchmarks.
>>
>> I would like to renew the discussion now that "there is a PEP" to see
>> if there are any outstanding things people would like to get resolved.
>> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>> both here and elsewhere that people would rather see this in sooner in
>> some form, rather than later (i.e.: 2.7/3.1).
>
> +1 from me (under the 'multiprocessing' name, with the understanding that
> some code duplication with other parts of the standard library may still
> remain in 2.6/3.0).
+1 from me as well, and I support the rename to multiprocessing
("processing" is just too much of a common word).
This will provide a good short-term answer to those who still clamor
for dropping the GIL.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Raymond Hettinger
>>> if there are any outstanding things people would like to get resolved.
>>> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>>> both here and elsewhere that people would rather see this in sooner in
>>> some form, rather than later (i.e.: 2.7/3.1).
>>
>> +1 from me (under the 'multiprocessing' name, with the understanding that
>> some code duplication with other parts of the standard library may still
>> remain in 2.6/3.0).
>
> +1 from me as well, and I support the rename to multiprocessing
> ("processing" is just too much of a common word).
>
> This will provide a good short-term answer to those who still clamor
> for dropping the GIL.
I'm very happy this is going forward. It's a big win.
Raymond
Farshid Lashkari
> I would like to renew the discussion now that "there is a PEP" to see
> if there are any outstanding things people would like to get resolved.
> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
> both here and elsewhere that people would rather see this in sooner in
> some form, rather than later (i.e.: 2.7/3.1).
I'm not sure if this is a big issue, but I just tested the module with
an application that embeds Python and it doesn't work properly.
Instead of spawning worker threads using python.exe, it attempts to
use the application. Does the processing module allow specifying the
exe to use for spawning worker threads? I would definitely like to see
this included in the next release, and having support for embedded
Python interpreters would be a big plus.
-Farshid
Jesse Noller
> On Wed, May 28, 2008 at 10:03 AM, Jesse Noller <jno...@gmail.com> wrote:
>> I would like to renew the discussion now that "there is a PEP" to see
>> if there are any outstanding things people would like to get resolved.
>> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>> both here and elsewhere that people would rather see this in sooner in
>> some form, rather than later (i.e.: 2.7/3.1).
>
> I'm not sure if this is a big issue, but I just tested the module with
> an application that embeds Python and it doesn't work properly.
> Instead of spawning worker threads using python.exe, it attempts to
> use the application. Does the processing module allow specifying the
> exe to use for spawning worker threads? I would definitely like to see
> this included in the next release, and having support for embedded
> Python interpreters would be a big plus.
>
> -Farshid
>
Let's take this off-list Farshid - but I would like to know more about
your application (I'm not up to speed with embedding python in other
applications) so I can either add it to the PEP or discuss how best to
address this within the module itself.
-jesse
Thomas Heller
> On Fri, May 30, 2008 at 2:19 AM, Farshid Lashkari <fla...@gmail.com> wrote:
>> On Wed, May 28, 2008 at 10:03 AM, Jesse Noller <jno...@gmail.com> wrote:
>>> I would like to renew the discussion now that "there is a PEP" to see
>>> if there are any outstanding things people would like to get resolved.
>>> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>>> both here and elsewhere that people would rather see this in sooner in
>>> some form, rather than later (i.e.: 2.7/3.1).
>>
>> I'm not sure if this is a big issue, but I just tested the module with
>> an application that embeds Python and it doesn't work properly.
>> Instead of spawning worker threads using python.exe, it attempts to
>> use the application. Does the processing module allow specifying the
>> exe to use for spawning worker threads? I would definitely like to see
>> this included in the next release, and having support for embedded
>> Python interpreters would be a big plus.
>>
>> -Farshid
>>
>
> Let's take this off-list Farshid - but I would like to know more about
> your application (I'm not up to speed with embedding python in other
> applications) so I can either add it to the PEP or discuss how best to
> address this within the module itself.
This may be an issue for py2exe users as well. Please keep me informed (or
is there a pyprocessing mailing list where this is discussed?).
Thanks,
Thomas
Nick Coghlan
> Jesse Noller schrieb:
>> On Fri, May 30, 2008 at 2:19 AM, Farshid Lashkari <fla...@gmail.com> wrote:
>>> On Wed, May 28, 2008 at 10:03 AM, Jesse Noller <jno...@gmail.com> wrote:
>>>> I would like to renew the discussion now that "there is a PEP" to see
>>>> if there are any outstanding things people would like to get resolved.
>>>> I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback
>>>> both here and elsewhere that people would rather see this in sooner in
>>>> some form, rather than later (i.e.: 2.7/3.1).
>>> I'm not sure if this is a big issue, but I just tested the module with
>>> an application that embeds Python and it doesn't work properly.
>>> Instead of spawning worker threads using python.exe, it attempts to
>>> use the application. Does the processing module allow specifying the
>>> exe to use for spawning worker threads? I would definitely like to see
>>> this included in the next release, and having support for embedded
>>> Python interpreters would be a big plus.
>>>
>>> -Farshid
>>>
>> Let's take this off-list Farshid - but I would like to know more about
>> your application (I'm not up to speed with embedding python in other
>> applications) so I can either add it to the PEP or discuss how best to
>> address this within the module itself.
>
> This may be an issue for py2exe users as well. Please keep me informed (or
> is there a pyprocessing mailing list where this is discussed?).
py2exe is explicitly supported (via the freezeSupport() call). That may
work for the embedded case as well, or it may be something that can be
addressed by modifying sys.executable.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
http://www.boredomandlaziness.org
Farshid Lashkari
> py2exe is explicitly supported (via the freezeSupport() call). That may work
> for the embedded case as well, or it may be something that can be addressed
> by modifying sys.executable.
Thanks for the tip Nick. Adding the following line before using the
module works for me:
sys.executable = os.path.join(sys.exec_prefix,'pythonw.exe')
I'm not sure if there will be any side affects to modifying
sys.executable though. Should this be the official way of supporting
embedded interpreters or should there be a
multiprocessing.setExecutable() method?
-Farshid
Paul Moore
> I'm not sure if there will be any side affects to modifying
> sys.executable though. Should this be the official way of supporting
> embedded interpreters or should there be a
> multiprocessing.setExecutable() method?
+1 for setExecutable (I'd prefer set_executable, to be PEP 8
compliant). Make it explicit, rather than fiddling with stuff in sys
manually.
Paul.
r.m.oudkerk
> 2008/5/30 Farshid Lashkari <fla...@gmail.com>:
>> I'm not sure if there will be any side affects to modifying
>> sys.executable though. Should this be the official way of supporting
>> embedded interpreters or should there be a
>> multiprocessing.setExecutable() method?
>
> +1 for setExecutable (I'd prefer set_executable, to be PEP 8
> compliant). Make it explicit, rather than fiddling with stuff in sys
> manually.
That is easy to do.
An issue not mentioned in the PEP is naming conventions. In recent
versions I have tried to consistently use mixedCase for functions and
methods (other than factory functions) because that is what threading
does (give or take settrace(), setprofile(), stack_size().)
I am certainly open to using lowercase/lower_case_with_underscores for
all functions/methods except for Process's methods and possibly
currentProcess(), but I would like some feed back on that.
Mark Hammond
> > I'm not sure if there will be any side affects to modifying
> > sys.executable though. Should this be the official way of supporting
> > embedded interpreters or should there be a
> > multiprocessing.setExecutable() method?
>
> +1 for setExecutable (I'd prefer set_executable, to be PEP 8
> compliant). Make it explicit, rather than fiddling with stuff in sys
> manually.
sys.executable typically means the "current" executable, and py2exe etc
already fiddles with that. The question in such a context seems to be "what
executable should I use for this processing functionality?". In a py2exe
like environment, it might not be unreasonable to assume that if a custom
executable is to be used, that custom executable may have a different
command-line or other special requirements. Further, I could imagine a
system that uses an alternative way of starting processes (eg, 'distributed
COM') where the concept of 'executable' (or even 'command-line') don't make
much sense.
So it seems that maybe simply "setExecutable()" isn't the correct
abstraction here, but maybe a "factory" approach, so the entire process
creation mechanism can be replaced rather than just the name of the
executable to spawn?
Cheers,
Mark
Paul Moore
> I am certainly open to using lowercase/lower_case_with_underscores for
> all functions/methods except for Process's methods and possibly
> currentProcess(), but I would like some feed back on that.
I dislike mixedCase, but consistency with the rest of the library is
more important - and as processing is matching the API of threading,
which used mixedCase, it should follow that convention.
Wasn't there some talk of changing modules to use PEP 8 conventions
(lowercase_with_underscore) as part of the Python 3.0 conversion? Did
that ever happen?
Paul.
Nick Coghlan
> Wasn't there some talk of changing modules to use PEP 8 conventions
> (lowercase_with_underscore) as part of the Python 3.0 conversion? Did
> that ever happen?
We fixed the module names that used mixed case - the amount of work that
turned out to be involved in just doing that much for PEP 3108 makes me
shudder at the thought of trying to fix all of the standard library APIs
that currently don't follow the style guide...
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
http://www.boredomandlaziness.org
sk...@pobox.com
>> If the 3.0 API of a module is going to involve breakage which
>> requires authors to update their applications wouldn't this be a good
>> time to PEP-8-ify the module? (Not suggesting that threading would
>> fall into this category.)
Nick> Updating application code to deal with a module name change is
Nick> easy - just use an "import x as y" statement. Catching the
Nick> ImportError for the 3.0 name and falling back to the 2.6 name (or
Nick> vice-versa) even makes it possible to support both names fairly
Nick> easily.
Nick> Changing the names of actual objects within the modules is tougher
Nick> though - there are many more ways to access those.
I think you misunderstood what I wrote. Suppose you decided that the API
for the threading modules needs significant rework, changes which will break
much, if not all current usage. I'm only suggesting that if you decide the
API change is worthwhile that you take the opportunity to make the class,
method and data names PEP8-compliant. I'm not talking about gratuitous
change to the identifier names, but situations where the user is going to
have to rework their code anyway.
Skip
Jesse Noller
> On 31/05/2008, Paul Moore <p.f....@gmail.com> wrote:
>> 2008/5/30 Farshid Lashkari <fla...@gmail.com>:
>>> I'm not sure if there will be any side affects to modifying
>>> sys.executable though. Should this be the official way of supporting
>>> embedded interpreters or should there be a
>>> multiprocessing.setExecutable() method?
>>
>> +1 for setExecutable (I'd prefer set_executable, to be PEP 8
>> compliant). Make it explicit, rather than fiddling with stuff in sys
>> manually.
>
> That is easy to do.
Also note - someone just pointed out to me that the executable
manipulation as-is breaks when you execute things within IDLE.
I'll add all of this to the PEP.
-jesse
Nick Coghlan
> Nick> We fixed the module names that used mixed case - the amount of
> Nick> work that turned out to be involved in just doing that much for
> Nick> PEP 3108 makes me shudder at the thought of trying to fix all of
> Nick> the standard library APIs that currently don't follow the style
> Nick> guide...
>
> If the 3.0 API of a module is going to involve breakage which requires
> authors to update their applications wouldn't this be a good time to
> PEP-8-ify the module? (Not suggesting that threading would fall into this
> category.)
Updating application code to deal with a module name change is easy -
just use an "import x as y" statement. Catching the ImportError for the
3.0 name and falling back to the 2.6 name (or vice-versa) even makes it
possible to support both names fairly easily.
Changing the names of actual objects within the modules is tougher
though - there are many more ways to access those.
Cheers,
Nick
Lisandro Dalcin
> So it seems that maybe simply "setExecutable()" isn't the correct
> abstraction here, but maybe a "factory" approach, so the entire process
> creation mechanism can be replaced rather than just the name of the
> executable to spawn?
Indeed. If the spawn mechanisms (and even the connection mechanisms)
were fully abstracted, then I believe that extending pyProcessing to
work in clusters with something like MPI would be far easier. But
perhaps I'm just dreaming...
--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
sk...@pobox.com
Nick> We fixed the module names that used mixed case - the amount of
Nick> work that turned out to be involved in just doing that much for
Nick> PEP 3108 makes me shudder at the thought of trying to fix all of
Nick> the standard library APIs that currently don't follow the style
Nick> guide...
If the 3.0 API of a module is going to involve breakage which requires
authors to update their applications wouldn't this be a good time to
PEP-8-ify the module? (Not suggesting that threading would fall into this
category.)
Skip