We have an ongoing group discussion within my organization about how
to unify the various python distributions we're using for the
supercomputers. Are most mpi4py users building their own python
interpreters and installing mpi4py into it on production HPC systems
or is their some consensus forming on a scientific python distribution
appropriate for HPC?
Thanks,
Chris
For what it's worth, our code (astrophysics analysis) provides an
installation script which builds the entire stack of dependencies,
which is how most users get it -- and we include notes about which
modules to load on different common SC centers. Recently, a few SC
centers have been building modules for it, but that's still in
process. We are still working on integrating mpi4py into the build
script, but that's part of a harder-than-just-mpi4py problem.
-Matt
> --
> You received this message because you are subscribed to the Google Groups "mpi4py" group.
> To post to this group, send email to mpi...@googlegroups.com.
> To unsubscribe from this group, send email to mpi4py+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mpi4py?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mpi4py/-/liLLZcu8dzAJ.
I think Nix (http://nixos.org) is *perfect* for clusters, but need some
work and customization to become easy enough to use.
My own thoughts on this are at
https://github.com/dagss/scidist/blob/master/ideas.rst
Since there seems like there's a lot of interest for this, how about a
BoF/Skype session/etc.?
On 09/19/2011 07:51 AM, Aron Ahmadia wrote:
> Hi Chris,
>
> This is an interesting, ambitious proposal. I am cc'ing the PyClaw
> developer's list (PyClaw developers, if you wish to respond, please
> reply to the mpi4py mailing list, though perhaps we need an
> hpc-p...@googlegroups.com <mailto:hpc-p...@googlegroups.com> mailing
> list to discuss these umbrella issues?)
>
> I have to admit that I am new to the Python HPC community, so although I
> have a lot of ideas, I'm sure you've heard or seen most of them before.
> I list them here mostly for completion in my own mind, but I'm happy
> to hear feedback on what has already been done or tried:
>
> [1/2/4] I agree that that maintaining a common build system or requiring
> a specific Python version will be a headache on all but the most generic
> of systems: Linux + x86_64+gfortran. I like to think that SiCortex was
> doomed when a series of clients pestered them to get SciPy ported to
> their MIPS processors :)
>
> I love virtualenv/pip, and I think that HPC centers should move more
> towards a 'toolchain' approach rather than the current 'module' system,
> which makes the fundamentally bad assumption that it is easy to mix and
> match various libraries and build settings.
>
> One thing that the Python community fundamentally gets wrong is the lack
> of support for building complicated extension modules. f2py/numpy do
> their best to unify this process, but just about every tool has their
> own slightly wonky way of getting packages/extensions built. We might
> be able to do the most good for the community by providing and enforcing
> a robust BuildSystem that deeply understands compiling and linking in a
> cross-platform way. Failing that, I would support the idea of a
> community maintained set of CMake configuration files for building
> Python packages, but we should make a strong effort encourage the
> package maintainers to support the CMake build.
There's also waf. I now use waf for building all my Cython + C + Fortran
projects. And David Cournapeau has put a lot of work into Bento, which
is intended to "fix" the packaging situation so that you can have a
proper Python egg package but doing the build using waf (or, in time,
CMake).
Dag Sverre
Great discussion.
I think Nix (http://nixos.org) is *perfect* for clusters, but need some work and customization to become easy enough to use.
My own thoughts on this are at
https://github.com/dagss/scidist/blob/master/ideas.rst
Since there seems like there's a lot of interest for this, how about a BoF/Skype session/etc.?
On 09/19/2011 07:51 AM, Aron Ahmadia wrote:
Hi Chris,
This is an interesting, ambitious proposal. Â I am cc'ing the PyClaw
developer's list (PyClaw developers, if you wish to respond, please
reply to the mpi4py mailing list, though perhaps we need an
hpc-p...@googlegroups.com <mailto:hpc-python@googlegroups.com> mailing
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to mpi4py+unsubscribe@googlegroups.com.
http://mail.python.org/mailman/listinfo/python-hpc
-Matt
On Mon, Sep 19, 2011 at 6:50 AM, Aron Ahmadia <ar...@ahmadia.net> wrote:
> Dag,
> Since there seems like there's a lot of interest for this, how about a
> BoF/Skype session/etc.?
> Please count me in. Â Perhaps a first step would be start a new Google group
> for discussing Python+HPC related issues in general, instead of piggybacking
> off of the mpi4py mailing list?
> Aron
> On Mon, Sep 19, 2011 at 12:57 PM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no> wrote:
>>
>> Great discussion.
>>
>> I think Nix (http://nixos.org) is *perfect* for clusters, but need some
>> work and customization to become easy enough to use.
>>
>> My own thoughts on this are at
>>
>> https://github.com/dagss/scidist/blob/master/ideas.rst
>>
>> Since there seems like there's a lot of interest for this, how about a
>> BoF/Skype session/etc.?
>>
>> On 09/19/2011 07:51 AM, Aron Ahmadia wrote:
>>>
>>> Hi Chris,
>>>
>>> This is an interesting, ambitious proposal. Â I am cc'ing the PyClaw
>>> developer's list (PyClaw developers, if you wish to respond, please
>>> reply to the mpi4py mailing list, though perhaps we need an
>>> hpc-p...@googlegroups.com <mailto:hpc-p...@googlegroups.com> mailing
>> mpi4py+un...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/mpi4py?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mpi4py" group.
> To post to this group, send email to mpi...@googlegroups.com.
> To unsubscribe from this group, send email to
> mpi4py+un...@googlegroups.com.
I'm also fine with switching this to the python-hpc mailing list as
well. I need to join it and am not sure it is really being used all
that much. Does anybody know Andreas Schreiber (the python-phc list
maintainer)?
Chris
How about trying to have something like a "structured discussion"?
Otherwise we derail into a 500-post long thread that leads nowhere?
Blogs are great for this as they create an asymmetry between blog posts
and comments, and allows editing posts after the fact.
So here's my proposal:
a) scientificcodedistribution.blogspot.com is created
b) We announce to others (numpy-discussion, sage-devel) that we're
going to have a "roundtable" (is that what it is called?) about "Finally
solving the scientific Python software building & distribution issues".
c) Each participant writes one blog post containing:
- Introduce yourself
- What kind of hardware are you using? What kind of computations do
you perform?
- What one has tried; what approaches have you used and are you using,
and what are their weaknesses
- What way do you see forward? How much time and other resources do
you have to spare to make things happen?
If more people than me promise to follow up with such a blog post I'm
happy to start the ball. Tell me what you think :-)
I believe *really* getting forward demands a workshop in the end. But we
just had another baby, so unless we can do it in Oslo, I may well not be
able to attend such a thing...
Dag Sverre
hpc-p...@googlegroups.com<mailto:hpc-python@googlegroups.com> Â mailing
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups
"mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to mpi4py+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to mpi4py+unsubscribe@googlegroups.com.
I'd be interested in sharing as well.
For what it's worth, several members of our group will be attending
the PyHPC session at SC11, and I could likely organize a small
workshop in NYC. I don't think [m]any of us could manage a trip to
Oslo.
-Matt
>>>>>>> hpc-p...@googlegroups.com<mailto:hpc-p...@googlegroups.com>
>>>>>> mpi4py+un...@googlegroups.com.
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/mpi4py?hl=en.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>> "mpi4py" group.
>>>>> To post to this group, send email to mpi...@googlegroups.com.
>>>>> To unsubscribe from this group, send email to
>>>>> mpi4py+un...@googlegroups.com.
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/mpi4py?hl=en.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "mpi4py" group.
>>>> To post to this group, send email to mpi...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> mpi4py+un...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/mpi4py?hl=en.
>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mpi4py" group.
>> To post to this group, send email to mpi...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> mpi4py+un...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/mpi4py?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mpi4py" group.
> To post to this group, send email to mpi...@googlegroups.com.
> To unsubscribe from this group, send email to
> mpi4py+un...@googlegroups.com.
I think PyHPC (or in conjunction) sounds like a great idea. Also it's
close to a lot of Sage people.
I'll try to kick off the blog tomorrow and post some invitations.
Dag Sverre
>
> -Matt
>
>>
>> On Mon, Sep 19, 2011 at 9:53 PM, Dag Sverre Seljebotn
>> <d.s.se...@astro.uio.no> wrote:
>>>
>>> Hmm. There's actually a lot to synchronize here for a good discussion.
>>> I've participated in numerous similar discussions in the past (sage-devel
>>> etc.). I wonder if things are a bit too much in flux for a Skype session --
>>> there's many parties/people who are interested in having this problem
>>> solved, there's been *a lot* of prior discussion without any obvious full
>>> solutions.
>>>
>>> How about trying to have something like a "structured discussion"?
>>> Otherwise we derail into a 500-post long thread that leads nowhere?
>>>
>>> Blogs are great for this as they create an asymmetry between blog posts
>>> and comments, and allows editing posts after the fact.
>>>
>>> So here's my proposal:
>>>
>>> a) scientificcodedistribution.blogspot.com is created
>>>
>>> b) We announce to others (numpy-discussion, sage-devel) that we're going
>>> to have a "roundtable" (is that what it is called?) about "Finally solving
>>> the scientific Python software building& distribution issues".
Also, I like the looks of Nix, and I agree with a lot of the points
you and Aron have mentioned. Thanks for kicking off the blog.
-Chris
On Mon, Sep 19, 2011 at 3:10 PM, Dag Sverre Seljebotn
On 09/19/2011 09:43 PM, Matthew Turk wrote:I think PyHPC (or in conjunction) sounds like a great idea. Also it's close to a lot of Sage people.
On Mon, Sep 19, 2011 at 3:40 PM, Aron Ahmadia<ar...@ahmadia.net> Â wrote:
Dag,
You have my support. Â I can definitely introduce our challenges at the KAUST
Supercomputing Laboratory in a contribution to the blog/github
repository/newsletter :)
A
PS - I am happy to visit Norway :)
I'd be interested in sharing as well.
For what it's worth, several members of our group will be attending
the PyHPC session at SC11, and I could likely organize a small
workshop in NYC. Â I don't think [m]any of us could manage a trip to
Oslo.
I'll try to kick off the blog tomorrow and post some invitations.
Dag Sverre
-Matt
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google
Groups
"mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google
Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups
"mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups
"mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mpi4py?hl=en.
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com.
To unsubscribe from this group, send email to mpi4py+unsubscribe@googlegroups.com.
Just to clarify: I hope that something will happen there, it sounds like
a great place to discuss this. But I don't think I'll be able to come
myself.
Dag Sverre
OK all, welcome to
http://fixingscientificsoftwaredistribution.blogspot.com/
Send me an email in private to be invited as an author. Once there's one
more blog post than mine (for balance...) I'll announce it on
numpy-discuss, sage-devel and so on.
Dag Sverre