Ideas for Hashdist project

[Dag Sverre Seljebotn]

I might have some time this spring for a project on Hashdist for 2
weeks; probably around March.

I'd like to work on core Hashdist itself, not the Hashstack. The things
I can think of myself are:

1) Improvements config/property system

This is the trickiest thing to get right, and I don't feel like we are
completely done. But it's also a tough thing to get right. And it's high
risk (as to whether something comes out of it in 2 weeks) as it's more
of a research project.

2) Smooth binary installs

"Compete with Conda", more or less. For beginning scientific Python
users I find myself recommending them to use Conda right now.


Dag Sverre

[Dag Sverre Seljebotn]

Alas, trying out Conda a bit just now I got:

ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
`GLIBCXX_3.4.20' not found (required by
/home/dagss/opt/anaconda/lib/python2.7/site-packages/healpy/_healpy_pixel_lib.so)


Dag Sverre

[Chris Kees]

That's great news. My view is that the binary installs will probably do the greatest good for the greatest number.  That's not  always the right criterion, but I see binary install support as really streamlining several areas that are really important for  our lab right now, including faster testing of our downstream tools, supporting small groups of users on supercomputers, and building several complex in-house stacks that have a lot of  new python users/developers.

Dag Sverre

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jimmy Tang]

+1 on binary installs, I'd like to see some effort put into the filtering of symlinks and possibly unicode related issues inside tarballs and git sources, it's been causing me no end to misery on certain packages which have symlinks or unicode in upstream releases.

Jimmy

[Dag Sverre Seljebotn]

I'll go for binary installs then. I'll have a think, then I'll draft up
a short spec and we'll do a Hangout meeting or something to discuss the
details etc. before I spend too much time experimenting or implementing.

As I said I aim for March for this, and it may get to April or May
instead before I can get around to it (to really dig down I mean, I'll
think and tinker until then).

Dag Sverre

[Ondřej Čertík]

That seems related to these bugs:

https://github.com/ContinuumIO/anaconda-issues/issues/182
https://github.com/sympy/csympy/issues/376

As to ideas for hashdist, here is what bothers me personally:

https://groups.google.com/d/topic/hashdist/NbdEKb8uZtQ/discussion

It's all improvements to hashdist, the main one being able to load
profiles by their names using the `hit` tool, which means the hashdist
database must know about profile names.

Ondrej

>
>
>
> Dag Sverre
>
> --
> You received this message because you are subscribed to the Google Groups
> "hashdist" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to hashdist+u...@googlegroups.com.

[François Bissey]

On Wednesday, January 28, 2015 at 1:13:55 PM UTC+13, ondrej.certik wrote:

On Tue, Jan 27, 2015 at 2:35 AM, Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:
> On 01/27/2015 10:33 AM, Dag Sverre Seljebotn wrote:
>>
>> I might have some time this spring for a project on Hashdist for 2
>> weeks; probably around March.
>>
>> I'd like to work on core Hashdist itself, not the Hashstack. The things
>> I can think of myself are:
>>
>> 1) Improvements config/property system
>>
>> This is the trickiest thing to get right, and I don't feel like we are
>> completely done. But it's also a tough thing to get right. And it's high
>> risk (as to whether something comes out of it in 2 weeks) as it's more
>> of a research project.
>>
>> 2) Smooth binary installs
>>
>> "Compete with Conda", more or less. For beginning scientific Python
>> users I find myself recommending them to use Conda right now.
>
>
> Alas, trying out Conda a bit just now I got:
>
> ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `GLIBCXX_3.4.20' not found (required by
> /home/dagss/opt/anaconda/lib/python2.7/site-packages/healpy/_healpy_pixel_lib.so)

That seems related to these bugs:

https://github.com/ContinuumIO/anaconda-issues/issues/182
https://github.com/sympy/csympy/issues/376

Similar. But the github issues are related to libm which is part glibc.

This is libstdc++ which is part of gcc.

Getting a newer compiler and setting LD_LIBRARY_PATH appropriately

will solve this problem. I doubt many people would know or be willing

to do that with glibc.

But the fact that people report these failure shows that continuum is

not conservative enough in selecting the environment in which they

build binaries. I am guessing they assume a recent ubuntu or fedora.

This is probably appropriate for individual in charge of their own desktops

but if you target enterprise people you may want to base you binaries

on RH6 or SLES11. Kitware ship binaries of paraview built against 

glibc 2.3.6 that's even older - I believe its debian but cannot find the details.

Francois

[Aron Ahmadia]

On Tue, Jan 27, 2015 at 7:57 PM, François Bissey <francoi...@canterbury.ac.nz> wrote:

But the fact that people report these failure shows that continuum is

not conservative enough in selecting the environment in which they

build binaries. I am guessing they assume a recent ubuntu or fedora.

This is probably appropriate for individual in charge of their own desktops

but if you target enterprise people you may want to base you binaries

on RH6 or SLES11. Kitware ship binaries of paraview built against 

glibc 2.3.6 that's even older - I believe its debian but cannot find the details.

If I recall correctly, Continuum uses a pretty old glibc.  Same deal with their OS X environment, which is based on the X.5 SDK.  C++ compatibility is a huge PITA, which brings us to bootstrapping... :)

[Aron Ahmadia]

On Tue, Jan 27, 2015 at 3:00 PM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

    1) Improvements config/property system

    This is the trickiest thing to get right, and I don't feel like we are
    completely done. But it's also a tough thing to get right. And it's
    high
    risk (as to whether something comes out of it in 2 weeks) as it's more
    of a research project.

    2) Smooth binary installs

    "Compete with Conda", more or less. For beginning scientific Python
    users I find myself recommending them to use Conda right now.

I've made some progress on the binary installs in a private branch.  To be honest, the HashDist side of things is not hard, the pain will come in testing the tools on the various systems we are interested in deploying on. 

I'm not sure what you have in mind for 1], but it's probably worth your time to have a look at: https://github.com/hashdist/hashdist/wiki/HDEP-1:-Crating-System-for-Redistributing-Packages and https://github.com/hashdist/hashdist/wiki/HDEP-2:-Automating-Configuration to see what discussions have happened already.

A

[Dag Sverre Seljebotn]

To be fair this was a package from binstar that anyone could make, not part of anaconda.

DS

28. jan. 2015 01:57 skrev François Bissey <francoi...@canterbury.ac.nz>:

On Wednesday, January 28, 2015 at 1:13:55 PM UTC+13, ondrej.certik wrote:

On Tue, Jan 27, 2015 at 2:35 AM, Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:
> On 01/27/2015 10:33 AM, Dag Sverre Seljebotn wrote:
>>
>> I might have some time this spring for a project on Hashdist for 2
>> weeks; probably around March.
>>
>> I'd like to work on core Hashdist itself, not the Hashstack. The things
>> I can think of myself are:
>>

>> 1) Improvements config/property system
>>
>> This is the trickiest thing to get right, and I don't feel like we are
>> completely done. But it's also a tough thing to get right. And it's high
>> risk (as to whether something comes out of it in 2 weeks) as it's more
>> of a research project.
>>
>> 2) Smooth binary installs
>>
>> "Compete with Conda", more or less. For beginning scientific Python
>> users I find myself recommending them to use Conda right now.
>
>

> Alas, trying out Conda a bit just now I got:
>
> ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `GLIBCXX_3.4.20' not found (required by
> /home/dagss/opt/anaconda/lib/python2.7/site-packages/healpy/_healpy_pixel_lib.so)

That seems related to these bugs:

https://github.com/ContinuumIO/anaconda-issues/issues/182
https://github.com/sympy/csympy/issues/376

Similar. But the github issues are related to libm which is part glibc.

This is libstdc++ which is part of gcc.

Getting a newer compiler and setting LD_LIBRARY_PATH appropriately

will solve this problem. I doubt many people would know or be willing

to do that with glibc.

But the fact that people report these failure shows that continuum is

not conservative enough in selecting the environment in which they

build binaries. I am guessing they assume a recent ubuntu or fedora.

This is probably appropriate for individual in charge of their own desktops

but if you target enterprise people you may want to base you binaries

on RH6 or SLES11. Kitware ship binaries of paraview built against 

glibc 2.3.6 that's even older - I believe its debian but cannot find the details.

Francois