--
You received this message because you are subscribed to the Google Groups "ray-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ray-dev+u...@googlegroups.com.
To post to this group, send email to ray...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ray-dev/CAFs1FxUBAag6AThj34twiAB6KY3t5sJSJF3g70K3SvF-%2BzGGgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/developers/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/CAGZdauXqQ9Gze6eAB0R3%3D2j6X2yWfh7QPbrGj1%3D5xuvQUninpQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/CADtzJKMzpDj2SfFRygaxKTgJD3eoKi7kKBUgZExN9cceMN2CyQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/CADtzJKMzpDj2SfFRygaxKTgJD3eoKi7kKBUgZExN9cceMN2CyQ%40mail.gmail.com.
Thank you Philipp for getting this started. We've been trying to get in touch and have tried via Nick Coghlan and Nathaniel Smith, but we never got far.I'm a little late to the party, but basically, what Soumith said. We have the exact same constraints (C++-11, CUDA/cuDNN). These would be extremely common for any computation-heavy packages, and properly solving this issue would be a huge boon for the Python community.Actual compliance with manylinux1 is out since it cannot fulfill those constraints. I'll also add that there is no way to build compliant wheels without using software beyond end-of-life (even beyond security updates).manylinux2010 is indeed promising, and I saw that Nick merged support for it recently, though I don't think there has been a pip release including the support yet (maybe that has now changed?).However, manylinux2010 still has (possible fatal) problems:- CUDA10's minimum versions are higher than manylinux2010's maximum versions: specifically, GCC 4.4.7 > 4.3.0.- NVIDIA's license terms for CUDA/cuDNN are not standard and redistribution can be problematic, and may depend on agreements you may have with NVIDIA. The libraries are also large, and including them would make distribution via pypi problematic. It would be much preferable if there was an approved way to distribute Python packages depending on external CUDA/cuDNN. I don't think this should be a problem, it is similar in spirit to the exception made for libGL.
You received this message because you are subscribed to the Google Groups "SIG Build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to build+un...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/build/.
hi Manuel,
Adding a couple more folks from Apache Arrow to the thread to make
sure they see this discussion.
On Tue, Jan 22, 2019 at 3:48 AM Manuel Klimek <kli...@google.com> wrote:
>
> Sorry if I'm missing something fundamental, but it seems like a new manylinux standard would come with the same problem of basically being static and growing outdated.
>
> I'd be interested in helping to provide a toolchain wheel, as mentioned in the initial post, at least for libc++ (potentially libstdc++) - it seems like that could be updated on an ongoing basis, use standard dependency management and if necessary be bootstrapped with a statically linked compiler.
>
> What would the requirements for such a toolchain wheel be for it to have a chance to be widely used? (note that I come from a C++ background and don't have a lot of experience with Python outside of modules using C++ under the hood :)
In principle I would think that the requirement would be that we
demonstrate that wheels built with the newer compiler toolchain and
libstdc++ dependency can coexist with manylinux1 / manylinux2010
packages. This is supposed to be the promise of devtoolset-produced
libraries anyhow. A potential problem might be projects that need to
pass std::* objects between shared libraries in their C++ API. For
example, the "turbodbc" package uses the "pyarrow" package's C++ API.
This would just mean that any wheel that needs to depend on a wheel in
the "TF/PyTorch-compatible toolchain" ecosystem would necessarily need
to use the alternative build toolchain instead of manylinux*
If so, are you going to claim that the given wheel is manylinux-compatible?
Regards
Antoine.
You received this message because you are subscribed to the Google Groups "SIG Build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to build+un...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/build/.
Regards
Antoine.
>>Fundamentally, the C++ dependency chain seems to be solvable with pip package deps down to the libstdc++/libc++ level.>>I think we'd basically need to provide:>>a) a toolchain pip package to depend onSounds like Anaconda :)>>b) a manylinux docker image with those libraries and a compiler toolchain targeting them installed so packagers have an easy way to build these packagesSounds like conda-forge :)Not to speak for Jonathan and Anconda, but I suspect they designed things the way they did for much of the same reasons as we are discussing here. pip and manylinux standards alone are not/were not good enough.
>>Once we have that in a way that folks are happy with it, it sounds like we'd be good to go?>>There are a couple of obvious questions:>>- how to handle updates of that toolchain package / toolchain?>>- what would we want to target as a first step?I'd also add:- libc is still a pain, so targeted versions there would need spec'd out
>>My proposal for something that we could work on would be:
>>clang & libc++ @ llvm-8Not that I disagree, but why clang over gcc? Seems like gcc may have a bit better compatibility across the board, but that might just be momentum talking.
Regards
Antoine.
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/developers/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/CAPuKSJbMA%3DJpvLj%2BFcUK%3DZXXJ5gdz%3D7-oPf8EzEkMNNDXq63ag%40mail.gmail.com.
Just as a heads-up: I would like to also join the meeting but am also located in Europe.I have spent quite some time with the packaging of wheels for pyarrow and turbodbc thus I would like to also give input on this. For Apache Arrow, I see newer manylinux2014 standard as a possible way to go. I'm not so fond of rolloing lib(std)c++ packages inside of pip. It's sadly the case that the features of pip don't allow a good dependency resolution, also with taking CUDA into account, a dependency resolution that differs between source and binary builds of a package. For this case, exactly conda was developed because it was considered out-of-scope for the core Python packaging system. I'm not sure whether we actually can fit all the requirements of the packages that take part in this mail thread into pip without simply reimplementing conda inside of pip.
> I think trying to package CUDA is the wrong way to think about it.
Instead, perhaps you should try to make the package compatible with
system CUDA installs.
I agree in principle.
The problem fundamentally stems from user expectation.
In my ~6+ years of supporting Torch and PyTorch, installing CUDA on a
system can take days, with a user mean approximately half a day. It might
be userland incompetence, or that CUDA is a magical snowflake, but the
reality is that installing CUDA is never great.
So, a huge amount of issues reported by userland are side-effects from
broken CUDA installs.
It doesn't help that the PyPI user expectations of "my package should just
work after a pip install".
If we can reliably install an up-to-date CUDA in a standardized way, and
NVIDIA simply doesn't sidestep the userland issues by saying "user our
docker", or "our PPA is 100% reliable", we would've been in a better state.
Until then, I think it's best that we find a solution for PyPI users that
can work out of box with PyPI.
On Mon, Feb 4, 2019 at 12:52 PM Antoine Pitrou <soli...@pitrou.net> wrote:
> On Tue, 5 Feb 2019 01:45:34 +0800
> Jason Zaman <ja...@perfinion.com> wrote:
> > On Tue, 5 Feb 2019 at 01:30, soumith <sou...@gmail.com> wrote:
> > >
> > > Unfortunately I'll be on a long flight, and cannot make it to the
> SIGBuild meeting.
> > > I'm definitely interested in the meeting notes and any follow-up
> meeting.
> > >
> > > > I think we should leave CUDA out of the
> > > discussion initially and see if we can get the cpu-only wheel working
> > > correctly. Hopefully cpu-only is viable on manylinux2014 then we can
> > > tackle CUDA afterwards.
> > >
> > > 50% of the complexity is in the CUDA packaging.
> > > The other 50% is in shipping a more modern libstdc++.so
> > > I believe we'll make progress if we ignore CUDA, but we'll not address
> half of the issue.
> >
> > Yeah, we'll definitely need both to solve it fully. My thinking is
> > that all packages need at least C++11 but only some need CUDA. Or
> > might we end up where the libstcc++.so is incompatible with CUDA if we
> > don't work on everything together?
>
> I think trying to package CUDA is the wrong way to think about it.
> Instead, perhaps you should try to make the package compatible with
> system CUDA installs.
>
> For example, the Numba pip wheel almost works out-of-the-box with a
> system CUDA install on Ubuntu 18.04. I say "almost" because I had to
> set two environment variables:
> https://github.com/numba/numba/issues/3738
>
> Regards
>
> Antoine.
>
>
>
Replying to the thread because the last two messages got dropped.On Mon, Feb 4, 2019 at 10:00 AM soumith <sou...@gmail.com> wrote:> I think trying to package CUDA is the wrong way to think about it.
Instead, perhaps you should try to make the package compatible with
system CUDA installs.
I agree in principle.
The problem fundamentally stems from user expectation.
In my ~6+ years of supporting Torch and PyTorch, installing CUDA on a
system can take days, with a user mean approximately half a day. It might
be userland incompetence, or that CUDA is a magical snowflake, but the
reality is that installing CUDA is never great.
So, a huge amount of issues reported by userland are side-effects from
broken CUDA installs.
It doesn't help that the PyPI user expectations of "my package should just
work after a pip install".
If we can reliably install an up-to-date CUDA in a standardized way, and
NVIDIA simply doesn't sidestep the userland issues by saying "user our
docker", or "our PPA is 100% reliable", we would've been in a better state.
Until then, I think it's best that we find a solution for PyPI users that
can work out of box with PyPI.
Hello Dimitri,Option "Userspace-2" sounds for me exactly like the thing that conda does. There is already a community around conda-forge that takes care of packaging all native requirements in separate packages including a modern toolchain that is separate from the host system. I still need to understand, why conda is not an option then? We would just be replicating this setup then.As previously mentioned, getting conda functionality into pip would be a valid option but we may face the same issues as to when conda was created. I doubt that the PyPA is more open to this scope expansion then they were then. The personell situation is still very limited in the packaging space. For the users of Arrrow, we definitely have had much better experience with users working in conda than those that were using pip, mainly due to the package manager taking care of all the binary dependencies between different packages like arrow, torch and tensorflow.Also to reiterate a point raised earlier: C++11 with manylinux1 works smoothly. With gcc 4.8.5, everything we need in Arrow supported. C++14 and more are out of scope and can only be used starting with manylinux{2010/2014}.
From the requirements side (Martin will correct me if I'm getting these wrong):- it seems like from the TF point of view, our users are on pip, so we need to deliver there- LLVM is going to require C++14 ~in March as far as I can tell- from trying to find info about manylinux2010 / 14, it seems like these have stalled? (but I'm happy to be proven wrong here :)
Le 05/02/2019 à 16:22, Manuel Klimek a écrit :
> On Tue, Feb 5, 2019 at 2:01 PM Uwe L. Korn <xho...@gmail.com
> <mailto:xho...@gmail.com>> wrote:
>
> Also to reiterate a point raised earlier: C++11 with manylinux1
> works smoothly. With gcc 4.8.5, everything we need in Arrow
> supported. C++14 and more are out of scope and can only be used
> starting with manylinux{2010/2014}.
>
> From the requirements side (Martin will correct me if I'm getting these
> wrong):
> - it seems like from the TF point of view, our users are on pip, so we
> need to deliver there
> - LLVM is going to require C++14 ~in March as far as I can tell
> - from trying to find info about manylinux2010 / 14, it seems like these
> have stalled? (but I'm happy to be proven wrong here :)
manylinux2010 hasn't stalled, it's been progressing slowly. Apparently
pip 19.0 is out which supports downloading and installing manylinux2010
packages. See status page here:
https://github.com/pypa/manylinux/issues/179#issuecomment-457002180
On Tue, Feb 5, 2019 at 4:28 PM Antoine Pitrou <ant...@python.org> wrote:
Le 05/02/2019 à 16:22, Manuel Klimek a écrit :
> On Tue, Feb 5, 2019 at 2:01 PM Uwe L. Korn <xho...@gmail.com
> <mailto:xho...@gmail.com>> wrote:
>
> Also to reiterate a point raised earlier: C++11 with manylinux1
> works smoothly. With gcc 4.8.5, everything we need in Arrow
> supported. C++14 and more are out of scope and can only be used
> starting with manylinux{2010/2014}.
>
> From the requirements side (Martin will correct me if I'm getting these
> wrong):
> - it seems like from the TF point of view, our users are on pip, so we
> need to deliver there
> - LLVM is going to require C++14 ~in March as far as I can tell
> - from trying to find info about manylinux2010 / 14, it seems like these
> have stalled? (but I'm happy to be proven wrong here :)
manylinux2010 hasn't stalled, it's been progressing slowly. Apparently
pip 19.0 is out which supports downloading and installing manylinux2010
packages. See status page here:
https://github.com/pypa/manylinux/issues/179#issuecomment-457002180
Cool! The problem is that it doesn't solve the C++14 issue, right?
manylinux2014 is an entirely different question. It needs interested
parties to gather and devise a spec and then get it accepted as a new PEP.
Regards
Antoine.
--
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/developers/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/CAOsfVvnY5jSsB6aF8qw-d1TFF3XX1OKgCpXC8%3DQ9dyfYbGed_w%40mail.gmail.com.
Le 06/02/2019 à 01:06, Philipp Moritz a écrit :
> Thanks for the meeting! One question concerning a point that is still
> not super clear to me:
>
> Say we define a new manylinux standard based on gcc >=5 (with stable
> c++11 support). There will still be a lot of wheels form the manylinux1
> days that are built against gcc 4.8 that might use the c++11 features
> before they became stable. How do we prevent bugs from that? Is the plan
> to convince everybody who uses these c++11 features to use the new
> manylinux standard?
Yes, that's a bit of a problem.
This discussion arised from the incompatibility between Tensorflow
wheels (compiled with a later toolchain) and other Python wheels
(compiled with a manylinux1-compatible toolchain).
On 2/5/19 9:29 AM, 'Manuel Klimek' via TensorFlow Developers wrote:
On Tue, Feb 5, 2019 at 4:28 PM Antoine Pitrou <ant...@python.org> wrote:
Le 05/02/2019 à 16:22, Manuel Klimek a écrit :
> On Tue, Feb 5, 2019 at 2:01 PM Uwe L. Korn <xho...@gmail.com
> <mailto:xho...@gmail.com>> wrote:
>
> Also to reiterate a point raised earlier: C++11 with manylinux1
> works smoothly. With gcc 4.8.5, everything we need in Arrow
> supported. C++14 and more are out of scope and can only be used
> starting with manylinux{2010/2014}.
>
> From the requirements side (Martin will correct me if I'm getting these
> wrong):
> - it seems like from the TF point of view, our users are on pip, so we
> need to deliver there
> - LLVM is going to require C++14 ~in March as far as I can tell
> - from trying to find info about manylinux2010 / 14, it seems like these
> have stalled? (but I'm happy to be proven wrong here :)
manylinux2010 hasn't stalled, it's been progressing slowly. Apparently
pip 19.0 is out which supports downloading and installing manylinux2010
packages. See status page here:
https://github.com/pypa/manylinux/issues/179#issuecomment-457002180
Cool! The problem is that it doesn't solve the C++14 issue, right?
Devtoolset-7 can be installed on RHEL6/CentOS 6 which is the reference distribution of manylinux2010. Devtoolset-7 includes GCC 7.3.1 which has full support for C++14. On RHEL6/CentOS 6 the devtoolset compilers target the older GCC C++ ABI (-D_GLIBCXX_USE_CXX11_ABI=0) and will not emit the newer ABI.