Caffe, TensorFlow, protobuf-3
171 views
Skip to first unread message
Predrag Punosevac
Oct 13, 2016, 10:38:33 AM10/13/16
to springdale-users
Dear All,
I keep building this manually but it would be nice to have RPMs in computational. Does anyone have a WIP RPM so that I don't have to duplicate the effort.
Predrag
Asya Dvorkin
Oct 13, 2016, 11:15:07 AM10/13/16
to springda...@googlegroups.com
Hello, Predrag.
We have caffe 1.00rc3 if you are interested.
ftp://ftp.cs.princeton.edu/pub/people/advorkin/SRPM/sd7/caffe-1.00rc3-3.sd7.src.rpm
TensforFlow and protobuf-3 work great with conda (http://conda.pydata.org). I just tried and had no problems installing it for Python 2.7 and 3.5
Thank you,
Asya
--
You received this message because you are subscribed to the Google Groups "springdale-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to springdale-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
We have caffe 1.00rc3 if you are interested.
ftp://ftp.cs.princeton.edu/pub/people/advorkin/SRPM/sd7/caffe-1.00rc3-3.sd7.src.rpm
TensforFlow and protobuf-3 work great with conda (http://conda.pydata.org). I just tried and had no problems installing it for Python 2.7 and 3.5
Thank you,
Asya
You received this message because you are subscribed to the Google Groups "springdale-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to springdale-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
JP
Oct 13, 2016, 11:15:37 AM10/13/16
to springdale-users
Ah, where to start. I've had plenty of experience building tensorflow - we had no recent requests for caffe or protobuf-3.
So for tensorflow we had to intervene because pip packages that they ship don't work on RHEL6 which is why I did spend time to build it but I only packaged its dependencies not tensorflow itself. This is documented at
and I have bazel rpms in computational/6. I don't remember if it had other dependencies but it does need cudatoolkit and cudann for GPU version (which we cannot redistribute so it's not visible in computational repo).
Just not sure how useful it would be to package such a fast moving target, especially due to various gpu/non gpu versions. We usually tell people to do
pip install --user http://..../tensorflor...whl
Maybe behind modules, hm.
Caffe - I see I didn't help anyone with it since February. As far as I can tell it's the same forever-in-development story with no real releases in a long while. It also depends on a few other newer things (boost/hdf5). So here too probably the only option is not a central install but /usr/local + environment module, rpm or not.
I didn't even realize protobuf 3 is out. You already have requests for it? And I thought our users were demanding...
Josko
--
Predrag Punosevac
Oct 13, 2016, 1:45:39 PM10/13/16
to plaz...@gmail.com, springda...@googlegroups.com
JP <plaz...@gmail.com> wrote:
> Ah, where to start. I've had plenty of experience building tensorflow - we
> had no recent requests for caffe or protobuf-3.
>
> So for tensorflow we had to intervene because pip packages that they ship
> don't work on RHEL6 which is why I did spend time to build it but I only
> packaged its dependencies not tensorflow itself. This is documented at
>
> http://tigress-web.princeton.edu/~plazonic/tensorflow/rhel6/
>
> and I have bazel rpms in computational/6. I don't remember if it had other
> dependencies but it does need cudatoolkit and cudann for GPU version (which
> we cannot redistribute so it's not visible in computational repo).
>
> Just not sure how useful it would be to package such a fast moving target,
> especially due to various gpu/non gpu versions. We usually tell people to
> do
>
> pip install --user http://..../tensorflor...whl
>
> Maybe behind modules, hm.
>
> Caffe - I see I didn't help anyone with it since February. As far as I can
> tell it's the same forever-in-development story with no real releases in a
> long while. It also depends on a few other newer things (boost/hdf5). So
> here too probably the only option is not a central install but /usr/local +
> environment module, rpm or not.
>
> I didn't even realize protobuf 3 is out. You already have requests for it?
> And I thought our users were demanding...
Yes my guys want to run the latest versions of Caffe and TensorFlow
> Ah, where to start. I've had plenty of experience building tensorflow - we
> had no recent requests for caffe or protobuf-3.
>
> So for tensorflow we had to intervene because pip packages that they ship
> don't work on RHEL6 which is why I did spend time to build it but I only
> packaged its dependencies not tensorflow itself. This is documented at
>
> http://tigress-web.princeton.edu/~plazonic/tensorflow/rhel6/
>
> and I have bazel rpms in computational/6. I don't remember if it had other
> dependencies but it does need cudatoolkit and cudann for GPU version (which
> we cannot redistribute so it's not visible in computational repo).
>
> Just not sure how useful it would be to package such a fast moving target,
> especially due to various gpu/non gpu versions. We usually tell people to
> do
>
> pip install --user http://..../tensorflor...whl
>
> Maybe behind modules, hm.
>
> Caffe - I see I didn't help anyone with it since February. As far as I can
> tell it's the same forever-in-development story with no real releases in a
> long while. It also depends on a few other newer things (boost/hdf5). So
> here too probably the only option is not a central install but /usr/local +
> environment module, rpm or not.
>
> I didn't even realize protobuf 3 is out. You already have requests for it?
> And I thought our users were demanding...
which forced me to manually upgrade protobuf to 3. Thanks God you did
that devtool-4-gcc-5.3 as somebody in our group used the features not
present in earlier GCC for his program. They were also pushing me
recently with some newest Python 3.5 stuff. Between I already moved
more or less completely to 7.
On the unrelated note I am hitting some MATLAB R2016b bugs on Titan X
(Pascal) cards. gpuDevice command just hangs. You will also need to
manually install drivers from NVIDIA website before installing CUDA.
That is not documented to my knowledge. Tesla K80 seems to work much
better for us but it is almost 3 times more expensive than Titan.
Predrag
>
> Josko
>
> On Thu, Oct 13, 2016 at 10:38 AM Predrag Punosevac <punos...@gmail.com>
> wrote:
>
> Dear All,
>
> I keep building this manually but it would be nice to have RPMs in
> computational. Does anyone have a WIP RPM so that I don't have to duplicate
> the effort.
>
>
> Predrag
>
> --
> You received this message because you are subscribed to the Google Groups
> "springdale-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to springdale-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/springdale-users/KSo1Z7E671Y/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to springdale-use...@googlegroups.com.
Predrag Punosevac
Oct 13, 2016, 2:10:22 PM10/13/16
to advo...@cs.princeton.edu, springda...@googlegroups.com
Asya Dvorkin <advo...@CS.Princeton.EDU> wrote:
> Hello, Predrag.
>
> We have caffe 1.00rc3 if you are interested.
>
> ftp://ftp.cs.princeton.edu/pub/people/advorkin/SRPM/sd7/caffe-1.00rc3-3.sd7.src.rpm
>
> TensforFlow and protobuf-3 work great with conda (http://conda.pydata.org). I just tried and had no problems installing it for Python 2.7 and 3.5
>
> Thank you,
> Asya
Asya,
> Hello, Predrag.
>
> We have caffe 1.00rc3 if you are interested.
>
> ftp://ftp.cs.princeton.edu/pub/people/advorkin/SRPM/sd7/caffe-1.00rc3-3.sd7.src.rpm
>
> TensforFlow and protobuf-3 work great with conda (http://conda.pydata.org). I just tried and had no problems installing it for Python 2.7 and 3.5
>
> Thank you,
> Asya
Are you guys using TensorFlow on GPUs. One of my guys is telling me that
according to the tensorflow site, the conda package doesn't support
GPUs.
Predrag
>
>
>
> ----- Original Message -----
> From: "Predrag Punosevac" <punos...@gmail.com>
> To: "springdale-users" <springda...@googlegroups.com>
> Sent: Thursday, October 13, 2016 10:38:32 AM
> Subject: Caffe, TensorFlow, protobuf-3
>
> Dear All,
>
> I keep building this manually but it would be nice to have RPMs in
> computational. Does anyone have a WIP RPM so that I don't have to duplicate
> the effort.
>
>
> Predrag
>
> --
> You received this message because you are subscribed to the Google Groups "springdale-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to springdale-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/springdale-users/KSo1Z7E671Y/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to springdale-use...@googlegroups.com.
JP
Oct 13, 2016, 2:44:58 PM10/13/16
to springda...@googlegroups.com
On Thu, Oct 13, 2016 at 1:45 PM Predrag Punosevac <punos...@gmail.com> wrote:
Yes my guys want to run the latest versions of Caffe and TensorFlow
which forced me to manually upgrade protobuf to 3. Thanks God you did
that devtool-4-gcc-5.3 as somebody in our group used the features not
present in earlier GCC for his program. They were also pushing me
recently with some newest Python 3.5 stuff. Between I already moved
more or less completely to 7.
I checked and I see now that protobuf 3 gets built during tensorflow build (found it in ~/.cache/...). You had to build it separately?
And indeed, 7 makes life so much easier. Actually so far we were able to use tensorflow whl files as distributed by them on 7 with no modifications. Just pip --user install - that didn't work for you?
On the unrelated note I am hitting some MATLAB R2016b bugs on Titan X
(Pascal) cards. gpuDevice command just hangs. You will also need to
manually install drivers from NVIDIA website before installing CUDA.
That is not documented to my knowledge. Tesla K80 seems to work much
better for us but it is almost 3 times more expensive than Titan.
We manually install cuda and xorg anyway. Custom build of xorg-x11-drv-nvidia that closely resembles the one from rpmfusion + a simple wrapper around cuda rpms that pulls relevant rpms and drops in a module.
Josko
Asya Dvorkin
Oct 13, 2016, 3:53:44 PM10/13/16
to springda...@googlegroups.com
Hello, Predrag.
I got it to work with GPU support. I used conda package manager for most of work, but pip for the actual (custom link) tensorflow installation.
$ python
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
>>> tf.test.is_built_with_cuda()
True
---------------------
Make sure to set the following environmental variables before installing tensorflow. In addition, install with pip by specifying a custom whl location of tensorflow GPU version. I only tested it with python 2.7 on SD7, but I assume it will work with python 3.5 as well.
export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda-7.5/targets/x86_64-linux/lib/"
export CUDA_HOME=/usr/local/cuda
You will need to manually install all prerequisite modules (conda installed correct versions, while pip didn't):
$ conda install tensorflow
Fetching package metadata .......
Solving package specifications: ..........
Package plan for installation in environment /usr/local/conda/2.7:
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-env-2.6.0 | 0 502 B
libprotobuf-3.0.0 | 0 6.0 MB
protobuf-3.0.0 | py27_0 407 KB
pbr-1.10.0 | py27_0 100 KB
mock-2.0.0 | py27_0 96 KB
tensorflow-0.10.0rc0 | np111py27_0 25.0 MB
------------------------------------------------------------
Total: 31.7 MB
The following NEW packages will be INSTALLED:
conda-env: 2.6.0-0
libprotobuf: 3.0.0-0
mock: 2.0.0-py27_0
pbr: 1.10.0-py27_0
protobuf: 3.0.0-py27_0
tensorflow: 0.10.0rc0-np111py27_0
Manually install all required packages (with conda). Do not install the version conda wants you to install (it's only CPU enabled).
Below path can be found at https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html#pip-installation, so use appropriate whl file.
$ pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0rc0-cp27-none-linux_x86_64.whl
That should be it. I got tf.test.is_built_with_cuda() test from the page below: https://www.tensorflow.org/versions/r0.8/api_docs/python/test.html
Please let me know if you are having any issues.
Thank you,
Asya
I got it to work with GPU support. I used conda package manager for most of work, but pip for the actual (custom link) tensorflow installation.
$ python
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
>>> tf.test.is_built_with_cuda()
True
---------------------
Make sure to set the following environmental variables before installing tensorflow. In addition, install with pip by specifying a custom whl location of tensorflow GPU version. I only tested it with python 2.7 on SD7, but I assume it will work with python 3.5 as well.
export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda-7.5/targets/x86_64-linux/lib/"
export CUDA_HOME=/usr/local/cuda
You will need to manually install all prerequisite modules (conda installed correct versions, while pip didn't):
$ conda install tensorflow
Fetching package metadata .......
Solving package specifications: ..........
Package plan for installation in environment /usr/local/conda/2.7:
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-env-2.6.0 | 0 502 B
libprotobuf-3.0.0 | 0 6.0 MB
protobuf-3.0.0 | py27_0 407 KB
pbr-1.10.0 | py27_0 100 KB
mock-2.0.0 | py27_0 96 KB
tensorflow-0.10.0rc0 | np111py27_0 25.0 MB
------------------------------------------------------------
Total: 31.7 MB
The following NEW packages will be INSTALLED:
conda-env: 2.6.0-0
libprotobuf: 3.0.0-0
mock: 2.0.0-py27_0
pbr: 1.10.0-py27_0
protobuf: 3.0.0-py27_0
tensorflow: 0.10.0rc0-np111py27_0
Manually install all required packages (with conda). Do not install the version conda wants you to install (it's only CPU enabled).
Below path can be found at https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html#pip-installation, so use appropriate whl file.
$ pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0rc0-cp27-none-linux_x86_64.whl
That should be it. I got tf.test.is_built_with_cuda() test from the page below: https://www.tensorflow.org/versions/r0.8/api_docs/python/test.html
Please let me know if you are having any issues.
Thank you,
Asya
Reply all
Reply to author
Forward
0 new messages