New to ubunrtu - Install problems (missing dependencies; unsure how to configure makefile.config)

[Sage Lee]

Hello - I think these kinds of questions are supposed to go here; pardon me if I have been misdirected.

I am brand new to Linux and Ubuntu and pretty much installed it just for the purposes of running Caffe.  I have Ubuntu 16.04, and a GTX 1080 ti.  After many failures and Ubuntu breakages I have successfully managed to install Ubuntu, cuda 8.0, cudnn (for cuda 8), and after a few snafus, everything seems to be detected and matched and tests were passed with flying colors.  So I installed Anaconda with Python 3.6.3 (please don't make me learn Python 2...), and, after confirming it worked, moved onto installing Caffe.

I've done my best to follow whatever walkthroughs/solutions I can find, but finally I am at a standstill.  When I go into caffe-rc5 (put it in /usr/local) and "make all", I get the following error:

PROTOC src/caffe/proto/caffe.proto
CXX .build_release/src/caffe/proto/caffe.pb.cc
CXX src/caffe/util/hdf5.cpp
In file included from ./include/caffe/blob.hpp:8:0,
                 from ./include/caffe/util/hdf5.hpp:9,
                 from src/caffe/util/hdf5.cpp:1:
./include/caffe/common.hpp:5:27: fatal error: gflags/gflags.h: No such file or directory
compilation terminated.
Makefile:582: recipe for target '.build_release/src/caffe/util/hdf5.o' failed
make: *** [.build_release/src/caffe/util/hdf5.o] Error 1

I understand this means I am missing some dependencies; I tried to follow this guide

http://olivernina.blogspot.com/2014/09/includecaffecommonhpp527-fatal-error.html

which says to write

wget https://github.com/schuhschuh/gflags/archive/master.zip
unzip master.zip
cd gflags-master
mkdir build && cd build
export CXXFLAGS="-fPIC" && cmake .. && make VERBOSE=1
make 
sudo make install

but when I get to the export line I get this error message:

___________

CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineSystem.cmake:184 (file):
  file failed to open for writing (No such file or directory):

    /usr/local/gflags-master/build/CMakeFiles/CMakeOutput.log
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


CMake Error: Could not open file for write in copy operation /usr/local/gflags-master/build/CMakeFiles/3.5.1/CMakeSystem.cmake.tmp
CMake Error: : System Error: No such file or directory
CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineSystem.cmake:196 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:141 (file):
  file problem creating directory:
  /usr/local/gflags-master/build/CMakeFiles/3.5.1/CompilerIdCXX
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:40 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /usr/share/cmake-3.5/Modules/CMakeDetermineCXXCompiler.cmake:113 (CMAKE_DETERMINE_COMPILER_ID)
  CMakeLists.txt:93 (project)


CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:141 (file):
  file problem creating directory:
  /usr/local/gflags-master/build/CMakeFiles/3.5.1/CompilerIdCXX
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:40 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /usr/share/cmake-3.5/Modules/CMakeDetermineCXXCompiler.cmake:113 (CMAKE_DETERMINE_COMPILER_ID)
  CMakeLists.txt:93 (project)


CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:647 (file):
  file failed to open for writing (No such file or directory):

    /usr/local/gflags-master/build/CMakeFiles/CMakeError.log
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/CMakeDetermineCompilerId.cmake:55 (CMAKE_DETERMINE_COMPILER_ID_VENDOR)
  /usr/share/cmake-3.5/Modules/CMakeDetermineCXXCompiler.cmake:113 (CMAKE_DETERMINE_COMPILER_ID)
  CMakeLists.txt:93 (project)


-- The CXX compiler identification is unknown
CMake Error: Could not open file for write in copy operation /usr/local/gflags-master/build/CMakeFiles/3.5.1/CMakeCXXCompiler.cmake.tmp
CMake Error: : System Error: No such file or directory
CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineCXXCompiler.cmake:179 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


CMake Error: Could not find cmake module file: /usr/local/gflags-master/build/CMakeFiles/3.5.1/CMakeCXXCompiler.cmake
-- Check for working CXX compiler: /usr/bin/c++
CMake Error at /usr/share/cmake-3.5/Modules/CMakeTestCXXCompiler.cmake:35 (file):
  file failed to open for writing (No such file or directory):

    /usr/local/gflags-master/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


CMake Error at /usr/share/cmake-3.5/Modules/CMakeTestCXXCompiler.cmake:40 (try_compile):
  Unknown extension ".cxx" for file

    /usr/local/gflags-master/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx

  try_compile() works only for enabled languages.  Currently these are:

    CXX

  See project() command to enable other languages.
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


-- Check for working CXX compiler: /usr/bin/c++ -- broken
CMake Error at /usr/share/cmake-3.5/Modules/CMakeTestCXXCompiler.cmake:51 (file):
  file failed to open for writing (No such file or directory):

    /usr/local/gflags-master/build/CMakeFiles/CMakeError.log
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


CMake Error at /usr/share/cmake-3.5/Modules/CMakeTestCXXCompiler.cmake:54 (message):
  The C++ compiler "/usr/bin/c++" is not able to compile a simple test
  program.

  It fails with the following output:

  

 

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:93 (project)


-- Configuring incomplete, errors occurred!
CMake Error: Cannot open file for write: /usr/local/gflags-master/build/CMakeCache.txt.tmp
CMake Error: : System Error: Permission denied
CMake Error: Unable to open cache file for save. /usr/local/gflags-master/build/CMakeCache.txt
CMake Error: : System Error: Permission denied

________________________

By now I'm used to sudoing my way over permission obstacles, since I've been trying to put everything in /usr/local per best practices advice, but in this case it doesn't work since we can't sudo export.  I tried to follow a few more walkthroughs/explanations about something something environment variables, but I'm too new to all of this and most of it is Greek to me.

If it matters I also built openCV using this

https://github.com/BVLC/caffe/wiki/OpenCV-3.3-Installation-Guide-on-Ubuntu-16.04

and am trying to follow this

https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-Installation-Guide

But in both places they talk about "entering the unpacked directory" and I don't know how to do that.  When I try to cd to master.zip (my best guess when they refer to caffe-master unpacked directory), it just tells me it's not a directory.

I am also unsure how to configure my makefile.config for my setup.  I've done my best and finally just used the makefile.config example file provided with the Cmake instructions in the link(s) above, but I'm still unsure.  Finally, in one of the openCV install tutorials, virtual environments were highly recommended, but i was unable to succeed in installing those as well.  Pretty sure it's a path problem (tried adding something to bashrc to fix this but getting error message:

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/home/swerve/anaconda3/bin/python and that PATH is
set properly.


Any chance I can get some very noob-friendly help with this problem here?  I'm in way over my head here but I'm learning, slowly, and really need to get Caffe up and running ASAP for a project.  I'm not sure if that's too much or not enough information, so please let me know what else I can do to help this along. 

Thanks in advance for any help you can provide. 

[Thomio Watanabe]

Suggestions:
- follow instructions from website:
http://caffe.berkeleyvision.org/installation.html
- use python2
- install gflags from ubuntu repositories
- try cmake instead makefile.

[Sage Lee]

Hi, thanks a lot for the response.

- Of course I have been trying to follow the instructions from Berkeley.  This was my starting point.  But not everything there has been exactly clear or reproducible for my own software/hardware choices.

- I'm not sure how to just "install something from an Ubuntu repository".  Note that I have been using Linux for about a week and the various install methods I've encountered are still very strange and unfamiliar to me, and apparently there are a lot of them.  There is apt-get, there is the --install command, I've seen one with just an -i, there are ppa commands and runfiles and .deb files, oh my.  I googled and found a gflags .deb file, but there is also a warning there about doing it manually; it says to try a package manager instead.  (Note that my default Ubuntu Software program has an apparently known glitch where it doesn't install anything and just hangs, so I've had to do everything manually.)  I'll explore the package manager links I found but I'd really love some advice on the best way to do this and don't want to bloat my system with extra things I don't need if I don't really need them.  I apologize though that I'm too much of a noob to be able to hear "go install this" and have a clue on the best way to do that, as I've been using Windows pretty much all of my life (give me an .exe file to double click on and I'm golden!)

- I also did already use Cmake.  I found a tutorial using Cmake that provided an example makeFile.config; I referenced this in my above post.  But that example file included "tweaking" instructions for Anaconda Python 3.5; I'm trying to use 3.6.3.  I've done my best to make necessary adjustments, and for all I know I got that part right, but there are several "tweaks" needed even given the example file, and I can't be sure I did them right.  I was hoping somebody could help look over that file in particular but I thought I'd make sure someone was willing first before I started posting every potential problem file here.

- Python 2 is the one thing I do not want to do.  I can't stress this enough.  I want to be able to run PyCharm with Anaconda and Python 3.6.3, and import pycaffe to use Caffe from that environment; this is the one steady rock I desperately want to have in this very unfamiliar sea.  I have yet to be told it's impossible but I also have yet to find explicit updated instructions to help me along.  I keep having to deviate somewhat from tutorial steps but apparently keep guessing wrong as to the adjustments I need to make on my own to make that setup work.


________

[Sage Lee]

Ok, I went back over the error messages carefully and I think I was able to fix/install everything.  At least up to compiling Caffe... with Python 2.7  Arrgh!  I don't know why it grabbed 2.7; when type

python --version

I get back Anaconda 3.6.3.  I see in the Dependencies.cmake file that it checks my python version and loads up 2.7 if it doesn't find it, but when I try to

cmake -D python_version=3

it can't find the python 3 versions of things.  Even though if I check manually, all the python3.5 versions of the 2.7 files that it finds are there in the same directories.  In an earlier attempt at using a make all tutorial, I carefully edited the Makefile.config file so that all of the python_lib and such paths pointed to the correct python3 versions of the files instead of the python2.  But I'm not sure if I did this wrong, or if using the above cmake command is using it's own config and ignoring that file.  If it matters, here is what I did to the relevant section of the Makefile.config file:

______________

[Sage Lee]

Oops, accidentally posted without pasting the code.  Don't see a place to edit posts...?

What i was saying is that here is what i put into my Makefile.config file in /usr/local/caffe-rc5:
________________________

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#        /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda3
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
        $(ANACONDA_HOME)/include/python3.6m \
        $(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.6m
PYTHON_LIBRARIES := /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0
PYTHON_INCLUDE := /usr/include/python3.6m \
                 /usr/lib/python3.6/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

_____________________________________

I'm not sure if there's a problem there, or if the cmake method I'm now using is ignoring this file.  I tried searching through all the cmake/make/config type files i could find in the Caffe directories, but I'm not see any lines that look like the above to try to similarly change.  Could someone tell me if the above looks right, or if there's some other file somewhere I need to change to get the cmake -D python_version=3 .. command to work?  Thanks.

[Sage Lee]

Progress, but slow progress:

I cleared out all the folders and redownloaded caffe to try again.  This time the cmake command did work with Python 3.5.2. 

I was under the impression that cmake was an alternative command for make; this appears not to be the case as I still have to do the command

make all

according to this python 3 -specific tutorial I'm trying to follow.:

https://yangcha.github.io/Caffe-Conda3/


I had to manually create build/include/caffe/proto and use

sudo mv src/caffe/proto/caffe.pb.h build/include/caffe/proto

to get past one error, but now I'm getting another one (on the make all step):

______________________

[  0%] Running C++/Python protocol buffer compiler on /usr/local/caffe-rc5/src/caffe/proto/caffe.proto
Scanning dependencies of target proto
[  0%] Building CXX object src/caffe/CMakeFiles/proto.dir/__/__/include/caffe/proto/caffe.pb.cc.o
[  1%] Linking CXX static library ../../lib/libproto.a
[  1%] Built target proto
[  1%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_log_layer.cu.o
/usr/local/caffe-rc5/include/caffe/util/cudnn.hpp(112): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00007548_00000000-5_log_layer.cpp4.ii".
CMake Error at cuda_compile_generated_log_layer.cu.o.cmake:266 (message):
  Error generating file
  /usr/local/caffe-rc5/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_log_layer.cu.o


src/caffe/CMakeFiles/caffe.dir/build.make:469: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_log_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_log_layer.cu.o] Error 1
CMakeFiles/Makefile2:272: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
________________________________________

The error appears to be "too few arguments in fucntion call" and I hve no idea what it's referring to.

[Sage Lee]

Whew.  Ok, so the problem was that i had the newest cudnn when only 5 is supported.  So I changed to version 5 and now I've successfully compiled, installed, and run a successful make testrun with caffe.  i have another problem now with pycaffe, but I'll mark this as completed as open a new thread since I've blabbed a bit in here.