Bob on Windows

[Ali Kh]

Hi,

I made some progress on bob dependencies. Here is the current state:

Starting at https://gitlab.idiap.ch/bob/bob.conda/blob/master/dependencies/bob-devel/meta.yaml:

Many already exist on conda-forge: (some older version, but this is not to be worried at the current stage, as it is easy to fix later I suppose!)

conda install numpy click curl cython cytoolz dask freetype hdf5 imagesize ipython jpeg libpng libtiff matplotlib mkl nose numba numexpr pandas pillow pip scikit-image scikit-learn scipy=0.19.1 setuptools six sphinx sqlalchemy toolz click-plugins cmake coverage cyvlfeat dlib docopt ffmpeg ipdb keras libogg libsvm opencv pyedflib schema sox sphinx_rtd_theme tensorflow virtualenv vlfeat boost pyzmq

Remaining:

caffe giflib kaldi libblitz libmatio madmom pkg-config pkgtools zc.buildout menpofit mr.developer menpo zc.recipe.egg menpowidgets mne

With the help of pip these go away:

madmom pkgtools zc.buildout mr.developer mne menpo menpowidgets menpofit

There is a comment stating #linux for the following, so I suppose I can ignore them?!

caffe kaldi

Remaining:

giflib libblitz libmatio pkg-config pkgtools zc.recipe.egg

Compiling the remaining:

requirement:

Microsoft Visual C++ Build Tools 2015

libblitz and libmatio can be compiled with msbuild after retargeting easily...

Currently working on the rest.

As I am not a windows person myself, and I am not familiar with conda in general, I really appreciate if I could get any feedback/contribution.

Sincerely,

Alikh

[Amir Mohammadi]

Hi Ali,

bob-devel contains all possible packages that we use here at Idiap.

To get Bob, you only need to install these dependencies:
https://www.idiap.ch/software/bob/docs/bob/bob/stable/source.html#dependencies

So you can definitely ignore the packages from bob-devel that are not in the link above.

I suggest you target only Python 3.6 so you can use a newer visual studio (2015).

Conda comes with extensive documentation. So if you want to install Bob, maybe you can

spare some time to learn conda: https://conda.io/docs/ ?

Tutorials on how to create recipes for packages and build them exist:
https://conda.io/docs/user-guide/tutorials/index.html

One for Windows specifically: https://conda.io/docs/user-guide/tutorials/build-windows.html

And the reference for building with conda is extensively documented here:
https://conda.io/docs/user-guide/tasks/build-packages/index.html

Of course, conda-forge has its additional documentation too:
https://conda-forge.org/docs/

Maybe you can use pkg-config from here:
https://anaconda.org/search?q=platform%3Awin-64+pkg-config

pkgtools and zc.recipe.egg are pip packages.

So looks like you just need to install giflib on Windows.

Once you have libblitz, libmatio, and giflib installed locally,

the best option is to update their packages in conda-forge

so you can conda install those too.

Good luck,

Amir

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ali Kh]

Hi Amir,

Thank you for your response. I am trying to make bob work on win64, python 3.6.

Unfortunately, I do not have enough time to build recipes but I will be more than happy to give support if needed.

Edit:

pkgtools was installed with pip already. I put it in remaining by mistake.

for zc.recipe.egg :

pip install --no-binary=zc.recipe.egg zc.recipe.egg

Does the job

Update:

giflib with VS solution is available here and compiles without errors:

http://www.chemformatter.com/build-with-visual-studio-2015/giflib

Then I should have everything I need.

Best,

Alikh 

[Amir Mohammadi]

Hi Ali,

You may think that building only the remaining 3 packages
without conda will save you time but it will just postpone to
another time.

If you have all the dependencies installed, then this guide:
https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/development.html

should help you compile Bob with buildout.

Best,

Amir

[Ali Kh]

Thanks to Amir, the packages are built for Conda and are accessible through the following link:

https://www.dropbox.com/sh/nsw11jxenst3d6z/AAAg4EUGm9PjLBSoIOmQyd9za?dl=0 

They can be installed with the following command:

conda install whatever.tar.bz2

Now moving to the next step, following the install order in pip:

pip install bob.extension

runs with no errors

pip install bob.blitz

Unable to find boost's "version.hpp"

Collecting bob.blitz
  Using cached bob.blitz-2.0.14.zip
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "~\AppData\Local\Temp\pip-build-44s54025\bob.blitz\setup.py", line 70, in <module>
        system_include_dirs=system_include_dirs,
      File "~\Playground\BobOnWin\Conda\lib\site-packages\bob\extension\__init__.py", line 356, in __init__
        boost_pkg = boost(boost_req.replace('boost', '').strip())
      File "~\Playground\BobOnWin\Conda\lib\site-packages\bob\extension\boost.py", line 69, in __init__
        raise RuntimeError("could not find boost's `version.hpp' - have you installed Boost on this machine?")
    RuntimeError: could not find boost's `version.hpp' - have you installed Boost on this machine?


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in ~\AppData\Local\Temp\pip-build-44s54025\bob.blitz\

Best,

Ali Khodabakhsh

On Thursday, January 25, 2018 at 2:07:17 PM UTC+1, Ali Kh wrote:

[Ali Kh]

Update:

Assuming being inside a conda environment, adding the following at line 165 of bob.extension\utils.py fixes the previous error:

  # Windows
  headerpaths += [os.path.join('Library','include')]

And now for something completely different!

Traceback (most recent call last):

  File "~\AppData\Local\Temp\tmpdzewo_qo", line 14, in <module>
    exec(compile(open('~\\bob_source\\src/bob.blitz\\setup.py').read(), '~\\bob_source\\src/bob.blitz\\setup.py', 'exec'))
  File "~\bob_source\src/bob.blitz\setup.py", line 70, in <module>
    system_include_dirs=system_include_dirs,
  File "~\bob_source\src\bob.extension\bob\extension\__init__.py", line 384, in __init__
    pkgs = check_packages(packages)
  File "~\bob_source\src\bob.extension\bob\extension\__init__.py", line 73, in check_packages
    p = pkgconfig(splitreq[0])
  File "~\bob_source\src\bob.extension\bob\extension\pkgconfig.py", line 133, in __init__
    status, stdout, stderr = call_pkgconfig(['--modversion', name], paths)
  File "~\bob_source\src\bob.extension\bob\extension\pkgconfig.py", line 60, in call_pkgconfig
    stdout=subprocess.PIPE
  File "~\conda\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "~\conda\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
PermissionError: [WinError 5] Access is denied

On Thursday, January 25, 2018 at 2:07:17 PM UTC+1, Ali Kh wrote:

[Amir Mohammadi]

Hi Ali,

It is STRONGLY encouraged that you compile bob using buildout: https://www.idiap.ch/software/bob/docs/bob/bob.extension/master/development.html
And the list of core bob packages are here:
https://gitlab.idiap.ch/bob/bob.nightlies/blob/master/core.txt

And your error seems to be related to calling pkg-config.
If you try calling pkg-config in a normal terminal, does it work?

Thanks,
Amir

[Ali Kh]

Dear Amir,

Thank you for your concern. There was a problem with library/header/executable directory definition in bob.extension/utils.py that was solved.

The current situation is that I don't have pkg-config file for blitz (blitz.pc). Do we need pkg-config files in windows?

Best,

Ali Kh

[Ali Kh]

Update:

following changes solves many problems:

fc /N src\bob.extension\bob\extension\utils.py git-diff\bob.extension\bob\extension\utils.py

Comparing files SRC\BOB.EXTENSION\BOB\EXTENSION\utils.py and GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\UTILS.PY

***** SRC\BOB.EXTENSION\BOB\EXTENSION\utils.py

  116:

  117:    print(search)

  118:

  119:    retval = []

***** GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\UTILS.PY

  116:

  117:    retval = []

*****

***** SRC\BOB.EXTENSION\BOB\EXTENSION\utils.py

  166:

  167:    # Windows

  168:    if os.name == 'nt':

  169:      headerpaths += [os.path.join('Library','include')]

  170:

  171:    # Exhaustive combination of paths and subpaths

***** GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\UTILS.PY

  164:

  165:    # Exhaustive combination of paths and subpaths

*****

***** SRC\BOB.EXTENSION\BOB\EXTENSION\utils.py

  233:

  234:    # Windows

  235:    if os.name == 'nt':

  236:      libpaths += [os.path.join('Library','lib')]

  237:

  238:    # Exhaustive combination of paths and subpaths

***** GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\UTILS.PY

  227:

  228:    # Exhaustive combination of paths and subpaths

*****

***** SRC\BOB.EXTENSION\BOB\EXTENSION\utils.py

  317:

  318:    # Windows

  319:    if os.name == 'nt':

  320:      binpaths += [os.path.join('Library','bin'), os.path.join('Library','mingw-w64','bin')]

  321:      name = name + '.exe'

  322:

  323:    # Exhaustive combination of paths and subpaths

***** GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\UTILS.PY

  307:

  308:    # Exhaustive combination of paths and subpaths

*****

fc /N src\bob.extension\bob\extension\pkgconfig.py git-diff\bob.extension\bob\extension\pkgconfig.py
Comparing files SRC\BOB.EXTENSION\BOB\EXTENSION\pkgconfig.py and GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\PKGCONFIG.PY
***** SRC\BOB.EXTENSION\BOB\EXTENSION\pkgconfig.py
   53:
   54:    # windows
   55:    if os.name == 'nt':
   56:      env['PKG_CONFIG_PATH'] = os.path.join(env.get("CONDA_PREFIX", False),"Library","lib","pkgconfig")
   57:
   58:    # calls the program
***** GIT-DIFF\BOB.EXTENSION\BOB\EXTENSION\PKGCONFIG.PY
   53:
   54:    # calls the program
*****

+ adding blitz.pc manually: (at %CONDA_PREFIX%\Library\lib\pkgconfig)

# generated by configure / remove this line to disable regeneration
prefix=%CONDA_PREFIX%/Library
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include


Name: blitz
Description: blitz Library
Version: 0.10
Requires:
Libs: -L${libdir} -lblitz
Cflags: -I${includedir} -pthread

Now for the current error:

Traceback (most recent call last):

  File "~\AppData\Local\Temp\tmplz4_f6oq", line 14, in <module>

    exec(compile(open('~\\bob_source\\src/bob.blitz\\setup.py').read(), '~\\bob_source\\src/bob.blitz\\setup.py', 'exec'))

  File "!\bob_source\src/bob.blitz\setup.py", line 99, in <module>
    'Topic :: Software Development :: Libraries :: Python Modules',
  File "!\conda\lib\site-packages\setuptools\__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "!\conda\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "!\conda\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "!\conda\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "!\conda\lib\site-packages\setuptools\command\develop.py", line 36, in run
    self.install_for_development()
  File "!\conda\lib\site-packages\setuptools\command\develop.py", line 136, in install_for_development
    self.run_command('build_ext')
  File "!\conda\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "!\conda\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "!\bob_source\src\bob.extension\bob\extension\__init__.py", line 638, in run
    return _build_ext.run(self)
  File "!\conda\lib\site-packages\setuptools\command\build_ext.py", line 75, in run
    _build_ext.run(self)
  File "!\conda\lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "!\conda\lib\distutils\command\build_ext.py", line 339, in run
    self.build_extensions()
  File "!\conda\lib\site-packages\Cython\Distutils\old_build_ext.py", line 194, in build_extensions
    self.build_extension(ext)
  File "!\bob_source\src\bob.extension\bob\extension\__init__.py", line 649, in build_extension
    self.compiler.compiler = [c for c in self.compiler.compiler if c != "-Wstrict-prototypes"]
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

I have no idea how to deal with this!

Best,

Ali Kh

On Thursday, January 25, 2018 at 2:07:17 PM UTC+1, Ali Kh wrote:

[Amir Mohammadi]

Hi Ali,

Have you made any progress in compiling Bob without pkg-config on Windows?

@Andre, @Manuel,
How can we avoid pkg-config when compiling Bob on Windows?

Assuming all libraries are within a known location of conda (CONDA_PREFIX\Library\{lib,include}),

Is it possible to skip pkg-config all together?

Thanks,

Amir

[Ali Kh]

Hi Amir,

Unfortunately, I have been quite busy this week and I could not make any progress.

I also need to make myself familiar with cmake. I will upload the code on github mirror if I make any progress.

Best,

Ali

[André Anjos]

Hello,

Not sure what the issue is with pkg-config - please let me know. That should run on Windows: https://www.freedesktop.org/wiki/Software/pkg-config/

The information provided by pkg-config concerns compilation flags, compile macros and other which is necessary to incorporate a given library into the build. Sure, one can avoid this by hard-coding everything, but I definitely don't advise that.

A

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. André Anjos
Idiap Research Institute
Centre du Parc - rue Marconi 19
CH-1920 Martigny, Suisse
Phone: +41277217763
http://andreanjos.org

[Ali Kh]

Thank you for your response. pkg-config is already there in Conda, but the .pc files are not commonly generated for Conda packages in windows.

Then you suggest we need to generate pkg-config files for all the dependencies?

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[André Anjos]

Yes I do. A

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Amir Mohammadi]

None of Bob dependencies coming from the defaults channel or conda-forge contain pkg-config files on Windows.

I keep wondering if pkg-config is not really required on Windows?!

Amir

[André Anjos]

I'm not knowledgeable in Windows programming. If you have any other programmatic way to retrieve *all* required information provided by pkg-config (not only include directories), please let us know.

A

Yes I do. A

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. André Anjos
Idiap Research Institute
Centre du Parc - rue Marconi 19
CH-1920 Martigny, Suisse
Phone: +41277217763
http://andreanjos.org

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/

---
You received this message because you are subscribed to the Google Groups "bob-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Amir Mohammadi]

When I run try to compile bob packages with cmake (e.g. bob.core), I see log lines like:

copying build/lib.linux-x86_64-3.6/bob/core/libbob_core.so -> bob/core                         
copying build/lib.linux-x86_64-3.6/bob/core/version.cpython-36m-x86_64-linux-gnu.so -> bob/core              
copying build/lib.linux-x86_64-3.6/bob/core/_convert.cpython-36m-x86_64-linux-gnu.so -> bob/core
copying build/lib.linux-x86_64-3.6/bob/core/_logging.cpython-36m-x86_64-linux-gnu.so -> bob/core                     
copying build/lib.linux-x86_64-3.6/bob/core/random/_library.cpython-36m-x86_64-linux-gnu.so -> bob/core/random
copying build/lib.linux-x86_64-3.6/bob/core/_test.cpython-36m-x86_64-linux-gnu.so -> bob/core

This part does not work on Windows somehow.

Would you know where is the implementation of this so that we can fix it in Windows?

We are getting very close to Windows builds of Bob. See:
https://github.com/bioidiap/bob.extension/pull/1
https://github.com/bioidiap/bob.blitz/pull/1
https://github.com/bioidiap/bob.core/pull/1

Thanks,

Amir

[Manuel Günther]

Amir,

the pure C++ libraries are generated in `bob.extension`. Particularly, the cmake build file (CMakeLists.txt) is written in https://gitlab.idiap.ch/bob/bob.extension/blob/d2f1140236fbd3e19fc37897c0c252067ca5c4cb/bob/extension/cmake.py#L94

So far, Unix-specific CMakeLists.txt files are written, it is possible that you need to adapt, e.g., the 'CFLAGS' and 'CXXFLAGS' to be Windows-specific files.

I am not sure, why the compiler/linker produces .so files. On Windows systems, it should generate .dll files automatically. Maybe the compiler setup is incorrect. Which compiler are you using? I cannot see that from your shortened compiler output. If it uses gcc, then you'd probably need to figure out how to use a Windows-specific compiler such as icc.

I hope this helped a little.

Manuel

[Amir Mohammadi]

Hi,

Thank you for your reply. The log I showed is from Linux not Windows.

The error that I get on Windows is:

bob.core>python setup.py develop
running develop
running egg_info
writing bob.core.egg-info\PKG-INFO
writing dependency_links to bob.core.egg-info\dependency_links.txt
writing requirements to bob.core.egg-info\requires.txt
writing top-level names to bob.core.egg-info\top_level.txt
reading manifest file 'bob.core.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bob.core.egg-info\SOURCES.txt'
running build_ext
Re-run cmake no build system arguments
-- Selecting Windows SDK version  to target Windows 10.0.16299.
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/Amir/Documents/bob/src/bob.core/build/build_cmake/bob_core
Calling the command: ['cmake', '--build', '.', '--target', 'ALL_BUILD', '--config', 'Release']
Microsoft (R) Build Engine version 14.0.25420.1
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 13/03/2018 16:35:05.
Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ALL_BUILD.vcxproj" on node 1 (default targ
ets).
Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ALL_BUILD.vcxproj" (1) is building "C:\Use
rs\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ZERO_CHECK.vcxproj" (2) on node 1 (default targets).
InitializeBuildStatus:
  Creating "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  All outputs are up-to-date.
FinalizeBuildStatus:
  Deleting file "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ZERO_CHECK.vcxproj" (default
 targets).

Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ALL_BUILD.vcxproj" (1) is building "C:\Use
rs\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\bob_core.vcxproj" (3) on node 1 (default targets).
InitializeBuildStatus:
  Touching "bob_core.dir\Release\bob_core.tlog\unsuccessfulbuild".
CustomBuild:
  All outputs are up-to-date.
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\CL.exe /c /IC:\Users\Amir\Documents\bob\src\bob.
  core\bob\core\include /Ic:\users\amir\documents\bob\src\bob.blitz\bob\blitz\include /I"C:\Users\Amir\Conda\envs\bob\l
  ib\site-packages\numpy\core\include" /IC:\Users\Amir\Conda\envs\bob\Library\include /nologo /W1 /WX- /O2 /Ob2 /D _WIN
  32_WINNT=0x501 /D NDEBUG /D PY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API /D NO_IMPORT_ARRAY=1 /D NPY_NO_DEPRECATED_API=NPY_
  1_7_API_VERSION /D HAVE_ZLIB=1 /D "ZLIB_VERSION=\"1.2.11\"" /D HAVE_BLITZ=1 /D "BLITZ_VERSION=\"0.10\"" /D "CMAKE_INT
  DIR=\"Release\"" /D _MBCS /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"bob_core.dir\Release
  \\" /Fd"bob_core.dir\Release\bob_core.pdb" /Gd /TP /errorReport:queue  -Qunused-arguments -O3 -g0 -mtune=native C:\Us
  ers\Amir\Documents\bob\src\bob.core\bob\core\cpp\zlib.cpp C:\Users\Amir\Documents\bob\src\bob.core\bob\core\cpp\loggi
  ng.cpp
cl : Command line warning D9002: ignoring unknown option '-Qunused-arguments' [C:\Users\Amir\Documents\bob\src\bob.core
\build\build_cmake\bob_core\bob_core.vcxproj]
  zlib.cpp
cl : Command line warning D9002: ignoring unknown option '-O3' [C:\Users\Amir\Documents\bob\src\bob.core\build\build_cm
ake\bob_core\bob_core.vcxproj]
cl : Command line warning D9002: ignoring unknown option '-g0' [C:\Users\Amir\Documents\bob\src\bob.core\build\build_cm
ake\bob_core\bob_core.vcxproj]
cl : Command line warning D9002: ignoring unknown option '-mtune=native' [C:\Users\Amir\Documents\bob\src\bob.core\buil
d\build_cmake\bob_core\bob_core.vcxproj]
  logging.cpp
  Generating Code...
Lib:
  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\Lib.exe /OUT:"C:\Users\Amir\Documents\bob\src\bo
  b.core\build\build_cmake\bob_core\Release\bob_core.lib" /NOLOGO /MACHINE:X64  /machine:x64 bob_core.dir\Release\zlib.
  obj
  bob_core.dir\Release\logging.obj
  bob_core.vcxproj -> C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\Release\bob_core.lib
FinalizeBuildStatus:
  Deleting file "bob_core.dir\Release\bob_core.tlog\unsuccessfulbuild".
  Touching "bob_core.dir\Release\bob_core.tlog\bob_core.lastbuildstate".
Done Building Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\bob_core.vcxproj" (default t
argets).

PrepareForBuild:
  Creating directory "x64\Release\ALL_BUILD\".
  Creating directory "x64\Release\ALL_BUILD\ALL_BUILD.tlog\".
InitializeBuildStatus:
  Creating "x64\Release\ALL_BUILD\ALL_BUILD.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Building Custom Rule C:/Users/Amir/Documents/bob/src/bob.core/build/build_cmake/bob_core/CMakeLists.txt
  CMake does not need to re-run because C:/Users/Amir/Documents/bob/src/bob.core/build/build_cmake/bob_core/CMakeFiles/
  generate.stamp is up-to-date.
FinalizeBuildStatus:
  Deleting file "x64\Release\ALL_BUILD\ALL_BUILD.tlog\unsuccessfulbuild".
  Touching "x64\Release\ALL_BUILD\ALL_BUILD.tlog\ALL_BUILD.lastbuildstate".
Done Building Project "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ALL_BUILD.vcxproj" (default
targets).


Build succeeded.

"C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\ALL_BUILD.vcxproj" (default target) (1) ->
"C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\bob_core.vcxproj" (default target) (3) ->
(ClCompile target) ->
  cl : Command line warning D9002: ignoring unknown option '-Qunused-arguments' [C:\Users\Amir\Documents\bob\src\bob.co
re\build\build_cmake\bob_core\bob_core.vcxproj]
  cl : Command line warning D9002: ignoring unknown option '-O3' [C:\Users\Amir\Documents\bob\src\bob.core\build\build_
cmake\bob_core\bob_core.vcxproj]
  cl : Command line warning D9002: ignoring unknown option '-g0' [C:\Users\Amir\Documents\bob\src\bob.core\build\build_
cmake\bob_core\bob_core.vcxproj]
  cl : Command line warning D9002: ignoring unknown option '-mtune=native' [C:\Users\Amir\Documents\bob\src\bob.core\bu
ild\build_cmake\bob_core\bob_core.vcxproj]

    4 Warning(s)
    0 Error(s)

Time Elapsed 00:00:11.39
building 'bob.core.version' extension
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
creating build\temp.win-amd64-3.6\Release\bob
creating build\temp.win-amd64-3.6\Release\bob\core
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DBOB_EXT_MODULE_PREFIX=\"bob.core\" -DBOB_EXT_MODULE_NAME=\"version\" -DBOB_EXT_ENTRY_NAME=PyInit_version -DBOB_EXT_MODULE_VERSION=\"2.1.10b0\" -DHAVE_BOOST=1 -DBOOST_VERSION=\"1.65.1\" -DHAVE_ZLIB=1 -DZLIB_VERSION=\"1.2.11\" -DHAVE_BLITZ=1 -DBLITZ_VERSION=\"0.10\" -DPY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -IC:\Users\Amir\Documents\bob\src\bob.core\bob\core\include -Ic:\users\amir\documents\bob\src\bob.blitz\bob\blitz\include -Ic:\users\amir\documents\bob\src\bob.extension\bob\extension\include -IC:\Users\Amir\Conda\envs\bob\include -IC:\Users\Amir\Conda\envs\bob\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /EHsc /Tpbob/core/version.cpp /Fobuild\temp.win-amd64-3.6\Release\bob/core/version.obj -std=c++0x -pthread -IC:\Users\Amir\Conda\envs\bob\lib\site-packages\numpy\core\include -IC:\Users\Amir\Conda\envs\bob\Library\include -IC:/Users/Amir/Conda/envs/bob/Library/include
cl : Command line warning D9002 : ignoring unknown option '-std=c++0x'
cl : Command line warning D9002 : ignoring unknown option '-pthread'
version.cpp
C:\Users\Amir\Conda\envs\bob\Library\include\boost/version.hpp(22): warning C4005: 'BOOST_VERSION': macro redefinition
C:\Users\Amir\Conda\envs\bob\Library\include\boost/version.hpp(22): note: command-line arguments:  see previous definition of 'BOOST_VERSION'
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Amir\Documents\bob\src\bob.core\build\lib.win-amd64-3.6\bob\core /LIBPATH:C:\Users\Amir\Conda\envs\bob\Library\lib /LIBPATH:C:\Users\Amir\Conda\envs\bob\libs /LIBPATH:C:\Users\Amir\Conda\envs\bob\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64" bob_core.lib boost_system-vc140-mt-1_65_1.lib z.lib blitz.lib /EXPORT:PyInit_version build\temp.win-amd64-3.6\Release\bob/core/version.obj /OUT:build\lib.win-amd64-3.6\bob\core\version.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\bob/core\version.cp36-win_amd64.lib
LINK : fatal error LNK1181: cannot open input file 'bob_core.lib'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1181

Now this bob_core.lib exists. It's in "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\Release\bob_core.lib" but it should be in "/LIBPATH:C:\Users\Amir\Documents\bob\src\bob.core\build\lib.win-amd64-3.6\bob\core" for the linker to find it.

My understanding is that somehow something (bob.extension or cmake) copies this over in Linux but this does not work on Windows.

Who is supposed to copy this over? Or how is it that the linker is looking for it somewhere else?

My generated cmake is:

# WARNING! This file is automatically generated. Do not change its contents.

cmake_minimum_required(VERSION 2.8)
project(bob_core)

# For both C and C++
set(COMMON_FLAGS "-pedantic -Wall")
if (WIN32)
  set(COMMON_FLAGS "-D_WIN32_WINNT=0x501") # Set min. Windows version to XP
else(WIN32)
  set(COMMON_FLAGS "${COMMON_FLAGS} -pthread")
endif (WIN32)
if (NOT CMAKE_COMPILER_IS_GNUCC)
  # Then, it must be clang/clang++
  set(COMMON_FLAGS "${COMMON_FLAGS} -Qunused-arguments")
endif ()

# Force __LP64__ scheme on Mac OSX
if(APPLE)
  set(CMAKE_MACOSX_RPATH TRUE CACHE BOOL "Enables the MACOS_RPATH feature for MacOSX builds" FORCE)
  set(COMMON_FLAGS "${COMMON_FLAGS} -m64")
endif(APPLE)

# For both RELEASE and DEBUG builds
if(APPLE AND CMAKE_COMPILER_IS_GNUCC)
  if(CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.4")
    message(FATAL_ERROR "Minimum GCC version required on OSX is 4.4, but you have ${CMAKE_CXX_COMPILER_VERSION}")
  endif()
  set(COMMON_FLAGS "${COMMON_FLAGS} -Wno-long-long -Wno-variadic-macros")
  set(COMMON_CXX_FLAGS "-std=c++0x")
  set(COMMON_C_FLAGS "-std=c99")
elseif(WIN32)
  set(COMMON_CXX_FLAGS "/EHsc")
  set(COMMON_C_FLAGS "")
else()
  set(COMMON_CXX_FLAGS "-std=c++0x")
  set(COMMON_C_FLAGS "-std=c99")
endif()

# These are used in type checks for cmake, be aware and don't change those
set(CMAKE_CXX_FLAGS "${COMMON_CXX_FLAGS} ${COMMON_FLAGS} $ENV{CXXFLAGS} $ENV{CPPFLAGS}" CACHE STRING "Flags used by the compiler during release builds" FORCE)
set(CMAKE_C_FLAGS "${COMMON_C_FLAGS} ${COMMON_FLAGS} $ENV{CFLAGS} $ENV{CPPFLAGS}" CACHE STRING "Flags used by the compiler during release builds" FORCE)

set(BUILD_SHARED_LIBS "ON" CACHE BOOL "Build shared libs")

include_directories(C:\\Users\\Amir\\Documents\\bob\\src\\bob.core\\bob\\core\\include)
include_directories(c:\\users\\amir\\documents\\bob\\src\\bob.blitz\\bob\\blitz\\include)
include_directories(SYSTEM C:\\Users\\Amir\\Conda\\envs\\bob\\lib\\site-packages\\numpy\\core\\include)
include_directories(SYSTEM C:\\Users\\Amir\\Conda\\envs\\bob\\Library\\include)
link_directories(C:\\Users\\Amir\\Conda\\envs\\bob\\Library\\lib)
add_definitions(/DPY_ARRAY_UNIQUE_SYMBOL=BOB_NUMPY_C_API)
add_definitions(/DNO_IMPORT_ARRAY=1)
add_definitions(/DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION)
add_definitions(/DHAVE_ZLIB=1)
add_definitions(/DZLIB_VERSION="1.2.11")
add_definitions(/DHAVE_BLITZ=1)
add_definitions(/DBLITZ_VERSION="0.10")

add_library(${PROJECT_NAME} STATIC
    C:/Users/Amir/Documents/bob/src/bob.core/bob/core/cpp/zlib.cpp
    C:/Users/Amir/Documents/bob/src/bob.core/bob/core/cpp/logging.cpp
)
set_target_properties(${PROJECT_NAME} PROPERTIES POSITION_INDEPENDENT_CODE TRUE)
set_target_properties(${PROJECT_NAME} PROPERTIES LIBRARY_OUTPUT_DIRECTORY C:/Users/Amir/Documents/bob/src/bob.core/build/lib.win-amd64-3.6/bob/core)

target_link_libraries(${PROJECT_NAME} boost_system-vc140-mt-1_65_1 boost_iostreams-vc140-mt-1_65_1 boost_filesystem-vc140-mt-1_65_1 z blitz)

Amir

--

[Manuel Günther]

As far as I remember, the copying is usually done by the python build system. It is some time ago that I wrote this piece of code. The compilation via cmake actually happens in here: https://gitlab.idiap.ch/bob/bob.extension/blob/d2f1140236fbd3e19fc37897c0c252067ca5c4cb/bob/extension/__init__.py#L626 I cannot see anything with respect to copying of files in that function.

I know that the build/linking system between Linux and Windows is very different. For Linux, the .so file contains all required information to be able to link against it. On Windows, there exist two files, the .lib that is required for the linker, and the .dll that is loaded on runtime. 

It is quite possible that you need some specific cmake commands in order to build .dll files on Windows. Though I have done that, this is more than 15 years ago and I do not remember the details, and I do not possess the sources of that project anymore.

To avoid the compiler warnings about unused flags, you need to change the code inside https://gitlab.idiap.ch/bob/bob.extension/blob/d2f1140236fbd3e19fc37897c0c252067ca5c4cb/bob/extension/cmake.py#L98 to only set these environmental variables on Windows systems (you might need some others on Windows, though). This does not solve your main issue, though.

Manuel

[Amir Mohammadi]

I may have misled you with my failure to understand the real problem.

The main question here is how can fix this error that I showed? It could because of anything.

Again, this bob_core.lib exists. It's in "C:\Users\Amir\Documents\bob\src\bob.core\build\build_cmake\bob_core\Release\bob_core.lib" but it should be in "/LIBPATH:C:\Users\Amir\Documents\bob\src\bob.core\build\lib.win-amd64-3.6\bob\core" for the linker to find it.

Maybe this file is supposed to be put there in the first place and there is no copying? (which I think is the case now that I look more closely)

Is it because of the way I invoke cmake: cmake --build . --target ALL_BUILD --config Release

We have done a quite number of changes in bob.extension to make it work on Windows:
https://github.com/bioidiap/bob.extension/pull/1/files

As you said we need to build the .dll files too but we already know how to do that with cmake.

As for compiler warnings, we are not worried about them as of now since they look like minor issues.

The main issue here is that the build will not continue unless I copy bob_core.lib manually in the folder that the linker is looking for it.

Thanks,

Amir

--

[Manuel Günther]

Yes, I understood that this is your problem, but I do not have a definite solution at hand.

One possible issue for the linker might be this line of code, which sets the output directory of the target: https://gitlab.idiap.ch/bob/bob.extension/blob/d2f1140236fbd3e19fc37897c0c252067ca5c4cb/bob/extension/cmake.py#L129. Instead of doing this, you might need to add an INSTALL cmake command, and then call "cmake install" after "cmake". 

Otherwise, I am sure that there are options in cmake to tell the linker where to find additional files. Maybe you'll need to add the directory that is set in line #129, too.