PyTorch module cannot be install correctly

[notfish the]

Hi all,

Recently I am trying to install plumed-dev to use the pytorch module for slow mode searching. I have followed the instruction on "PLUMED: PYTORCH (Machine Learning Collective Variables)" using the command 

"./configure --prefix=/my/work/dir/dp-dev --enable-modules=+crystallization:+pytorch:+annfunc:+opes:+ves --enable-libtorch". 

It seemed that no error occurs at the installing and compiling process. The plumed itself works well. However, when I check the directory /my_dir/include/plumed, no pytorch folder has been found. 

Does anyone know what's the problem? Thank you very much!

Best regards,

Taoyuze Lyu

[Luigi Bonati]

Dear Taoyuze,
did you follow the instructions on the page you listed regarding downloading LibTorch and exporting environment variables?

If while configuring PLUMED it can't find the library or fails to link it correctly it will give the following warning (and not an error):

configure: WARNING: cannot enable __PLUMED_HAS_LIBTORCH

Otherwise, it will display the following line:

checking libtorch with -lc10... yes

To see the reasons for the error you need instead to look in the config.log. In particular, check for the output of the abovementioned command (checking libtorch with -lc10). 

If the libraries are correctly linked, the reason for the failure could be that the version of Pytorch you have downloaded is not compatible with your system, and you need to download the ABI Pre-cxx11 one from the Pytorch website (remember to disable the corresponding flag in the configure line as reported in the PLUMED: PYTORCH page). If this does not solve the issue you can try sharing the config.log file. 

Best,

---------------------------------------------------------

Luigi Bonati

Postdoctoral Researcher

Atomistic Simulations

Italian Institute of Technology, Genova, Italy

---------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "PLUMED users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plumed-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/plumed-users/92574872-7cb7-44a0-b749-86166d3bb754n%40googlegroups.com.

[notfish the]

Hi Luigi,

Thank you for your reply! I have built the two versions of Libtorch and checked the output in the config.log files and they said "configure:9652: result: yes". Attached are the two config.log files. I don't know whether they contain the information needed.

Thank you so much for your help!

Best regards,

Taoyuze Lyu

[Luigi Bonati]

From your logs, it seems that it is correctly installed and enabled.  To be sure that everything is working you can execute the regtests:

cd regtest; make pytorch

Note that there are no headers file in plumed/src/pytorch,  hence no subfolder is created in plumed/include/

Best,

---------------------------------------------------------

Luigi Bonati

Postdoctoral Researcher

Atomistic Simulations

Italian Institute of Technology, Genova, Italy

---------------------------------------------------------

To view this discussion on the web visit https://groups.google.com/d/msgid/plumed-users/2ebbfb61-c158-4f55-b0b0-c159f4a01428n%40googlegroups.com.

[notfish the]

many thanks!

Plumed is working properly

[Francisco Carrascoza]

Hello,

I have encountered an error when trying to compile PLUMED 2.8 with libtorch. I'm working on a cluster node. Full details of the architecture are  attached.

I was able to successfully compile PLUMED with libtorch in laptops and PC's. With both, GNU and Intel compilers.  But when trying to do it on a Intel Xeon Platinum,  I get:

checking whether mpicxx accepts -std=c++14... no
checking libtorch without extra libs... no
checking libtorch with -ltorch_cpu... no
checking libtorch with -lc10... no

configure: WARNING: cannot enable __PLUMED_HAS_LIBTORCH

The rest of the compilation  can be successful, but without libtorch. 

I'm attaching below all relevant logs, I think, if you need more information, please let me know.

Regards,

Francisco

[Luigi Bonati]

Dear Francisco,
when installing LibTorch on clusters sometimes you need to download the ABI Pre-cxx11 version. Have you already tried it?

Since the error occurs during configuration, I would also need the config.log file to figure out the reason.

Best, 

---------------------------------------------------------

Luigi Bonati

Postdoctoral Researcher

Atomistic Simulations

Italian Institute of Technology, Genova, Italy

---------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "PLUMED users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plumed-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/plumed-users/61880bbf-bf6f-4c6a-a420-68e3986b22fbn%40googlegroups.com.

[Francisco Carrascoza]

Thank you Luigi for your reply.

Yes, I tried with the ABI Pre-cxx11 version  with its corresponding flag CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0", but no lucky. It fails the same.

The log file is quite big (24MB).  I'am attaching you a link to my google drive for you download it.  When I try to upload here that file, the browser crashes.

https://drive.google.com/file/d/1wOO0_nldWBOFsX8Xr9V3BNZMJuKgqBju/view?usp=sharing

So, here is the link I will try again to upload the file directly.

Regards,

Francisco

[Francisco Carrascoza]

No, I cannot add the original config.log here, it is too big for Google Groups.  However I copy paste a fragment of it, where it is the very last printing of the error:

----------------------------------------------------------------------------------------------------------------

In file included from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/library.h:63:0,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/runtime/operator.h:13,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/ir/ir.h:7,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/api/function_impl.h:4,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/api/method.h:5,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/api/object.h:5,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/jit/frontend/tracer.h:9,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/autograd/generated/variable_factories.h:12,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/types.h:7,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/torch.h:3,
                 from conftest.cpp:37:
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/ATen/core/op_registration/infer_schema.h: In static member function 'static constexpr std::array<c10::detail::infer_schema::ArgumentDef, sizeof (Ts ...)> c10::detail::infer_schema::createReturns<std::tuple<_Elements ...>, void>::call() [with ReturnTypes = {}]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/ATen/core/op_registration/infer_schema.h:85:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
In file included from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/adagrad.h:4:0,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim.h:3,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/all.h:14,
                 from /home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/torch.h:3,
                 from conftest.cpp:37:
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::SGDOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::RMSpropOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::LBFGSOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::AdamWOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::AdamOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h: In member function 'std::unique_ptr<torch::optim::OptimizerOptions> torch::optim::OptimizerCloneableOptions<Derived>::clone() const [with Derived = torch::optim::AdagradOptions]':
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/torch/csrc/api/include/torch/optim/optimizer.h:61:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
configure:9650: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "PLUMED"
| #define PACKAGE_TARNAME "plumed"
| #define PACKAGE_VERSION "2"
| #define PACKAGE_STRING "PLUMED 2"
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define __PLUMED_HAS_EXTERNAL_BLAS 1
| #define __PLUMED_HAS_EXTERNAL_LAPACK 1
| #define __PLUMED_HAS_MOLFILE_PLUGINS 1
| #define __PLUMED_HAS_MPI 1
| #define __PLUMED_HAS_ASMJIT 1
| #define __PLUMED_HAS_CREGEX 1
| #define __PLUMED_HAS_DLOPEN 1
| #define __PLUMED_HAS_RTLD_DEFAULT 1
| #define __PLUMED_HAS_CHDIR 1
| #define __PLUMED_HAS_SUBPROCESS 1
| #define __PLUMED_HAS_GETCWD 1
| #define __PLUMED_HAS_POPEN 1
| #define __PLUMED_HAS_EXECINFO 1
| #define __PLUMED_HAS_ZLIB 1
| #define __PLUMED_HAS_GSL 1
| #define __PLUMED_HAS_FFTW 1
| #define __PLUMED_HAS_PYTHON 1
| /* end confdefs.h.  */
|
| #include <torch/torch.h>
| #include <torch/script.h>
| int main() {
|   torch::Tensor tensor = torch::rand({2, 3});
|   return 0;
| }
|
configure:9655: result: no
configure:9673: WARNING: cannot enable __PLUMED_HAS_LIBTORCH
configure:9682: Release mode, adding -DNDEBUG
configure:9734: *** Special settings for dynamic libraries on Linux ***
configure:9736: Dynamic library extension is 'so'

[Luigi Bonati]

Hi, 

thanks for sharing the log. if you search for the output of the command `checking libtorch with -lc10` inside config.log you will find the following errors:

configure:9632: checking libtorch with -lc10
...
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/c10/util/C++17.h:16:2: error: #error "You're trying to build PyTorch with a too old version of GCC. We need GCC 5 or later."
/home/users/francisco/grant_462/project_data/SOFTWARE/cp2k-intel/libtorch/include/c10/util/C++17.h:24:2: error: #error You need C++14 to compile PyTorch

Can you try compiling with a newer version of gcc?

I will update the Plumed manual to highlight the requirements for enabling this module.

Best, 

Luigi

---------------------------------------------------------

Luigi Bonati

Postdoctoral Researcher

Atomistic Simulations

Italian Institute of Technology, Genova, Italy

---------------------------------------------------------

To view this discussion on the web visit https://groups.google.com/d/msgid/plumed-users/c8919861-0d6b-4c56-adb1-41f5d2a6ce6bn%40googlegroups.com.

[Francisco Carrascoza]

Luigi,

I have tried your answer: I tried gcc/9.3.0 and gcc/8.2.0. Both of them send me to the same kind of error. I also tried Intel and GNU compilers.  I attach here both config.log's

The error, I think is this, but I'm not sure:

libtorch/lib/libtorch_cpu.so: undefined reference to `lgamma@GLIBC_2.23'

I was searching for that error on internet, but nothing is clear to me. I also tried different MPICH and one openmpi/4.1.0_gcc820.

Again, I will really appreciate some clue on this.

Additionally I'm attaching the list of modules loaded (dirty after many trials) with some extra env variables.

Regards,

Francisco

[Luigi Bonati]

Hi,
this is a problem related to the precompiled LibTorch binaries. If you also tried with the pre-CXX11 abi version (see e.g. https://discuss.pytorch.org/t/glibc-linker-errors-with-libtorch-nightly-build/66030) and it gives you the same errors I think the only solution is to compile libtorch from scratch, as suggested here (https://discuss.pytorch.org/t/glibc-linker-errors-when-linking-against-libtorch-on-ubuntu-18-04/55183/2) .

Best,

---------------------------------------------------------

Luigi Bonati

Postdoctoral Researcher

Atomistic Simulations

Italian Institute of Technology, Genova, Italy

---------------------------------------------------------

To view this discussion on the web visit https://groups.google.com/d/msgid/plumed-users/02e9a5bb-1e8c-4eb9-a3b2-e416964e6f70n%40googlegroups.com.

[Francisco Carrascoza]

Thank you Luigi!  I solved the problem!

compiling plumed with libtorch pre-cxx11 ABI LibTorch binaries worked, however when trying to link with CP2K  it was failing (PLUMED-libtorch-CP2K).

The final solution was just what you said:

I compiled PyTorch from source. Then PLUMED-PyTorch and t hen PLUMED-PyTorch-CP2K was successful.

The problem seems to be the GCC versions, they are conflicting with each other.

[peng liu]

Dear Francisco

   Recently, i try to compile cp2k+plumed with the  pytorch_model. Here, cp2k-2023.1, plumed2-2.9.0, pytorch1.13, gcc-12.2.0, and python 3.10 are used. However, I alway failed.

  I see that you did it.

  So, can you tell me the version of softwares involved in your compiling process.  

  

   Best wishes, 

Peng