Nimble model failed due to the error of "Failed to create the shared library"

[Qianyu Li]

Hi Nimble community,

I recently met the issue that our nimble model didn't work due to the error of "Failed to create the shared library". It went well previously and I didn't change any system configuration. The current version of my Nimble package is 0.12.1. I attached a simple example below to show the error details. Any help would be greatly appreciated! 

> foo <- nimbleFunction( run = function(x = double(1)) {return(sum(x)); returnType(double())})
 compileNimble(foo)> cfoo <- compileNimble(foo)
Compiling
  [Note] This may take a minute.
  [Note] Use 'showCompilerOutput = TRUE' to see C++ compilation details.
Error: Failed to create the shared library. Run 'printErrors()' to see the compilation errors.
> printErrors()
using C++ compiler: ‘g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
using C++11
/usr/bin/ld: cannot find -lnimble
collect2: error: ld returned 1 exit status
make: *** [/usr/share/R/make/shlib.mk:10: dynamicRegistrations_02_08_01_31_44.so] Error 1

[Perry de Valpine]

Dear Qianyu,

Thanks for posting the question. It is difficult to diagnose compilation problems from afar. I see that you are on linux, and I think that means your package installation would be from source. I believe the error of not finding -lnimble means that the installation from source failed to compile the package C++ library (nimble.so, to which on-the-fly nimble compilation links) successfully. The package was still installed, but without the library. (Another possibility is if something changed in the paths or environmental variables on your system such that the library was installed but is not being found, but I'd guess that is not very likely.) In any case, my first suggestion would be to update your version of nimble (and of R if needed). Version 1.1.0 was released very recently, and you are multiple versions behind.

If it still fails, please look at the output from installing the package from source and post any errors shown from that.

HTH

Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/afaa7f4a-274f-47be-b644-6e7eafd58728n%40googlegroups.com.

[Qianyu Li]

Dear Perry,

Thanks very much for the prompt reply. I updated the nimble version to 1.1.0, and there are still some issues. Because I am using a shared server, I can only install the nimble package in a local dir. I can confirm the C++ library is successfully installed, and the issue should be related to the environmental variables. 

I added the local nimble libs path to “LIBRARY_PATH” and “INCLUDE_PATH”. The error of "not finding -lnimble” was solved, but I got a new error msg: 

```

foo <- nimbleFunction( run = function(x = double(1)) {return(sum(x)); returnType(double())})

eNimble(foo)> cfoo <- compileNimble(foo)>  compileNimble(foo)> cfoo <- compileNimble(foo)

Compiling

  [Note] This may take a minute.

  [Note] Use 'showCompilerOutput = TRUE' to see C++ compilation details.

Error in dyn.load(basename(ssDllName), local = TRUE) : 

  unable to load shared object '/tmp/Rtmp9v1qoX/nimble_generatedCode/dynamicRegistrations_02_08_12_56_32.so':

  /tmp/Rtmp9v1qoX/nimble_generatedCode/dynamicRegistrations_02_08_12_56_32.so: undefined symbol: getModelValuesPtrFromModel

```

My questions:

  1, Is “getModelValuesPtrFromModel” an internal function in nimble? If not, I think I could probably need add other lib to the path.

  2, Could I install the nimble package with a static .a lib? Dynamic link in my current environmental seems like a pain :(.

I really appreciate your help!

Best,

Qianyu Li

[Perry de Valpine]

Yes, you can try the configure arg "--enable-dylib=false".

The "dynamicRegistrations" compilation is a once-per-session step, done during the first call to compileNimble in a session. Yes, that undefined symbol is part of nimble's C++ library. I am not sure what is going on. Maybe it is finding libnimble but then libnimble is not well formed?

If you are on a shared server, could there be a problem with R's use of tempdir() (this is a session-specific directory in which R allows packages to put temporary files). You can try dirName="." in compileNimble to place files in your current working directory, as an experiment.

Perry

[Qianyu Li]

I tried dirName=“.”, but it still failed.

Under nimble package dir, I only found “nimble.so”, and I copied it to “libnimble.so”. I am not sure if this is the cause of `undefined symbol: getModelValuesPtrFromModel`. Is there a compiled linux “libnimble.so" that I can directly use?


Thanks very much
Qianyu

[Chris Paciorek]

Hi Qianyu,

If you do use the default of the dynamic libnimble.so, then if you run `ldd` on the `dynamicRegistrations` .so file, it should show that it is able to find `libnimble.so`. If it is not found, then that probably indicates that the "-L" flag is not being set correctly when the dynamicRegistrations compilatoin is being done.

Regardless of whether you are using libnimble.so or libnimble.a, when you do your compilation you should see something like this:

g++ -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o dynamicRegistrations_02_08_14_15_40.so dynamicRegistrations_02_08_14_15_40.o -L/tmp/libnima/nimble/CppCode -lnimble -llapack -lblas -L/usr/lib/R/lib -lR

Note the bold that is how libnimble is found.

Please run `compileNimble` like this and show us the complete result of what you see:

compileNimble(foo, showCompilerOutput = TRUE)

Also please look in the CppCode directory of your installed nimble package and make sure there is a file `libnimble.a` or `libnimble.so` in that directory.

-Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/F80F2D2B-CA58-4668-813C-B97AD4D4E1D4%40gmail.com.

[Qianyu Li]

Thanks Chris for the reply. 

Here is the result with `showCompilerOutput = TRUE`:

```

/usr/bin/g++ -std=gnu++11 -I"/usr/include/R" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/data2/Projects/NASA_CMS/R/library/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations  -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c dynamicRegistrations_02_09_02_12_52.cpp -o dynamicRegistrations_02_09_02_12_52.o

/usr/bin/g++ -std=gnu++11 -shared -L/usr/lib64/R/lib -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o dynamicRegistrations_02_09_02_12_52.so dynamicRegistrations_02_09_02_12_52.o -L/data2/Projects/NASA_CMS/R/library/nimble/CppCode -lnimble -Wl,-rpath /data2/Projects/NASA_CMS/R/library/nimble/CppCode -lopenblaso -lopenblaso -L/usr/lib64/R/lib -lR

using C++ compiler: ‘g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’

Error in dyn.load(basename(ssDllName), local = TRUE) : 

  unable to load shared object '/tmp/Rtmpe3tv6M/nimble_generatedCode/dynamicRegistrations_02_09_02_12_52.so':

  /tmp/Rtmpe3tv6M/nimble_generatedCode/dynamicRegistrations_02_09_02_12_52.so: undefined symbol: getModelValuesPtrFromModel

```

I checked the CppCode directory, and there NO `libnimble.a` or `libnimble.so`. Could you please let me know how to compile the file? I used install.packages() in R to install nimble, but it seems failed.

Qianyu 

[Chris Paciorek]

Ok, in that case something is going wrong during the installation procedure, as you say.  Please show us the complete output you get when you run `install.packages('nimble')`. 

[Chris Paciorek]

Hi Qianyu,

Thanks for providing the results of trying to install nimble from source off-list.  

It looks like libnimble.so is never being built because one of the files (predefinedNimbleLists.cpp) that gets compiled into libnimble.so is failing to compile with this error:

```

In file included from ../include/nimble/Utils.h:28,
                 from ../include/nimble/smartPtrs.h:25,
                 from ../include/nimble/predefinedNimbleLists.h:7,
                 from predefinedNimbleLists.cpp:4:
/usr/include/R/Rmath.h:210:15: error: 'std::Rf_beta' has not been declared
 #define beta  Rf_beta

```

Do you have a `Makevars` file in your ~/.R directory or in the directory you are in when you install NIMBLE? I see that in the output you shared with me that the nimble installation is using the `-std=gnu++11` flag rather than `-std=gnu++17`, and I am wondering if that is related to the problem. If you do have a Makevars file try moving it elsewhere or renaming it temporarily and see if the installation succeeds. You'll need to look carefully at the output produced to see if that same error causes libnimble.so not to get built.

If it succeeds, you should see something like this:

```

g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -DR_NO_REMAP -I"../include"      -fpic  -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c nimDerivs_atomic_probit.cpp -o nimDerivs_atomic_probit.o
linking libnimble.so
Linux  
g++ -std=gnu++17 -shared -o libnimble.so  nimDerivs_atomic_dyn_ind.o nodeFun.o accessorClasses.o nimbleCppAD.o RcppUtils.o EigenTypedefs.o eigenUsingClasses.o RcppNimbleUtils.o Utils.o NamedObjects.o ModelClassUtils.o dists.o nimDists.o dllFinalizer.o nimbleGraph.o smartPtrs.o predefinedNimbleLists.o nimOptim.o nimIntegrate.o nimDerivs_atomic_classes.o nimDerivs_atomic_matmult.o nimDerivs_atomic_matinverse.o nimDerivs_atomic_backsolve.o nimDerivs_atomic_forwardsolve.o nimDerivs_atomic_cholesky.o nimDerivs_atomic_solve_base.o nimDerivs_atomic_pow_int.o nimDerivs_atomic_zround.o nimDerivs_atomic_log_pow_int.o nimDerivs_atomic_probit.o 

```

(Side note for future reference: this SO post looks relevant.)

-chris

On Fri, Feb 9, 2024 at 5:28 PM Qianyu Li <liqy...@gmail.com> wrote:

Hi Chris,

Please find the installation output in the txt file below. Thanks a lot for your help!

Best,

Qianyu

[Chris Paciorek]

Hi Qianyu (and looping the list back in),

It looks like this is a nimble installation issue that occurs under Red Hat Enterprise Linux, and is not something specific to you. I've been able to reproduce the problem separately. I wouldn't be surprised if it affected other Linux distributions that are related to Red Hat such as CentOS.

For anyone reading this in the future, the error occurs when installing nimble from source, in building libnimble.so, in particular in compiling one of our C++ files, predefinedNimbleLists.cpp, which produces this error:

```

In file included from ../include/nimble/Utils.h:28,
                 from ../include/nimble/smartPtrs.h:25,
                 from ../include/nimble/predefinedNimbleLists.h:7,
                 from predefinedNimbleLists.cpp:4:

/opt/R/4.3.2/lib/R/include/Rmath.h:210:15: error: 'std::Rf_beta' has not been declared
 #define beta  Rf_beta
               ^~~~~~~

```

I don't have a solution for the moment, but we will look into it.

-Chris

On Tue, Feb 13, 2024 at 11:01 AM Qianyu Li <liqy...@gmail.com> wrote:

Hi Chris,

Please find the info below:

> sessionInfo()

R version 4.3.2 (2023-10-31)

Platform: x86_64-redhat-linux-gnu (64-bit)

Running under: Red Hat Enterprise Linux 8.9 (Ootpa)

Matrix products: default

BLAS/LAPACK: /usr/lib64/libopenblaso-r0.3.15.so;  LAPACK version 3.9.0

locale:

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              

 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    

 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   

 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 

 [9] LC_ADDRESS=C               LC_TELEPHONE=C            

[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/New_York

tzcode source: system (glibc)

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] nimble_1.1.0

loaded via a namespace (and not attached):

 [1] compiler_4.3.2      R6_2.5.1            magrittr_2.0.3     

 [4] parallel_4.3.2      igraph_1.3.1        coda_0.19-4.1      

 [7] grid_4.3.2          numDeriv_2016.8-1.1 pkgconfig_2.0.3    

[10] pracma_2.4.4        lattice_0.21-9     

Thanks!

Qianyu

On Feb 13, 2024, at 8:45 AM, Chris Paciorek <paci...@stat.berkeley.edu> wrote:

sessionInfo()

[Chris Paciorek]

It turns out the issue seems to be a general one preventing installation of nimble on Red Hat Enterprise Linux (RHEL). We have a fix for this that will be in the next nimble release, but in the interim,

RHEL users can install nimble like this:

library(remotes)
install_github("nimble-dev/nimble", ref = "fix-rhel", subdir = "packages/nimble")

I don't know if the issue affects other Red Hat-related Linux variants, but it does not seem to affect Fedora or Scientific Linux, as far as I can tell. If it does affect other Linux variants, I am hopeful the fix will also work on those affected variants.

-Chris

[Harry Wells]

Hi, i just wanted to say that i had the same issue on a shared Linux cluster and the dev version has fixed it - thank you!

harry

[Jacob Maronge]

Hi Chris, I believe I am having the same issue with installing nimble but when I use 

library(remotes)
install_github("nimble-dev/nimble", ref = "fix-rhel", subdir = "packages/nimble")

I receive the error

Using github PAT from envvar GITHUB_PAT Error: Failed to install 'unknown package' from GitHub: HTTP error 404. No commit found for the ref fix-rhel

Is this fix still available?

Best,

Jake

[Chris Paciorek]

The fix has been merged into the devel branch, so do:

install_github("nimble-dev/nimble", ref = "devel", subdir = "packages/nimble")

Sorry, I forgot that we had publicized that branch so I should have left it alive

rather than the usual strategy of deleting a branch after it is merged in.

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/ab14007e-0acb-4900-a98b-de57b216da4an%40googlegroups.com.

[Jacob Maronge]

No problem! Confirmed that this fixed the problem for me

[MA21D501 SHIVSHANKAR CHANDRAKANT NILA]

Dear all 

I am encountering an error msg: "Failed to create the shared library when using the nimble package."  please have a look at the below error msg and kindly help if one knows.

Error in { : task 1 failed - "Failed to create the shared library.
g++ -std=gnu++17 -I"/home/ma21d501/R-4.4.0/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/home/ma21d501/R-4.4.0/library/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno->
g++ -std=gnu++17 -I"/home/ma21d501/R-4.4.0/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/home/ma21d501/R-4.4.0/library/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno->
using C++ compiler: 'g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0'
In file included from P_4_BivExtMixc_MID_4.h:6,
                 from P_4_BivExtMixc_MID_4.cpp:7:
/home/ma21d501/R-4.4.0/library/nimble/include/nimble/NimArr.h: In instantiation of 'NimArr<nDim, T>& VecNimArr<ndim, T
Calls: %dopar% -> <Anonymous>
Execution halted

[Perry de Valpine]

It is difficult to say what might be wrong without seeing your code. Taking a guess, did you try to use %dopar% in a nimbleFunction or in model code? That is not a supported keyword in the language for either nimbleFunctions or models. Feel free to narrow down whatever example you have and send it for review.

Perry

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/5aa8f227-f074-495e-a2f1-d701690e6875n%40googlegroups.com.

[MA21D501 SHIVSHANKAR CHANDRAKANT NILA]

Dear Sir,

 The code is parallelized and runs fine in Windows. I'm using R-Studio.

The same code is running fine in Ubuntu using Rscript, and I am using the function  %dopar%, it working fine in the window and in Linux(in ubuntu), but 

When the same code is submitted in the cluster to a node with 40 cores, the following error is generated every time.

Error in { : task 2 failed - "Failed to create the shared library.
g++ -std=gnu++17 -I"/home/ma21d501/R-4.4.0/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/home/ma21d501/R-4.4.0/library/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations  -I/usr/local/include    -fpic  -g -O2   -c P_2_MCMC.cpp -o P_2_MCMC.o
g++ -std=gnu++17 -I"/home/ma21d501/R-4.4.0/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/home/ma21d501/R-4.4.0/library/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations  -I/usr/local/include    -fpic  -g -O2   -c P_2_BivExtMixc_MID_2.cpp -o P_2_BivExtMixc_MID_2.o

using C++ compiler: 'g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0'

In file included from /home/ma21d501/R-4.4.0/library/nimble/include/nimble/RcppNimbleUtils.h:25,
                 from /home/ma21d501/R-4.4.0/library/nimble/include/nimble/NamedObjects.h:28,
                 from /home/ma21d501/R-4.4.0/library/nimble/inc
Calls: %dopar% -> <Anonymous>

Here are the specifications of the setup I am using:

1. Cluster OS: Ubuntu 20.04.1 LTS
2. R Version: 4.4
3. NIMBLE Version: 1.2.0
4. Scheduler: PBS

This error could also be due to some misconfiguration of R in the cluster. Your insight might help.

Attached the pbs script, code and the error log.

I've also run a toy example with the following code, which was running fine.

library("nimble");

foo <- nimbleFunction( run = function(x = double(1)) {return(sum(x)); returnType(double())})

cfoo <- compileNimble(foo)
cfoo(1:11)
printErrors() 
showCompilerOutput = TRUE

[Chris Paciorek]

Hi Shivshankar,

This may be difficult for us to diagnose given we don't have access to
a PBS-based system, but I have a few comments:

1) When you run this under Linux, but not using PBS, are you able to
get %dopar% to work on a Linux machine with multiple cores?
2) Please run your code using the `showCompilerOutput=TRUE` argument
to `compileNimble` and show the full output that you get.
3) If you could simplify this down to a minimal reproducible example
(e.g., a very basic MCMC) that would be helpful. Then I could try
running it in parallel to reproduce on a cluster that I have access to
(albeit using Slurm rather than PBS). When you send more information,
please make sure to send the code for `run_MCMC_parallel`, as you did
not send that in your attachments.

-chris

On Sun, Jun 23, 2024 at 8:26 PM MA21D501 SHIVSHANKAR CHANDRAKANT NILA

> To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/d4cb7295-73ca-4b44-b2ed-09d20e10c036n%40googlegroups.com.

[Harry Wells]

Hi All,

Firstly, thank you so much for developing nimble! :)

i'm having a similar issue with parallelising nimble models on a shared Dell Linux Cluster (https://researchcomputing.princeton.edu/systems/adroit). 

Attached is a simple reproducible example (R v4.4.0; nimble v1.2.1).

Running 3 chains in series works fine (series.R & slurm_series.txt), but attempting parallelisation with the parallel (parallel_fork.R & slurm_parallel.txt) and doParallel (doParallel_fork.R & slurm_doParallel.txt) packages both give similar 'Failed to create the shared library' errors. The parallel script works fine on my laptop (MacOS 15.1.1; R v4.4.2; nimble v1.2.1).

The slurm_XXXX.txt files contain both the slurm scripts and outputs. The output of 'showCompilerOutput=TRUE' in compileNimble() is in the slurm_series.txt file.

If there's a way to parallelise just the runMCMC() part that would be ideal to avoid building & compiling in each thread, but that didn't work either when i tried it.

Any advice would be greatly appreciated :)

Many thanks in advance!

Cheers,

harry

[Chris Paciorek]

hi Harry, I think the problem is that you're trying to use a fork-based cluster. We have some discussion of this in our example on parallelizing NIMBLE. Please try the ideas there (either not using forking or making sure to set up separate temp directories for the processes) and let me know if that doesn't solve the problem.

As far as not having to separately build and compile in each thread/process, we are working on that for nimble v. 2.0. But it's not possible yet.

-chris

To view this discussion visit https://groups.google.com/d/msgid/nimble-users/44bdad3d-065b-47ee-93d7-04f938c539b0n%40googlegroups.com.

[Harry Wells]

Hi Chris,

Thank you for your swift response. That was indeed the issue and it works perfectly now — apologies for missing that note in which you clearly explain the issue with forks..

That’s great to hear! I’ll certainly keep an eye out for news that functionality in v. 2.0.

Thanks again for all the work that you folks are doing developing this fantastic tool!

Cheers,

harry

[Harry Wells]

Hi Chris,

Sorry — me again..

I’m encountering the same ‘Failed to create the shared library’ error with nimbleHMC v.0.2.3, nimble v.1.2.1, and R v.4.4.1 on the same Dell Linux Cluster (https://researchcomputing.princeton.edu/systems/adroit) without parallelisation.

Attached is the example from the NIMBLE User Manual 7.11.2.1 with the slurm output.

Any thoughts would be much appreciated ☺️

Cheers,

harry

[Chris Paciorek]

hi Harry,

That code runs fine for me under Linux (Ubuntu). You showed the compiler output from compiling the *model*, but that actually finishes successfully in your case (though it gives some warnings that make it look like there may be a problem). But the real error is occurring in compiling the *MCMC*, so please show the output of the following:

CHMC <- compileNimble(HMC, project = model, showCompilerOutput = TRUE)

-chris

[Harry Wells]

Hi Chris,

Apologies — attached is the MCMC compiler output.

harry

[Chris Paciorek]

Hmm, now I am confused. I thought I fixed this issue (which occurs on RHEL, and is discussed here) a year ago, and that fix worked for you.

Is there anything that has changed on your side (either your user account or the Princeton cluster) since then (i.e., since your email to nimble-users on 2024-02-28)?

-chris

[Harry Wells]

Thanks for looking into this, Chris!

I’m not aware of any changes to the user account or cluster. Scripts that use regular nimble functions (e.g., nimble::buildMCMC as opposed to nimbleHMC::buildHMC) are working fine.

Cheers,

harry 

[Chris Paciorek]

Ok, it may be that in the original fix from a year ago, I missed a piece of the fix.

Could you try re-installing nimble from the branch "test_math_include" and see if that works?

library(remotes)
install_github("nimble-dev/nimble", ref = "test_math_include", subdir = "packages/nimble")

[Harry Wells]

That seems to have done the trick and also appears to work in parallel — thanks so much, Chris! Really appreciate it ☺️

[Chris Paciorek]

Ah, great. Thanks for letting us know about your difficulty as it identified as shortcoming in my original fix. We'll get the additional fix into the next release.