Parallelization in NimbleExternalCall

[Colton Padilla]

Hi everyone, 

I am attempting to use the OpenMP package to parallelize a loop within an nimbleExternalCall; however, I am getting an error that the package cannot be found when I attempt to compile the nimble function that has the external call within it. I was able to parallelize the loop within an Rcpp function that I used an nimbleRcall to run, but the overhead of transferring the vectors and matrices is too computationally expensive. I have tried doing something like this prior to compilation:

Sys.setenv("PKG_CXXFLAGS"= '-fopenmp') 

However, this does not seem to solve the issue. I found a single thread on here talking about openMP and parallelization, but it does not have an answer. While the externalCall has saved time already, without the parallelization of the rcpp function, it is still computationally slow. Thanks to anyone who may be able to shed light on this. 

[Chris Paciorek]

Hi Colton,

Hmmm, your approach of using `Sys.setenv` works for me on my Linux machine, at least in the sense of injecting `-fopenmp` into the compiler invocations (see below for example).

Can you tell if (1) `-fopenmp` is not being injected into the compiler invocation or (2) somehow OpenMP is not enabled/used despite it being injected.

```

library(nimble)

Sys.setenv("PKG_CXXFLAGS"= '-fopenmp')

<run the example code from help(nimbleExternalCall)>

```

Note the `-fopenmp` inserted below.

```

Defining model
Building model
Setting data and initial values
Checking model sizes and dimensions
  [Note] This model is not fully initialized. This is not an error.
         To see which variables are not initialized, use model$initializeInfo().
         For more information on model initialization, see help(modelInitialization).
Compiling
  [Note] This may take a minute.
  [Note] On some systems there may be some compiler warnings that can be safely ignored.
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/accounts/vis/paciorek/R/x86_64-pc-linux-gnu-library-ubuntu-24.04/4.5/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations     -fopenmp -fpic  -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/r-base-qkOIYD/r-base-4.5.0=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/r-base-qkOIYD/r-base-4.5.0=/usr/src/r-base-4.5.0-3.2404.0 -Wdate-time -D_FORTIFY_SOURCE=3   -c dynamicRegistrations_10_12_12_35_21.cpp -o dynamicRegistrations_10_12_12_35_21.o
g++ -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -o dynamicRegistrations_10_12_12_35_21.so dynamicRegistrations_10_12_12_35_21.o -L/accounts/vis/paciorek/R/x86_64-pc-linux-gnu-library-ubuntu-24.04/4.5/nimble/CppCode -lnimble -Wl,-rpath /accounts/vis/paciorek/R/x86_64-pc-linux-gnu-library-ubuntu-24.04/4.5/nimble/CppCode -llapack -lblas -L/usr/lib/R/lib -lR
using C++ compiler: ‘g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0’
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG -DR_NO_REMAP   -DEIGEN_MPL2_ONLY=1 -I"/accounts/vis/paciorek/R/x86_64-pc-linux-gnu-library-ubuntu-24.04/4.5/nimble/include" -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations     -fopenmp -fpic  -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/r-base-qkOIYD/r-base-4.5.0=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/r-base-qkOIYD/r-base-4.5.0=/usr/src/r-base-4.5.0-3.2404.0 -Wdate-time -D_FORTIFY_SOURCE=3   -c P_1_demoCode_MID_1.cpp -o P_1_demoCode_MID_1.o
```

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nimble-users/1ba5b1d3-8516-46ee-8f11-6162505543fan%40googlegroups.com.

[Colton Padilla]

Hey Chris,

Using the example code within Nimble External Call works for me, I even added the one line that appears to be causing my code to error. 

cat('
 #include <cstdio>
 #include "add1.h"
 void my_internal_function(double *p, double *ans, int n) {
   printf("In my_internal_function\\n");
     /* cat reduces the double slash to single slash */
   #pragma omp parallel for
   for(int i = 0; i < n; i++)
     ans[i] = p[i] + 1.0;
 }
')

This compiles with 0 issues, but when I try to use a similar workflow to use my function it errors. The line of code that I am adding to deal with parallelizing the loop is "#pragma omp parallel for" which in the context of the NimbleExternalCall example code works and does compile; however, when I try to compile my own function it will only compile without that line of code and it gives me these C++ errors.

C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/Rtmp6ZFC7h/HI1D.o:HI1D.cpp:(.text+0x1a72): undefined reference to `GOMP_parallel'

C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/Rtmp6ZFC7h/HI1D.o:HI1D.cpp:(.text+0x2b7c): undefined reference to `omp_get_num_threads' C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/Rtmp6ZFC7h/HI1D.o:HI1D.cpp:(.text+0x2b83): undefined reference to `omp_get_thread_num' collect2.exe: error: ld returned 1 exit status

[Colton Padilla]

I also just saw in the compiler output that on the last step the -fopenmp flag is not within the compilation call.     

g++ -std=gnu++17 -I"C:/PROGRA~1/R/R-45~1.0/include" -DNDEBUG -DR_NO_REMAP -I"C:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\include" -I"" -DEIGEN_MPL2_ONLY=1 -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations -std=c++11 -I"C:/rtools45/x86_64-w64-mingw32.static.posix/include" -fopenmp -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -fopenmp -c dynamicRegistrations_10_14_09_17_48.cpp -o dynamicRegistrations_10_14_09_17_48.o g++ -std=gnu++17 -shared -s -static-libgcc -o dynamicRegistrations_10_14_09_17_48.dll tmp.def dynamicRegistrations_10_14_09_17_48.o -LC:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\CppCode -lnimble_x64 -lRlapack -lRblas -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R/R-45~1.0/bin/x64 -lR using C++ compiler: 'G__~1.EXE (GCC) 14.2.0' g++ -std=gnu++17 -I"C:/PROGRA~1/R/R-45~1.0/include" -DNDEBUG -DR_NO_REMAP -I"C:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\include" -I"" -DEIGEN_MPL2_ONLY=1 -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations -std=c++11 -I"C:/rtools45/x86_64-w64-mingw32.static.posix/include" -fopenmp -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -fopenmp -c P_1_rcFun_R_GlobalEnv7.cpp -o P_1_rcFun_R_GlobalEnv7.o g++ -std=gnu++17 -shared -s -static-libgcc -o P_1_rcFun_R_GlobalEnv7_10_14_09_17_48.dll tmp.def P_1_rcFun_R_GlobalEnv7.o C:/Users/cp224670/AppData/Local/Temp/Rtmp4mZ28S/HI1D.o -LC:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\CppCode -lnimble_x64 -lRlapack -lRblas -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R/R-45~1.0/bin/x64 -lR using C++ compiler: 'G__~1.EXE (GCC) 14.2.0'

[Chris Paciorek]

Hmm, odd that it works fine in our example. Can you share your code (ideally a minimal example) with me to try on my end?

As far as the -fopenmp in the last step, it shouldn't be needed there as that is just taking the compiled (.o) files and creating a shared object file. 

To view this discussion visit https://groups.google.com/d/msgid/nimble-users/259348de-d41e-47d3-bd7c-062c0399501bn%40googlegroups.com.

[Colton Padilla]

Hey Chris, I've never used the private message here so I am not sure if I did it right. Here is the reproducible example.

[Chris Paciorek]

hi Colton, I see you are compiling HI1D.cpp yourself and giving us the .o file rather than having us compile it.

In doing so, it looks like you left out `-fpic`, which we include when we create the .o file(s). (Hence the error messages about 'relocation' and the mention of `-fPIC` when I run on Linux - you might have seen something different under Windows.)

I think if you add that in to the compilation in line 47 then it should work. (The compilation worked for me on Linux with that change, though I didn't try to actually run the compiled nimbleFunction.)

To view this discussion visit https://groups.google.com/d/msgid/nimble-users/b20e93d5-2e1e-4d6c-9adb-54cd6d43c657n%40googlegroups.com.

[Colton Padilla]

Hey Chris,

I have tried this and I am getting the same error. I am guessing that you mean that I can create the .o file without actually compiling it using the system command? I am following the example from the nimbleExternalCall documentation, so I thought that was the method to get an external call to compile correctly. I have also tried to compile directly with Nimble and that is not working. Here's another file that is much simpler that is giving me the same compilation error when I add the openMP call.

[Colton Padilla]

Also here is the error message I am getting when trying to use compileNimble() directly. It appears here that -fpic is missing as well even though I have tried to inject it using the Sys.setenv() call as well.

g++ -std=gnu++17 -I"C:/PROGRA~1/R/R-45~1.0/include" -DNDEBUG -DR_NO_REMAP -I"C:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\include" -I"" -DEIGEN_MPL2_ONLY=1 -Wno-misleading-indentation -Wno-ignored-attributes -Wno-deprecated-declarations -std=c++11 -I"C:/rtools45/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -fopenmp -c P_2_rcFun_R_GlobalEnv9.cpp -o P_2_rcFun_R_GlobalEnv9.o g++ -std=gnu++17 -shared -s -static-libgcc -o P_2_rcFun_R_GlobalEnv9_10_15_08_40_05.dll tmp.def P_2_rcFun_R_GlobalEnv9.o C:/Users/cp224670/AppData/Local/Temp/RtmpEvx9LA/Eff_Like_Vec.o -LC:\Users\cp224670\AppData\Local\R\win-library\4.5\nimble\CppCode -lnimble_x64 -lRlapack -lRblas -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools45/x86_64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R/R-45~1.0/bin/x64 -lR using C++ compiler: 'G__~1.EXE (GCC) 14.2.0' C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/RtmpEvx9LA/Eff_Like_Vec.o:Eff_Like_Vec.c:(.text+0x24): undefined reference to `omp_get_num_threads' C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/RtmpEvx9LA/Eff_Like_Vec.o:Eff_Like_Vec.c:(.text+0x2b): undefined reference to `omp_get_thread_num' C:\rtools45\x86_64-w64-mingw32.static.posix\bin/ld.exe: C:/Users/cp224670/AppData/Local/Temp/RtmpEvx9LA/Eff_Like_Vec.o:Eff_Like_Vec.c:(.text+0x1af): undefined reference to `GOMP_parallel' collect2.exe: error: ld returned 1 exit status

[Chris Paciorek]

Hi Colton, just to make sure we're on the same page, did you try putting in the `-fpic` like this in line 47?

cmd <- sprintf(
  'g++ -std=gnu++17 -fopenmp -fpic -I"%s" -I"%s" -I"%s" -I"%s" HI1D.cpp -c -o HI1D.o',
  spl2, rcpp, arma, Rhome
)

If you did and that didn't work, then I'll try to reproduce things from what you sent most recently. 

To view this discussion visit https://groups.google.com/d/msgid/nimble-users/49775578-0035-49ed-a431-10faac0f44b7n%40googlegroups.com.

[Colton Padilla]

Yes I have tried that and that gives me the same error. For reference, I am on a windows computer. Thanks for all your help with this.

[Chris Paciorek]

Hi Colton, I'm going to follow up with you off-list while we try to sort this out.

Also, I realized I was confusing in talking about the .o file. Yes, you create that yourself (as is shown in the example). For some reason, I mistakenly thought that Nimble might handle that behind the scenes. So ignore my comments about "giving us the .o file rather than having us compile it".

-chris

To view this discussion visit https://groups.google.com/d/msgid/nimble-users/8edbee38-a7a3-413d-aebb-d599cd8be608n%40googlegroups.com.

[Colton Padilla]

Hey everyone,

After going back and forth with Chris over this, we found a solution to use openMP. In my .R/makevars.win file, we added in "PKG_CXXFLAGS += -fopenmp" and  "PKG_LIBS += -fopenmp" and this solved the issue. The openMP package was not being linked on the final step of compilation and this got it linked in. Hope this helps out someone in the future.

Colton