Issue with RStan code running non-interactively on HPC cluster

[H. X.]

Hi All,

I've tested my code to work fine on my on computer and on interactive nodes of an HPC cluster. However, when I schedule my RStan code to run non-interactively on an HPC environment, I repeatedly get a related error message along the lines of:


Loading required package: StanHeaders
rstan (Version 2.12.1, packaged: 2016-09-11 13:07:50 UTC, GitRev: 85f7a56811da)
For execution on a local, multicore CPU with excess RAM we recommend calling
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
Error in file(con, "r") : cannot open the connection
Calls: stan ... cxxfunctionplus -> cxxfunction -> compileCode -> readLines -> file
In addition: Warning messages:
1: running command '/data/apps/R/3.2.3/lib64/R/bin/R CMD SHLIB file66140ff794a.cpp 2> file66140ff794a.cpp.err.txt' had status 1
2: In file(con, "r") :
  cannot open file 'file66140ff794a.cpp.err.txt': No such file or directory

I think this has to do with the temporary files that Stan writes and that there is some folder that Stan does not have access to? I am just mystified because my code runs without issue again on my computer and interactively on the HPC. Has anybody had a similar issue running their code non-interactively on an HPC cluster?

[H. X.]

Could someone tell me what the files like file66140ff794a.cpp and other similarly named ones do? The file name in question changes every time, so I assume the file names are randomly generated.

[Ben Goodrich]

On Sunday, November 13, 2016 at 12:05:56 PM UTC-5, H. X. wrote:

Could someone tell me what the files like file66140ff794a.cpp and other similarly named ones do? The file name in question changes every time, so I assume the file names are randomly generated.

That is the C++ translation of your .stan file, and yes, it does have a randomly chosen filename to prevent name clashes. I'm not sure why you got the original error, but I think I fixed something similar in the develop branch on GitHub a couple weeks ago.

Ben

[H. X.]

Could you provide a link to that issue? Thanks.

[Ben Goodrich]

On Monday, November 14, 2016 at 12:27:29 AM UTC-5, H. X. wrote:

Could you provide a link to that issue? Thanks.

It was fixed by commit https://github.com/stan-dev/rstan/commit/6a8b7934980dc99cd059382c95550385ed5fe324

Ben

[H. X.]

Here is an additional issue I am getting:

Error in compileCode(f, code, language = language, verbose = verbose) :
  Compilation ERROR, function(s)/method(s) not created! In file included from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core/ddv_vari.hpp:4:0,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core.hpp:7,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/mat.hpp:4,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math.hpp:4,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/src/stan/model/model_header.hpp:4,
                 from file366150774d1.cpp:8:
/data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core/vari.hpp:99:9: internal compiler error: Illegal instruction
         adj_ = 1.0;
         ^
I'm not sure it's the same issue as the one you fixed?

[Ben Goodrich]

On Tuesday, November 15, 2016 at 3:25:00 PM UTC-5, H. X. wrote:

Here is an additional issue I am getting:

Error in compileCode(f, code, language = language, verbose = verbose) :
  Compilation ERROR, function(s)/method(s) not created! In file included from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core/ddv_vari.hpp:4:0,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core.hpp:7,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/mat.hpp:4,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math.hpp:4,
                 from /data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/src/stan/model/model_header.hpp:4,
                 from file366150774d1.cpp:8:
/data/apps/R/3.2.3/lib64/R/library/StanHeaders/include/stan/math/rev/core/vari.hpp:99:9: internal compiler error: Illegal instruction
         adj_ = 1.0;
         ^
I'm not sure it's the same issue as the one you fixed?

That is something completely different. Can you send Stan code that triggers the internal compiler error as soon as possible?

Thanks,
Ben

[H. X.]

Yes, attached are both stan files are used which should produce the same error, and then also the necessary data file.

I cannot reproduce this error on my Clang Mac. Only happens on the HPC.

Thanks for looking at this.

[Ben Goodrich]

On Tuesday, November 15, 2016 at 4:50:54 PM UTC-5, H. X. wrote:

Yes, attached are both stan files are used which should produce the same error, and then also the necessary data file.

I was able to compile both AWB_adriana.stan and CON_adriana.stan without encountering any error. Maybe your computer simply ran out of memory? If not, what operating system and compiler version are you using?

Ben

[H. X.]

Compiles fine on my computer. The issue is with the gcc and the Scientific Linux 5.7 OS. I've allotted the process 32 GB of memory and have the same issue, so I don't think it's a memory issue?

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/g7qFrCGnaPc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+unsubscribe@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Tuesday, November 15, 2016 at 5:00:03 PM UTC-5, H. X. wrote:

Compiles fine on my computer. The issue is with the gcc and the Scientific Linux 5.7 OS. I've allotted the process 32 GB of memory and have the same issue, so I don't think it's a memory issue?

Can you tell us what is the output of

g++ --version

on the cluster?

[H. X.]

g++ (GCC) 4.8.2

[H. X.]

What system did you test on?

[Ben Goodrich]

On Tuesday, November 15, 2016 at 5:05:28 PM UTC-5, H. X. wrote:

g++ (GCC) 4.8.2

The g++-4.8 on my laptop compiles those Stan files fine. Are you sure the cluster could access the 32GB of RAM when compiling? Sometimes people compile on the parent node and run the model on a child node in which case the parent node would need enough RAM. Or maybe you are compiling the models on the child node and it doesn't have 32GB available? Or maybe the 32GB is overall but it is split among 10 nodes, so they can each only access 3GB?

Ben

[Ben Goodrich]

On Tuesday, November 15, 2016 at 5:12:49 PM UTC-5, H. X. wrote:

What system did you test on?

Debian unstable

[H. X.]

So I reserved 64GB overall spread across 2 nodes. About the parent/child, I'm not sure.

So describing exactly what I am doing, first I log onto login node of my university's HPC.

I then cd to my working directory. From here I pull up my bash script

#!/bin/bash

#$ -N adriana_data
#$ -t 2
#$ -q bio,abio,free64
#$ -l mem_free=64G

module load R/3.2.3

Rscript stan_AWB_adriana.r

and then I qsub this script. At this point, the error message pops up.

The same thing happens when I qrsh to access an interactive node and run my code interactively.

Very strangely, the code works when I run it interactively from the login node, not something I am supposed to do, of course.

[Ben Goodrich]

On Tuesday, November 15, 2016 at 5:34:41 PM UTC-5, H. X. wrote:

Very strangely, the code works when I run it interactively from the login node, not something I am supposed to do, of course.

This sounds more like a cluster problem than a Stan problem. One thing that may work for you would be to do on the login node

rstan_options(auto_write = TRUE)
sm <- stan_model("path/to/AWB_adriana.stan")
q()

which will write a AWB_adriana.rda file in the same directory as the AWB_adriana.stan file. Then when you do MCMC on the child nodes, they should automatically load the model from AWB_adriana.rda without recompiling it.

Ben

[H. X.]

Issue was fixed after R was re-compiled on Clang on the cluster. Thanks again for your help.