Error statuses from inla.exe

[Sam Clifford]

Hi R-INLA devs,

I'm attempting to work with my own data and getting an error message along the lines of

running command '"C:/R/R-2.14.0/library/INLA/bin/windows/inla.exe"  -b -s   "C:/BLAHBLAHBLAH/Model.ini"' had status 5

I thought it might've been something wrong with the way I was setting up the model so I tried testing with the Epil example and am still getting the error message with status 5. Upgrading with testing=TRUE doesn't fix it, either. I can't find any information about the error statuses in the INLA documentation nor on the website here.

What does status 5 mean and is there a way to stop getting these errors? I'm running R 2.14 on Windows XP.

Sam

[Håvard Rue]

There are some problems with Windows and R-2.14 and in particular with
Windows XP. We believe these problems are outside our control; its
related to the shell/system-interface from R to Windows. We have tried
to bypass them in various ways without succeeding completely. If you run
with option verbose=TRUE these problems seems to go away. You may try
this simple model

r = inla(y~1, data = data.frame(y=1))

if it fails, then add option verbose

r = inla(y~1, data = data.frame(y=1), verbose=TRUE)

and it is fine.

We have not found any way to capture the output from inla() with
verbose=TRUE, which could easily be turned into a work-around for the
issue..

--
Håvard Rue
Department of Mathematical Sciences
Norwegian University of Science and Technology
N-7491 Trondheim, Norway
Voice: +47-7359-3533 URL : http://www.math.ntnu.no/~hrue
Mobile: +47-9260-0021 Email: havar...@math.ntnu.no

[Sam Clifford]

Thanks.

I will try to run it remotely on a supercomputer running GNU/Linux if it's a case of R and Windows not playing nicely.

What do the error statuses mean, though? The two I most frequently come across are status 3 and 5.

[Håvard Rue]

I don't know. It is whatever is returned from the Windows-shell. To be
sure, can you try a inla.upgrade(testing=TRUE) once more?

I guess you're aware of this option:

http://www.r-inla.org/faq#TOC-I-have-access-to-a-remote-Linux-ser

[Finn Lindgren]

According to
  http://msdn.microsoft.com/en-us/library/ms681382.aspx
error codes 3 and 5 may refer to "path not found" and "access denied", respectively.

[Sam Clifford]

Thanks Finn.

I will try running my code via our supercomputer, I haven't been
successful with the remote access from a local R session over ssh, but
I will keep plugging away.

Sam

[Finn Lindgren]

One option is to log in to the remote machine with ssh, and run the R session in "screen".  That allows the connection to be broken while R is still running.  I then usually save an .RData file with the results and copy that to my own machine for further result analysis.
/Finn

[Sam Clifford]

I've spoken to my university's computing group and have managed to get a connection to the supercomputer and run inla() with the "remote" call. It fails to store the results though. I've used a toy example and checked that it runs without the remote call but when I run it remotely, the joint.dat file is empty on the server. I still get information on the local instance of R about the run time. The files on the server are stored in ~/tmp/.inla.remote/inla-job-1217-19873-7184/, and the computing guy says it might be worth checking to see if I can control whether the files are stored anywhere other than ~/tmp/. He's also suggested looking at using a wrapper script to submit to the PBS queue on the supercomputer, store the results and then pipe them back to R.

Sam

[Daniel Simpson]

Hi Sam,

Quick response - Håvard certainly knows more about this.  But there is a "working.directory" option in the inla() call that sets the temp directory.  I've never tried it when running things remotely, but it may work.

Also - are you running on the "proper" supercomputer?  INLA is made with OpenMP, which can only make use of shared-memory machines.  Worst case, it only runs on one processor, but you might want to check 'top' or something to see if you're actually parallelising anything.

Best,

Dan

[Sam Clifford]

I'm running it on the "head node" rather than via the queue system, but the head node is an eight core machine with 48GB or so of RAM. So even if it's only running on one processor it's better than my desktop.

I'll ask Ashley about disk access to other directories than /home, apparently the file I/O can cause problems sometimes.

I have used the PBS queue to submit long jobs with R-INLA. I'm not sure how many processors it's used when running though.

[Sam Clifford]

Håvard said:

Sam,

it might be that your Supercomputer behaves differently than we have
assumed. Do you have R installed on that computer? Will INLA run if you
run R interactively?

what happen, is that your local R session creates all the files
required, and its all copied to your remote computer in a spesific
directory (~/tmp/.inla-remote/....), and then the inla-program is
started on your remote computer reading files from  ~/tmp/.inla... and
writing the results to that directory. then your local computer copies
all the results-files back, and R reads the results back in.

can you run with options

       inla(..., debug=TRUE,verbose=TRUE)

?

The code I am running is

x <- rnorm(10,0,1)

y = 2*x

result3 <- inla(data=data.frame(x=x,y=y),formula = y ~ x, keep=T, verbose=T, inla.call="remote",debug=T) 

which gives

[1] "MPredictor 0"

[1] "NPredictor 10"

[1] "NData 10"

[1] "y...fake has length 10"

n.family 1 

Model and results are stored in working directory [inla.model]

[1] "prepare problem section"

Prepare inla file.....inla.create.data.file: n.data =  10 

[1] "\nOpen file to write inla.model/data.files/filef602d6c7076"

[1]  0 30 10  3  0  1  0  1

[1] "prepare data section"

[1] "prepare predictor section"

write label[ (Intercept) ]

[1] "\nOpen file to write inla.model/data.files/filef601b776aef"

[1]  0 20 10  2  0  1  0  1

file fixed $inladatadir/filef601b776aef 

write label[ x ]

[1] "\nOpen file to write inla.model/data.files/filef60e8544c8"

[1]  0 20 10  2  0  1  0  1

file fixed $inladatadir/filef60e8544c8 

...done

Run inla...

        hgid: adf568995b8a  date: Thu Jan 12 09:05:04 2012 +0100

        Run ~/R/x86_64-unknown-linux-gnu-library/2.12/INLA/bin/linux/inla32 at host lyra.qut.edu.au

and appears to give normal output for running the model.

* ============================================================================ *

* Access to this computer system is only for authorised QUT staff and students *

* and external persons authorised by QUT.                                      *

* ---------------------------------------------------------------------------- *

* WARNING:        It is a criminal offence to:                                 *

*                 i.  Obtain access to data without authority.                 *

*                 ii. Damage, delete, alter or insert data without authority.  *

* ---------------------------------------------------------------------------- *

* Communications on or through QUT's computer systems are monitored and        *

* recorded to secure effective system operation and for other lawful purposes. *

* ============================================================================ *

        hgid: ca9ea3169e18  date: Sat Nov 19 00:35:54 2011 +0100

Report bugs to <hr...@math.ntnu.no>

Processing file [Model.ini] max_threads=[24]

inla_build...

        number of sections=[6]

        parse section=[0] name=[INLA.Model] type=[PROBLEM]

        inla_parse_problem...

                name=[INLA.Model]

                strategy=[default]

        store results in directory=[results.files]

                output:

                        cpo=[0]

                        dic=[0]

                        kld=[1]

                        mlik=[1]

                        q=[0]

                        graph=[0]

                        hyperparameters=[1]

                        summary=[1]

                        return.marginals=[1]

                        nquantiles=[3]  [ 0.025 0.5 0.975 ]

                        ncdf=[0]  [ ]

        parse section=[2] name=[Predictor] type=[PREDICTOR]

        inla_parse_predictor ...

                section=[Predictor]

                dir=[predictor]

                PRIOR->name=[loggamma]

                PRIOR->from_theta=[function (x) <<NEWLINE>>exp(x)]

                PRIOR->to_theta = [function (x) <<NEWLINE>>log(x)]

                PRIOR->PARAMETERS=[1, 1e-05]

                initialise log_precision[11]

                fixed=[1]

                user.scale=[1]

                n=[10]

                m=[0]

                ndata=[10]

                compute=[0]

                Aext=[(null)]

                AextPrecision=[1e+08]

                output:

                        summary=[1]

                        return.marginals=[1]

                        nquantiles=[3]  [ 0.025 0.5 0.975 ]

                        ncdf=[0]  [ ]

        parse section=[1] name=[INLA.Data1] type=[DATA]

        inla_parse_data [section 1]...

                tag=[INLA.Data1]

                likelihood=[GAUSSIAN]

                file->name=[data.files/filef602d6c7076]

                read n=[30] entries from file=[data.files/filef602d6c7076]

                        0/10  (idx,a,y) = (0, 1, -0.468659)

                        1/10  (idx,a,y) = (1, 1, -2.93478)

                        2/10  (idx,a,y) = (2, 1, -1.34445)

                        3/10  (idx,a,y) = (3, 1, 1.42867)

                        4/10  (idx,a,y) = (4, 1, -1.0294)

                use variant [0]

                        bit 0 is off

                        bit 1 is off

                        bit 2 is off

                        bit 3 is off

                initialise log_precision[4]

                fixed=[0]

                PRIOR->name=[loggamma]

                PRIOR->from_theta=[function (x) <<NEWLINE>>exp(x)]

                PRIOR->to_theta = [function (x) <<NEWLINE>>log(x)]

                PRIOR->PARAMETERS=[1, 5e-05]

        section=[3] name=[(Intercept)] type=[LINEAR]

        inla_parse_linear...

                section[(Intercept)]

                dir=[fixed.effect00000001]

                file for covariates=[data.files/filef601b776aef]

                read n=[20] entries from file=[data.files/filef601b776aef]

                file=[data.files/filef601b776aef] 0/10  (idx,y) = (0, 1)

                file=[data.files/filef601b776aef] 1/10  (idx,y) = (1, 1)

                file=[data.files/filef601b776aef] 2/10  (idx,y) = (2, 1)

                file=[data.files/filef601b776aef] 3/10  (idx,y) = (3, 1)

                file=[data.files/filef601b776aef] 4/10  (idx,y) = (4, 1)

                prior mean=[0]

                prior precision=[0]

                compute=[1]

                output:

                        summary=[1]

                        return.marginals=[1]

                        nquantiles=[3]  [ 0.025 0.5 0.975 ]

                        ncdf=[0]  [ ]

        section=[4] name=[x] type=[LINEAR]

        inla_parse_linear...

                section[x]

                dir=[fixed.effect00000002]

                file for covariates=[data.files/filef60e8544c8]

                read n=[20] entries from file=[data.files/filef60e8544c8]

                file=[data.files/filef60e8544c8] 0/10  (idx,y) = (0, -0.23433)

                file=[data.files/filef60e8544c8] 1/10  (idx,y) = (1, -1.46739)

                file=[data.files/filef60e8544c8] 2/10  (idx,y) = (2, -0.672224)

                file=[data.files/filef60e8544c8] 3/10  (idx,y) = (3, 0.714334)

                file=[data.files/filef60e8544c8] 4/10  (idx,y) = (4, -0.514702)

                prior mean=[0]

                prior precision=[0.001]

                compute=[1]

                output:

                        summary=[1]

                        return.marginals=[1]

                        nquantiles=[3]  [ 0.025 0.5 0.975 ]

                        ncdf=[0]  [ ]

        Index table: number of entries[3], total length[12]

                tag                            start-index     length

                Predictor                               0         10

                (Intercept)                            10          1

                x                                      11          1

        parse section=[5] name=[INLA.Parameters] type=[INLA]

        inla_parse_INLA...

                section[INLA.Parameters]

                        lincomb.derived.only = [Yes]

                        lincomb.derived.correlation.matrix = [No]

                global.node.factor = 0.500

                reordering = -1

Contents of ai_param 0x85f5608

        Optimiser: DEFAULT METHOD

                Option for domin-BFGS: epsx = 0.0005

                Option for domin-BFGS: epsf = 0.0005

                Option for domin-BFGS: epsg = 0.0005

                Option for GSL-BFGS2: tol  = 0.1

                Option for GSL-BFGS2: epsg = 0.0005

                Option for GSL-BFGS2: step_size = 1

                Restart: 0

                Mode known: No

        Gaussian approximation:

                abserr_func = 0.0005

                abserr_step = 0.0005

                optpar_fp = 0

                optpar_nr_step_factor = -0.1

        Gaussian data: Yes

        Strategy:       Use the Gaussian approximation

        Fast mode:      On

        Use linear approximation to log(|Q +c|)? Yes

                Method:  Compute the derivative exact

        SI directory:   <NONE>

        Parameters for improved approximations

                Number of points evaluate:       9

                Step length to compute derivatives numerically:  0.000122

                Cutoff value to construct local neigborhood:     0.000000

        Log calculations:        On

        Log calculated marginal for the hyperparameters:         On

        Integration strategy:    Use points from Central Composite Design (CCD)

                f0 (CCD only):   1.100000

                dz (GRID only):  1.000000

                Adjust weights (GRID only):      On

                Difference in log-density limit (GRID only):     2.500000

                Skip configurations with (presumed) small density (GRID only):   On

        Gradient is computed using Forward difference with step-length 0.010000

        Hessian is computed using Central difference with step-length 0.100000

        Hessian matrix is forced to be a diagonal matrix? [No]

        Compute effective number of parameters? [Yes]

        Perform a Monte Carlo error-test? [No]

        Interpolator [Auto]

        CPO required diff in log-density [3]

        Adaptive estimation of the Hessian:

                Status     [On]

                Max trials [1000]

                Scale      [1.01]

        Numerical integration of hyperparameters:

                Maximum number of function evaluations [100000]

                Relative error ....................... [1e-05]

                Absolute error ....................... [1e-06]

        To stabalise the numerical optimisation:

                Minimum value of the -Hesssian [0]

        CPO manual calculation[No]

inla_build: check for unused entries in[Model.ini]

        Unused entry [INLA.Data1|weights]=[$inladatadir/filef6033146c0c]

inla_build: [Model.ini] contain[1] unused entries. PLEASE CHECK

 *** Computations finished at Wed Jan 25 13:03:33 EAST 2012

 *** Transfer the results...

 *** Since you are using Cygwin/Windows, the transfer is done in a

 *** more robust way which is somewhat slower; sorry about that...

* ============================================================================ *

* Access to this computer system is only for authorised QUT staff and students *

* and external persons authorised by QUT.                                      *

* ---------------------------------------------------------------------------- *

* WARNING:        It is a criminal offence to:                                 *

*                 i.  Obtain access to data without authority.                 *

*                 ii. Damage, delete, alter or insert data without authority.  *

* ---------------------------------------------------------------------------- *

* Communications on or through QUT's computer systems are monitored and        *

* recorded to secure effective system operation and for other lawful purposes. *

* ============================================================================ *

..done

[1] "collect fixed effects"

[1] "No fixed effects"

[1] "lincomb "

[1] "collect lincombs"

No lincomb effets

[1] "collect lincombs"

No lincomb effets

[1] "collect random effects"

No random effets

[1] "no si configurations to collect from dir= inla.model/results.files/si"

[1] "collect random effects"

No random effets

 

The  computation all appears to happen properly, the Model.ini file doesn't look ridiculous but it looks like nothing gets written to results.files/joint.dat

[Finn Lindgren]

I think the problem may be a mismatch between the rinla package on you own computer and the inla version on your supercomputer. In your output I can see two "hgid" lines and they identify very different versions of inla; one from this year and one from November last year. It also warns about unused lines in model.ini, which also indicates a version mismatch. You'll want to have the "testing" version of inla on both systems.

Finn

[Sam Clifford]

Thanks! That did it.