error: caught illegal operation -- illegal operand

1,174 views

Skip to first unread message

Jae Choi

unread,

Aug 13, 2015, 5:19:37 PM8/13/15

to R-inla discussion group

Hi inla gurus,

I am trying to do a very basic SPDE model (of bathymetry). Works excellent when testing on my desktop ( Intel(R) Core(TM) i7-5960X ) and takes only a few seconds to solve, with very reasonable results.

When I try to run the exact same model on another system ( Intel(R) Xeon(R) CPU X7460 ) with the same software it breaks with an illegal operation error (see below for traceback). I have tried on another sister server with the exact same specifications and get the same error, so suspecting it might be a hardware issue? but not sure why that would be?

It seems to go through all processing fine to the "Compute the marginal for the hyperparameters..." and timings calculation stage and a bit further. Then crashes. I cannot even catch it with a "try" statement which in itself is strange.

I am wondering if I can get a hint as to why this may be happening. Maybe I am specifying something incorrectly or is there a software option that I should try? I have tried with and without priors, modifying the h-parameter, number of cpu's to no avail. Both systems have the same version of all software and plenty of RAM:

- OS - GNU/linux-4.1.4

- R version 3.2.1 (2015-06-18)

- INLA version ............: 0.0-1438458503

INLA date ...............: Sat 1 Aug 21:48:23 CEST 2015

(I've also tried with the "Stable" version and it results in the same issue)

Help would be appreciated as I think this is beyond me! :)

Thanks,

Jae

---

hgid: ee0736049c67 date: Sat Aug 01 21:38:18 2015 +0200

Report bugs to <he...@r-inla.org>

Processing file [/tmp/RtmpctKzhr/kaos/498439/Model.ini] max_threads=[2]

inla_build...

number of sections=[7]

parse section=[6] name=[INLA.Expert] type=[EXPERT]

inla_parse_expert...

section[INLA.Expert]

cpo.manual=[0]

jp.R_HOME=[(null)]

jp.Rfile=[(null)]

jp.func=[(null)]

parse section=[0] name=[INLA.Model] type=[PROBLEM]

inla_parse_problem...

name=[INLA.Model]

openmp.strategy=[default]

store results in directory=[/tmp/RtmpctKzhr/kaos/498439/results.files]

output:

cpo=[0]

po=[0]

dic=[1]

kld=[1]

mlik=[1]

q=[0]

graph=[0]

gdensity=[0]

hyperparameters=[1]

summary=[1]

return.marginals=[1]

nquantiles=[3] [ 0.025 0.5 0.975 ]

ncdf=[0] [ ]

parse section=[2] name=[Predictor] type=[PREDICTOR]

inla_parse_predictor ...

section=[Predictor]

dir=[predictor]

PRIOR->name=[loggamma]

PRIOR->from_theta=[function (x) <<NEWLINE>>exp(x)]

PRIOR->to_theta = [function (x) <<NEWLINE>>log(x)]

PRIOR->PARAMETERS=[1, 1e-05]

initialise log_precision[11]

fixed=[1]

user.scale=[1]

n=[396]

m=[693]

ndata=[693]

compute=[1]

read offsets from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e]

read n=[2178] entries from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e]

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 0/1089 (idx,y) = (0, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 1/1089 (idx,y) = (1, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 2/1089 (idx,y) = (2, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 3/1089 (idx,y) = (3, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 4/1089 (idx,y) = (4, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 5/1089 (idx,y) = (5, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 6/1089 (idx,y) = (6, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 7/1089 (idx,y) = (7, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 8/1089 (idx,y) = (8, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 9/1089 (idx,y) = (9, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 10/1089 (idx,y) = (10, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 11/1089 (idx,y) = (11, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 12/1089 (idx,y) = (12, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 13/1089 (idx,y) = (13, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 14/1089 (idx,y) = (14, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 15/1089 (idx,y) = (15, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 16/1089 (idx,y) = (16, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 17/1089 (idx,y) = (17, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 18/1089 (idx,y) = (18, 0)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb237a4490e] 19/1089 (idx,y) = (19, 0)

Aext=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21428def2]

AextPrecision=[3.269e+06]

output:

summary=[1]

return.marginals=[1]

nquantiles=[3] [ 0.025 0.5 0.975 ]

ncdf=[0] [ ]

parse section=[1] name=[INLA.Data1] type=[DATA]

inla_parse_data [section 1]...

tag=[INLA.Data1]

family=[GAUSSIAN]

likelihood=[GAUSSIAN]

file->name=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb27b3da4fb]

file->name=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb2110a46cf]

read n=[2079] entries from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb27b3da4fb]

0/693 (idx,a,y,d) = (0, 1, 6.90915, 1)

1/693 (idx,a,y,d) = (1, 1, 6.90776, 1)

2/693 (idx,a,y,d) = (2, 1, 6.90885, 1)

3/693 (idx,a,y,d) = (3, 1, 6.90723, 1)

4/693 (idx,a,y,d) = (4, 1, 6.90921, 1)

5/693 (idx,a,y,d) = (5, 1, 6.91301, 1)

6/693 (idx,a,y,d) = (6, 1, 6.91567, 1)

7/693 (idx,a,y,d) = (7, 1, 6.91001, 1)

8/693 (idx,a,y,d) = (8, 1, 6.90822, 1)

9/693 (idx,a,y,d) = (9, 1, 6.90877, 1)

10/693 (idx,a,y,d) = (10, 1, 6.90864, 1)

11/693 (idx,a,y,d) = (11, 1, 6.91177, 1)

12/693 (idx,a,y,d) = (12, 1, 6.91479, 1)

13/693 (idx,a,y,d) = (13, 1, 6.91694, 1)

14/693 (idx,a,y,d) = (14, 1, 6.91969, 1)

15/693 (idx,a,y,d) = (15, 1, 6.92163, 1)

16/693 (idx,a,y,d) = (16, 1, 6.9233, 1)

17/693 (idx,a,y,d) = (17, 1, 6.92471, 1)

18/693 (idx,a,y,d) = (18, 1, 6.9274, 1)

19/693 (idx,a,y,d) = (19, 1, 6.92915, 1)

likelihood.variant=[0]

initialise log_precision[4]

fixed=[0]

PRIOR->name=[loggamma]

PRIOR->from_theta=[function (x) <<NEWLINE>>exp(x)]

PRIOR->to_theta = [function (x) <<NEWLINE>>log(x)]

PRIOR->PARAMETERS=[1, 5e-05]

Link model [IDENTITY]

Link order [-1]

Link ntheta [0]

mix.use[0]

parse section=[4] name=[spatial.field] type=[FFIELD]

inla_parse_ffield...

section=[spatial.field]

dir=[random.effect00000001]

model=[spde2]

PRIOR0->name=[mvnorm]

PRIOR0->from_theta=[function (x) <<NEWLINE>>x]

PRIOR0->to_theta = [function (x) <<NEWLINE>>x]

PRIOR0->PARAMETERS0[0]=[0]

PRIOR0->PARAMETERS0[1]=[0]

PRIOR0->PARAMETERS0[2]=[0.1]

PRIOR0->PARAMETERS0[3]=[0]

PRIOR0->PARAMETERS0[4]=[0]

PRIOR0->PARAMETERS0[5]=[0.1]

correct=[-1]

constr=[0]

diagonal=[0]

id.names=<not present>

compute=[1]

nrep=[1]

ngroup=[1]

read covariates from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2]

read n=[792] entries from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2]

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 0/396 (idx,y) = (0, 109)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 1/396 (idx,y) = (1, 110)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 2/396 (idx,y) = (2, 111)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 3/396 (idx,y) = (3, 112)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 4/396 (idx,y) = (4, 113)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 5/396 (idx,y) = (5, 114)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 6/396 (idx,y) = (6, 115)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 7/396 (idx,y) = (7, 116)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 8/396 (idx,y) = (8, 117)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 9/396 (idx,y) = (9, 118)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 10/396 (idx,y) = (10, 119)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 11/396 (idx,y) = (11, 120)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 12/396 (idx,y) = (12, 121)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 13/396 (idx,y) = (13, 122)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 14/396 (idx,y) = (14, 123)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 15/396 (idx,y) = (15, 124)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 16/396 (idx,y) = (16, 125)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 17/396 (idx,y) = (17, 126)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 18/396 (idx,y) = (18, 127)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb21897a2a2] 19/396 (idx,y) = (19, 128)

spde2.prefix = [/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb260ab6c14/file1eb2519a83ea.]

spde2.transform = [identity]

ntheta = [2]

initialise theta[0]=[0]

fixed[0]=[0]

initialise theta[1]=[0]

fixed[1]=[0]

computed/guessed rank-deficiency = [0]

output:

summary=[1]

return.marginals=[1]

nquantiles=[3] [ 0.025 0.5 0.975 ]

ncdf=[0] [ ]

section=[3] name=[intercept] type=[LINEAR]

inla_parse_linear...

section[intercept]

dir=[fixed.effect00000001]

file for covariates=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b]

read n=[792] entries from file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b]

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 0/396 (idx,y) = (0, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 1/396 (idx,y) = (1, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 2/396 (idx,y) = (2, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 3/396 (idx,y) = (3, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 4/396 (idx,y) = (4, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 5/396 (idx,y) = (5, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 6/396 (idx,y) = (6, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 7/396 (idx,y) = (7, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 8/396 (idx,y) = (8, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 9/396 (idx,y) = (9, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 10/396 (idx,y) = (10, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 11/396 (idx,y) = (11, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 12/396 (idx,y) = (12, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 13/396 (idx,y) = (13, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 14/396 (idx,y) = (14, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 15/396 (idx,y) = (15, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 16/396 (idx,y) = (16, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 17/396 (idx,y) = (17, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 18/396 (idx,y) = (18, 1)

file=[/tmp/RtmpctKzhr/kaos/498439/data.files/file1eb26cad128b] 19/396 (idx,y) = (19, 1)

prior mean=[0]

prior precision=[0.001]

compute=[1]

output:

summary=[1]

return.marginals=[1]

nquantiles=[3] [ 0.025 0.5 0.975 ]

ncdf=[0] [ ]

Index table: number of entries[3], total length[2121]

tag start-index length

Predictor 0 1089

spatial.field 1089 1031

intercept 2120 1

parse section=[5] name=[INLA.Parameters] type=[INLA]

inla_parse_INLA...

section[INLA.Parameters]

lincomb.derived.only = [Yes]

lincomb.derived.correlation.matrix = [No]

global_node.factor = 2.000

global_node.degree = 2147483647

reordering = -1

Contents of ai_param 0x31c7b20

Optimiser: DEFAULT METHOD

Option for domin-BFGS: epsx = 0.005

Option for domin-BFGS: epsf = 1e-05 (rounding error)

Option for domin-BFGS: epsg = 0.005

Option for GSL-BFGS2: tol = 0.1

Option for GSL-BFGS2: step_size = 1

Option for GSL-BFGS2: epsx = 0.005

Option for GSL-BFGS2: epsf = 0.000353553

Option for GSL-BFGS2: epsg = 0.005

Restart: 0

Mode known: No

Gaussian approximation:

abserr_func = 0.0005

abserr_step = 0.0005

optpar_fp = 0

optpar_nr_step_factor = -0.1

Gaussian data: Yes

Strategy: Use the Gaussian approximation

Fast mode: On

Use linear approximation to log(|Q +c|)? Yes

Method: Compute the derivative exact

Parameters for improved approximations

Number of points evaluate: 9

Step length to compute derivatives numerically: 0.00142527

Stencil to compute derivatives numerically: 5

Cutoff value to construct local neigborhood: 0.0001

Log calculations: On

Log calculated marginal for the hyperparameters: On

Integration strategy: Use points from Central Composite Design (CCD)

f0 (CCD only): 1.100000

dz (GRID only): 1.000000

Adjust weights (GRID only): On

Difference in log-density limit (GRID only): 2.500000

Skip configurations with (presumed) small density (GRID only): On

Gradient is computed using Central difference with step-length 0.005000

Hessian is computed using Central difference with step-length 0.070711

Hessian matrix is forced to be a diagonal matrix? [No]

Compute effective number of parameters? [Yes]

Perform a Monte Carlo error-test? [No]

Interpolator [Auto]

CPO required diff in log-density [3]

Stupid search mode:

Status [On]

Max iter [1000]

Factor [1.05]

Numerical integration of hyperparameters:

Maximum number of function evaluations [100000]

Relative error ....................... [1e-05]

Absolute error ....................... [1e-06]

To stabalise the numerical optimisation:

Minimum value of the -Hesssian [0]

CPO manual calculation[No]

Laplace-correction is Disabled.

inla_build: check for unused entries in[/tmp/RtmpctKzhr/kaos/498439/Model.ini]

inla_INLA...

Strategy = [DEFAULT]

Size is [2121] and strategy [LARGE] is chosen

Size of graph=[2121] constraints=[0]

Found optimal reordering=[amdbarc] nnz(L)=[60426] and use_global_nodes(user)=[no]

List of hyperparameters:

theta[0] = [Log precision for the Gaussian observations]

theta[1] = [Theta1 for spatial.field]

theta[2] = [Theta2 for spatial.field]

Optimise using DEFAULT METHOD

Max.post.marg(theta): log(dens) = 717.492523 fn = 1 theta = 4.005000 0.000000 0.000000

Max.post.marg(theta): log(dens) = 1054.712316 fn = 7 theta = 4.998903 -0.037655 0.027826

Max.post.marg(theta): log(dens) = 1056.394384 fn = 8 theta = 5.003903 -0.037655 0.027826

Max.post.marg(theta): log(dens) = 2908.414383 fn = 14 theta = 13.989033 -0.376551 0.278256

Max.post.marg(theta): log(dens) = 2909.401406 fn = 15 theta = 13.984033 -0.376551 0.278256

Max.post.marg(theta): log(dens) = 2991.393538 fn = 21 theta = 12.557107 -0.322572 0.238368

Max.post.marg(theta): log(dens) = 2991.614182 fn = 23 theta = 12.562107 -0.322572 0.238368

Max.post.marg(theta): log(dens) = 2991.995876 fn = 24 theta = 12.557107 -0.327572 0.238368

Max.post.marg(theta): log(dens) = 3001.960293 fn = 28 theta = 12.944199 -0.337164 0.249151

Max.post.marg(theta): log(dens) = 3001.993920 fn = 29 theta = 12.939199 -0.337164 0.249151

Max.post.marg(theta): log(dens) = 3002.550075 fn = 32 theta = 12.944199 -0.342164 0.249151

Iter=1 |grad|=165 |x-x.old|=5.17 |f-f.old|=2.29e+03

Max.post.marg(theta): log(dens) = 3055.725297 fn = 35 theta = 13.339122 -0.997631 0.887759

Max.post.marg(theta): log(dens) = 3056.087676 fn = 36 theta = 13.334122 -0.997631 0.887759

Max.post.marg(theta): log(dens) = 3057.966729 fn = 42 theta = 13.265766 -0.874951 0.769139

Max.post.marg(theta): log(dens) = 3058.262647 fn = 43 theta = 13.260766 -0.874951 0.769139

Iter=2 |grad|=65.3 |x-x.old|=0.47 |f-f.old|=56

Max.post.marg(theta): log(dens) = 3071.228127 fn = 50 theta = 12.853923 -0.987862 0.877251

Max.post.marg(theta): log(dens) = 3071.255671 fn = 52 theta = 12.858923 -0.987862 0.877251

Iter=3 |grad|=9.21 |x-x.old|=0.254 |f-f.old|=13.3

Max.post.marg(theta): log(dens) = 3071.501537 fn = 58 theta = 12.869235 -1.025263 0.912092

Max.post.marg(theta): log(dens) = 3071.517461 fn = 60 theta = 12.874235 -1.025263 0.912092

Max.post.marg(theta): log(dens) = 3071.526870 fn = 68 theta = 12.878289 -1.035165 0.921317

Iter=4 |grad|=3.07 |x-x.old|=0.039 |f-f.old|=0.286

Max.post.marg(theta): log(dens) = 3071.547014 fn = 74 theta = 12.892994 -1.030746 0.915450

Iter=5 |grad|=0.179 |x-x.old|=0.0121 |f-f.old|=0.033

Max.post.marg(theta): log(dens) = 3071.559237 fn = 82 theta = 12.889278 -1.123142 0.800162

Max.post.marg(theta): log(dens) = 3071.560543 fn = 84 theta = 12.894278 -1.123142 0.800162

Max.post.marg(theta): log(dens) = 3071.565031 fn = 85 theta = 12.889278 -1.128142 0.800162

Iter=6 |grad|=1.79 |x-x.old|=0.0853 |f-f.old|=0.0122

Max.post.marg(theta): log(dens) = 3071.583295 fn = 89 theta = 12.890527 -1.223007 0.696208

Max.post.marg(theta): log(dens) = 3071.583688 fn = 91 theta = 12.895527 -1.223007 0.696208

Max.post.marg(theta): log(dens) = 3071.589122 fn = 92 theta = 12.890527 -1.228007 0.696208

Max.post.marg(theta): log(dens) = 3071.629395 fn = 97 theta = 12.893364 -1.449694 0.460240

Max.post.marg(theta): log(dens) = 3071.634236 fn = 101 theta = 12.893364 -1.454694 0.460240

Max.post.marg(theta): log(dens) = 3071.638291 fn = 104 theta = 12.894359 -1.529261 0.377415

Max.post.marg(theta): log(dens) = 3071.642324 fn = 107 theta = 12.894359 -1.534261 0.377415

Max.post.marg(theta): log(dens) = 3071.643522 fn = 114 theta = 12.894755 -1.565919 0.344461

Max.post.marg(theta): log(dens) = 3071.643622 fn = 121 theta = 12.894978 -1.583751 0.325899

Iter=7 |grad|=1.17 |x-x.old|=0.38 |f-f.old|=0.081

Max.post.marg(theta): log(dens) = 3071.645932 fn = 126 theta = 12.893005 -1.571263 0.347379

Iter=8 |grad|=0.094 |x-x.old|=0.0132 |f-f.old|=0.00567

Max.post.marg(theta): log(dens) = 3071.645954 fn = 135 theta = 12.893614 -1.571369 0.347485

Number of function evaluations = 141

Compute the Hessian using central differences and step_size[0.0707107]. Matrix-type [dense]

147.784207 -1.301336 1.307154

-1.301336 66.427667 -61.227226

1.307154 -61.227226 57.428953

Eigenvectors of the Hessian

0.997205 -0.074708 -0.000487

-0.054398 -0.730543 0.680697

0.051209 0.678768 0.732565

Eigenvalues of the Hessian

147.922321

123.182555

0.535951

StDev/Correlation matrix (scaled inverse Hessian)

0.082270 -0.007076 -0.008882

0.932141 0.991298

1.002529

Compute corrected stdev for theta[0]: negative 1.013434 positive 0.986952

Compute corrected stdev for theta[1]: negative 0.991293 positive 1.000963

Compute corrected stdev for theta[2]: negative 0.193551 positive 1.650463

Max.post.marg(theta): log(dens) = 3071.645954 fn = 169 theta = 12.893614 -1.571369 0.347485

config 0/15=[ 0.00 0.00 0.00] log(rel.dens)= 0.00, [0] accept, compute, 0.12s

config 1/15=[ -1.11 -1.09 -0.21] log(rel.dens)=-1.23, [1] accept, compute, 0.12s

config 2/15=[ 1.88 0.00 0.00] log(rel.dens)=-1.81, [0] accept, compute, 0.12s

config 3/15=[ -1.11 1.10 1.82] log(rel.dens)=-2.52, [1] accept, compute, 0.12s

config 4/15=[ -1.93 0.00 0.00] log(rel.dens)=-1.82, [0] accept, compute, 0.12s

config 5/15=[ -1.11 1.10 -0.21] log(rel.dens)=-1.27, [1] accept, compute, 0.12s

config 6/15=[ 0.00 1.91 0.00] log(rel.dens)=-1.81, [0] accept, compute, 0.12s

config 7/15=[ 1.09 -1.09 1.82] log(rel.dens)=-1.11, [1] accept, compute, 0.12s

config 8/15=[ 0.00 -1.89 0.00] log(rel.dens)=-1.81, [0] accept, compute, 0.12s

config 9/15=[ 1.09 -1.09 -0.21] log(rel.dens)=-1.22, [1] accept, compute, 0.12s

config 10/15=[ 0.00 0.00 3.14] log(rel.dens)=-1.96, [0] accept, compute, 0.12s

config 11/15=[ 1.09 1.10 1.82] log(rel.dens)=-2.62, [1] accept, compute, 0.12s

config 12/15=[ 0.00 0.00 -0.37] log(rel.dens)=-0.14, [0] accept, compute, 0.12s

config 13/15=[ 1.09 1.10 -0.21] log(rel.dens)=-1.27, [1] accept, compute, 0.12s

config 14/15=[ -1.11 -1.09 1.82] log(rel.dens)=-1.02, [0] accept, compute, 0.11s

Combine the densities with relative weights:

config 0/15=[ 0.00 0.00 0.00] weight = 0.553 neff = 376.20

config 1/15=[ 1.88 0.00 0.00] weight = 0.189 neff = 385.98

config 2/15=[ -1.93 0.00 0.00] weight = 0.188 neff = 366.17

config 3/15=[ 0.00 1.91 0.00] weight = 0.188 neff = 374.35

config 4/15=[ 0.00 -1.89 0.00] weight = 0.189 neff = 378.31

config 5/15=[ 0.00 0.00 3.14] weight = 0.162 neff = 375.13

config 6/15=[ 0.00 0.00 -0.37] weight = 1.000 neff = 376.32

config 7/15=[ -1.11 -1.09 1.82] weight = 0.415 neff = 370.88

config 8/15=[ -1.11 -1.09 -0.21] weight = 0.338 neff = 371.73

config 9/15=[ -1.11 1.10 1.82] weight = 0.093 neff = 368.68

config 10/15=[ -1.11 1.10 -0.21] weight = 0.323 neff = 369.34

config 11/15=[ 1.09 -1.09 1.82] weight = 0.380 neff = 382.29

config 12/15=[ 1.09 -1.09 -0.21] weight = 0.341 neff = 383.05

config 13/15=[ 1.09 1.10 1.82] weight = 0.084 neff = 380.25

config 14/15=[ 1.09 1.10 -0.21] weight = 0.325 neff = 380.84

Done.

Expected effective number of parameters: 376.162(4.933), eqv.#replicates: 1.842

DIC:

Mean of Deviance................. -7005.75

Deviance at Mean................. -7382.63

Effective number of parameters... 376.871

DIC.............................. -6628.88

Marginal likelihood: Integration 3070.383813 Gaussian-approx 3069.809447

Compute the marginal for each of the 3 hyperparameters

Interpolation method: Auto

Compute the marginal for theta[0] to theta[2] using numerical integration...

Compute the marginal for theta[0] to theta[2] using numerical integration... Done.

Compute the marginal for the hyperparameters... done.

Store results in directory[/tmp/RtmpctKzhr/kaos/498439/results.files]

Wall-clock time used on [/tmp/RtmpctKzhr/kaos/498439/Model.ini]

Preparations : 0.043 seconds

Approx inference: 5.451 seconds [1.9|0.0|64.4|23.5|10.2]%

Output : 0.231 seconds

---------------------------------

Total : 5.724 seconds

*** caught illegal operation ***

address 0x7f15b2d81a2c, cause 'illegal operand'

Traceback:

1: inla.collect.misc(results.dir, debug)

2: inla.collect.results(results.dir, control.results = cont.results, debug = debug, only.hyperparam = only.hyperparam, file.log = file.log)

3: doTryCatch(return(expr), name, parentenv, handler)

4: tryCatchOne(expr, names, parentenv, handlers[[1L]])

5: tryCatchList(expr, classes, parentenv, handlers)

6: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L msg <- conditionMessage(e) sm <- strsplit(msg, "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && identical(getOption("show.error.messages"), TRUE)) { cat(msg, file = stderr()) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})

7: try(inla.collect.results(results.dir, control.results = cont.results, debug = debug, only.hyperparam = only.hyperparam, file.log = file.log), silent = FALSE)

8: inla(depth ~ -1 + intercept + f(spatial.field, model = S0), data = inputstack, control.compute = list(dic = TRUE, config = TRUE), control.results = list(return.marginals.random = FALSE, return.marginals.predictor = FALSE), control.predictor = list(A = A, compute = TRUE), control.inla = list(h = 0.05), num.threads = 2, working.directory = file.path(tempdir(), Sys.info()["nodename"], dd), verbose = FALSE)

9: doTryCatch(return(expr), name, parentenv, handler)

10: tryCatchOne(expr, names, parentenv, handlers[[1L]])

11: tryCatchList(expr, classes, parentenv, handlers)

12: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L msg <- conditionMessage(e) sm <- strsplit(msg, "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && identical(getOption("show.error.messages"), TRUE)) { cat(msg, file = stderr()) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})

13: try(inla(depth ~ -1 + intercept + f(spatial.field, model = S0), data = inputstack, control.compute = list(dic = TRUE, config = TRUE), control.results = list(return.marginals.random = FALSE, return.marginals.predictor = FALSE), control.predictor = list(A = A, compute = TRUE), control.inla = list(h = 0.05), num.threads = 2, working.directory = file.path(tempdir(), Sys.info()["nodename"], dd), verbose = FALSE), silent = TRUE)

Håvard Rue

unread,

Aug 14, 2015, 7:08:50 AM8/14/15

to Jae Choi, R-inla discussion group

On Thu, 2015-08-13 at 14:19 -0700, Jae Choi wrote:
> Hi inla gurus,
>
> I am trying to do a very basic SPDE model (of bathymetry). Works
> excellent when testing on my desktop ( Intel(R) Core(TM) i7-5960X )
> and takes only a few seconds to solve, with very reasonable results.
>
> When I try to run the exact same model on another system ( Intel(R)
> Xeon(R) CPU X7460 ) with the same software it breaks with an illegal
> operation error (see below for traceback). I have tried on another
> sister server with the exact same specifications and get the same
> error, so suspecting it might be a hardware issue? but not sure why
> that would be?
>
> It seems to go through all processing fine to the "Compute the
> marginal for the hyperparameters..." and timings calculation stage
> and a bit further. Then crashes. I cannot even catch it with a "try"
> statement which in itself is strange.
>
> I am wondering if I can get a hint as to why this may be happening.
> Maybe I am specifying something incorrectly or is there a software
> option that I should try? I have tried with and without priors,
> modifying the h-parameter, number of cpu's to no avail. Both systems
> have the same version of all software and plenty of RAM:
>
> - OS - GNU/linux-4.1.4
> - R version 3.2.1 (2015-06-18)
> - INLA version ............: 0.0-1438458503
> INLA date ...............: Sat 1 Aug 21:48:23 CEST 2015

Hi.

how weird. could be something in the compile of the inla-program that
make it crash for Xeon CPU.

can you try this binary

www.math.ntnu.no/~hrue/hfiles/inla.Xeon

?

do

inla.setOption(inla.call="~/..path this this binary")
inla(...)

let me know how this goes.

H

--
Håvard Rue
Department of Mathematical Sciences
Norwegian University of Science and Technology
N-7491 Trondheim, Norway
Voice: +47-7359-3533 URL : http://www.math.ntnu.no/~hrue
Mobile: +47-9260-0021 Email: havar...@math.ntnu.no

R-INLA: www.r-inla.org

Choi, Jae S.

unread,

Aug 18, 2015, 11:40:32 AM8/18/15

to Håvard Rue, R-inla discussion group

Hello Havard:

Thanks for the binary. Unfortunately, the binary gives the same error. Some operation in the Wall-clock time calculations stage or just afterwards persists in causing the error.

I've tried to run as root in case there was some strange permissions issue. But no such luck. I've also completely re-installed R and all underlying packages in case there were some gcc-related issues, but the same error persists. For now I will run on the I7 processor, but would love to be able to run on the Xeons in future. And would love to be able to catch the error with a 'try(..)'. :)

Not sure what else I could try. Any ideas would be appreciated?

Best wishes,

Jae

Wall-clock time used on [/tmp/RtmpTZqi5x/file5d6c1137abfb/Model.ini]

Preparations : 0.042 seconds

Approx inference: 3.579 seconds [1.9|0.0|71.3|20.8|6.1]%

Output : 0.175 seconds

---------------------------------

Total : 3.796 seconds

*** caught illegal operation ***

address 0x7f74ccbafa2c, cause 'illegal operand'

Håvard Rue

unread,

Aug 18, 2015, 12:28:37 PM8/18/15

to Choi, Jae S., R-inla discussion group

On Tue, 2015-08-18 at 12:39 -0300, Choi, Jae S. wrote:
> Hello Havard:
>
> Thanks for the binary. Unfortunately, the binary gives the same
> error. Some operation in the Wall-clock time calculations stage or
> just afterwards persists in causing the error.
>
> I've tried to run as root in case there was some strange permissions
> issue. But no such luck. I've also completely re-installed R and all
> underlying packages in case there were some gcc-related issues, but
> the same error persists. For now I will run on the I7 processor, but
> would love to be able to run on the Xeons in future. And would love
> to be able to catch the error with a 'try(..)'. :)
>
> Not sure what else I could try. Any ideas would be appreciated?

I have downgraded the compiler from 4.8 to 4.6, and compiled a new one
at the same location

www.math.ntnu.no/~hrue/hfiles/inla.Xeon

can you try that one?

let me know...

Choi, Jae S.

unread,

Aug 18, 2015, 12:41:42 PM8/18/15

to Håvard Rue, R-inla discussion group

Still the same error. See below:

---

Combine the densities with relative weights:

...

Done.

Expected effective number of parameters: 376.137(4.909), eqv.#replicates: 1.842

DIC:

Mean of Deviance................. -7005.62

Deviance at Mean................. -7382.27

Effective number of parameters... 376.649

DIC.............................. -6628.97

Marginal likelihood: Integration 3070.754678 Gaussian-approx 3069.942690

Compute the marginal for each of the 3 hyperparameters

Interpolation method: Auto

Compute the marginal for theta[0] to theta[2] using numerical integration...

Compute the marginal for theta[0] to theta[2] using numerical integration... Done.

Compute the marginal for the hyperparameters... done.

Store results in directory[/tmp/RtmpMlFRZ7/file2ea17ded491/results.files]

Wall-clock time used on [/tmp/RtmpMlFRZ7/file2ea17ded491/Model.ini]

Preparations : 0.042 seconds

Approx inference: 3.126 seconds [2.4|0.0|67.5|23.3|6.8]%

Output : 0.178 seconds

---------------------------------

Total : 3.345 seconds

*** caught illegal operation ***

address 0x7f07ab514a2c, cause 'illegal operand'

Traceback:

1: inla.collect.misc(results.dir, debug)

2: inla.collect.results(results.dir, control.results = cont.results, debug = debug, only.hyperparam = only.hyperparam, file.log = file.log)

3: doTryCatch(return(expr), name, parentenv, handler)

4: tryCatchOne(expr, names, parentenv, handlers[[1L]])

5: tryCatchList(expr, classes, parentenv, handlers)

7: try(inla.collect.results(results.dir, control.results = cont.results, debug = debug, only.hyperparam = only.hyperparam, file.log = file.log), silent = FALSE)

9: doTryCatch(return(expr), name, parentenv, handler)

10: tryCatchOne(expr, names, parentenv, handlers[[1L]])

11: tryCatchList(expr, classes, parentenv, handlers)

Håvard Rue

unread,

Aug 18, 2015, 1:11:41 PM8/18/15

to Choi, Jae S., R-inla discussion group

On Tue, 2015-08-18 at 13:41 -0300, Choi, Jae S. wrote:
> Still the same error. See below:

try again with another new file

Jae Choi

unread,

Aug 20, 2015, 1:31:38 PM8/20/15

to Håvard Rue, R-inla discussion group

Thought I should report back in case anyone else encounters this strange error. Initially it seemed like it was due to aggressive gcc compile ‎flags -O3 being placed in ~/R/Makevars by the package rstan. Now it is confirmed that the issue was really due to Openblas (v 0.2.14 = BLAS 3.5.0) being on some of the servers rather than BLAS (on Xeon based systems). Replacing with BLAS solved all issues.

Original Message
From: Håvard Rue
Sent: Tuesday, August 18, 2015 2:11 PM
To: Choi, Jae S.
Reply To: hr...@math.ntnu.no
Cc: R-inla discussion group
Subject: Re: [r-inla] error: caught illegal operation -- illegal operand

Reply all

Reply to author

Forward

0 new messages