segfault issues on array indices?

[Bob Carpenter]

> On Saturday, October 27, 2012 5:45:29 PM UTC-4, tds151 wrote:
>
> Hello stan developers, I attach an Rstan script and some sample data. You've already generously provided me with
> feedback on model constructions over more complex versions. So I send this note to suggest that this model should
> not fail under Rstan 1.0.3, but it does; actually, it crashes the R session.

Ack -- this shouldn't cause a seg fault, though.

I saw Ben's diagnosis that this is an issue with arrays out
of bounds. I thought we were catching this in the Stan code.

Is the issue that we're not checking the I/O for bounds
enough? Or is it crashing within the model itself?
Maybe I was off-by-one on indexing. The -1 we need
everywhere to translate indexing-by-one and indexing-by-zero
could be biting me.

- Bob

[Bob Carpenter]

Ben wrote this:

> goodrich@CYBERPOWERPC:/tmp/SU/tds$ ./ordMultiProt_E_test --data=datstan_mp.Rdump
>
> Exception: INDEX OPERATOR [] OUT OF BOUNDS; index=1215; lower bound=1; upper bound=1214; index position=1; c3
>
> Diagnostic information:
> Dynamic exception type: std::out_of_range
> std::exception::what: INDEX OPERATOR [] OUT OF BOUNDS; index=1215; lower bound=1; upper bound=1214; index position=1; c3

1. I should've paid more attention to Ben's output.
It's good that Stan's picking up the error -- this is
what it's supposed to do.

Is the exception causing RStan to crash? If so, we
probably just need to put a catch in the appropriate place
to write the exception's type and message back to the R stderr
stream.

I'm not sure what happens during I/O. Our message could
be a lot better in terms of identifying what was going on
when the error occurred (I/O or model, specifically).

As is, I only know from looking at the code that it arose
when some function called stan::math::raise_range_error,
which throws a std::out_of_range exception.

2. I'd like to figure out a simpler policy for RStan user
error reporting than having them run through Stan command
line. It just seems like too much of an installation and
manual reading exercise for an error report.

- Bob

[Ben Goodrich]

On Saturday, October 27, 2012 6:27:22 PM UTC-4, Bob Carpenter wrote:

2.  I'd like to figure out a simpler policy for RStan user
error reporting than having them run through Stan command
line.  It just seems like too much of an installation and
manual reading exercise for an error report. 

If the exception doesn't take down the R process, then the message should  get back to the R process eventually. I think in v1.0.1, R may have been crashing before the error flushed. This may vary depending on what GUI the user is accessing R from.

Ben

[Jiqiang Guo]

I did not try this example.  But tried a simple one:

parameters {

    real y[1];

}

model {

    y[2] ~ normal(0, 1);

}

In RStan, if we set_cppo('debug'), R would not crash for this example.  set_cppo('debug') would not set -DNDEBUG flag. By default, Stan does not set this flag.  If -DNDEBUG is set in Stan, Stan will exit with a segfault for this example.  

When set_cppo('debug') is set in RStan, for this example, R would not crash. 

> set_cppo('debug')
mode debug for compiling C++ code is set
> 
> scode <- '
+ parameters {
+ real y[1];
+ }
+ 

+ model {
+ y[2] ~ normal(0, 1);
+ }
+ '

>
> library(rstan)
> a <- stan(model_code = scode, iter = 10, chains = 1)
TRANSLATING MODEL 'scode' FROM Stan CODE TO C++ CODE NOW.
COMPILING THE C++ CODE FOR MODEL 'scode' NOW.
SAMPLING FOR MODEL 'scode' NOW (CHAIN 1).
Error in sampler$call_sampler(args_list[[i]]) : 

  INDEX OPERATOR [] OUT OF BOUNDS; index=2; lower bound=1; upper bound=1; index position=1; y

error occurred during calling the sampler; sampling not done

> 

Last, just a reminder: when set_cppo set the mode as debug, it would stay in that mode.  So remember to change that back to 'fast' mode if 'debug' mode is not needed any more. 

--

Jiqiang

Ben

--
 
 

[Ben Goodrich]

On Saturday, October 27, 2012 8:14:32 PM UTC-4, Jiqiang Guo wrote:

In RStan, if we set_cppo('debug'), R would not crash for this example.  set_cppo('debug') would not set -DNDEBUG flag. By default, Stan does not set this flag.  

But recent versions of R define NDEBUG by default. See the part about asserts at

http://cran.r-project.org/doc/manuals/R-exts.html#Writing-portable-packages

Ben

[Jiqiang Guo]

Yes, I know that.  So if a user do not call set_cppo at all, R would crash in this case.  I do not know how the exception is checked under different compilation flags.  All I am saying is that currently, it needs -DDEBUG to catch the exception and report the error. 

--

Jiqiang

Ben

--
 
 

[Ben Goodrich]

Is it possible to pass -DDEBUG to R CMD SHLIB unless overruled by a -DNDEBUG in $HOME/.R/Makevars? 

[Jiqiang Guo]

I am afraid not.  In detail, at most I can do is to pass -DDEBUG.  Then the flag used would be 

-DNDEBUG -DDEBUG 

I do not know what would happen for gcc and specific C++ code if both of them are defined. 

--

Jiqiang

--
 
 

[Ben Goodrich]

On Saturday, October 27, 2012 9:25:53 PM UTC-4, Jiqiang Guo wrote:

I am afraid not.  In detail, at most I can do is to pass -DDEBUG.  Then the flag used would be 

-DNDEBUG -DDEBUG 

I do not know what would happen for gcc and specific C++ code if both of them are defined. 

It looks to me that -DNDEBUG overrules -DDEBUG. Try adding combinations of both the CFLAGS in the Stan makefile. In your example, I get a segfault unless only -DDEBUG is given.

Ben

[Ben Goodrich]

Perhaps another question is why does it segfault for some compiler options and throw an exception for others?

For Mav's model,

parameters {

    real y[1];

}

model {

    y[2] ~ normal(0, 1);

}

I compile with -DNDEBUG -g and O0 and run under gdb:

goodrich@CYBERPOWERPC:/tmp/SU/oob$ gdb ./oob

GNU gdb (GDB) 7.4.1-debian

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /tmp/SU/oob/oob...done.

(gdb) catch throw

Catchpoint 1 (throw)

(gdb) run

Starting program: /tmp/SU/oob/oob 

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

STAN SAMPLING COMMAND

data = (specified model requires no data)

init = random initialization

init tries = 1

samples = samples.csv

append_samples = 0

save_warmup = 0

seed = 2890366062 (randomly generated)

chain_id = 1 (default)

iter = 2000

warmup = 1000

thin = 1 (default)

equal_step_sizes = 0

leapfrog_steps = -1

max_treedepth = 10

epsilon = -1

epsilon_pm = 0

delta = 0.5

gamma = 0.05

Program received signal SIGSEGV, Segmentation fault.

0x0000000000411df7 in stan::agrad::var::val (this=0x7fffffffcda0) at src/stan/agrad/agrad.hpp:415

415             return vi_->val_;

(gdb) bt

#0  0x0000000000411df7 in stan::agrad::var::val (this=0x7fffffffcda0) at src/stan/agrad/agrad.hpp:415

#1  0x0000000000412396 in boost::math::isnan<stan::agrad::var> (v=...) at src/stan/agrad/error_handling.hpp:70

#2  0x000000000040e7db in stan::math::(anonymous namespace)::not_nan<stan::agrad::var, double, boost::math::policies::policy<boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy>, false>::check (function=0x49477c "stan::prob::normal_log(%1%)", y=..., name=0x48ca0f "Random variable", result=0x7fffffffcf48)

    at src/stan/math/error_handling.hpp:101

#3  0x000000000042f5c3 in stan::math::check_not_nan<stan::agrad::var, double, boost::math::policies::policy<boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> > (function=0x49477c "stan::prob::normal_log(%1%)", y=..., name=0x48ca0f "Random variable", result=0x7fffffffcf48)

    at src/stan/math/error_handling.hpp:149

#4  0x0000000000424c58 in stan::prob::normal_log<true, stan::agrad::var, int, int, boost::math::policies::policy<boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> > (y=..., mu=@0x7fffffffd10c: 0, sigma=@0x7fffffffd108: 1) at src/stan/prob/distributions/univariate/continuous/normal.hpp:63

#5  0x000000000041c2ff in stan::prob::normal_log<true, stan::agrad::var, int, int> (y=..., mu=@0x7fffffffd10c: 0, sigma=@0x7fffffffd108: 1)

    at src/stan/prob/distributions/univariate/continuous/normal.hpp:137

#6  0x0000000000417a51 in oob_model_namespace::oob_model::log_prob (this=0x7fffffffd610, params_r__=..., params_i__=..., pstream__=0x0) at /tmp/SU/oob/oob.cpp:101

#7  0x00000000004170a3 in stan::model::prob_grad_ad::grad_log_prob (this=0x7fffffffd610, params_r=..., params_i=..., gradient=..., output_stream=0x0) at src/stan/model/prob_grad_ad.hpp:44

#8  0x000000000040ba73 in stan::mcmc::leapfrog (model=..., z=..., x=..., m=..., g=..., epsilon=1, error_msgs=0x0, output_msgs=0x0) at src/stan/mcmc/util.hpp:67

---Type <return> to continue, or q <return> to quit---

#9  0x000000000043a760 in stan::mcmc::hmc_base<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::find_reasonable_parameters (this=0x7fffffffd770) at src/stan/mcmc/hmc_base.hpp:64

#10 0x00000000004309e8 in stan::mcmc::hmc_base<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::hmc_base (this=0x7fffffffd770, model=..., epsilon=-1, epsilon_pm=0, epsilon_adapt=true, delta=0.5, gamma=0.050000000000000003, 

    rand_int=..., params_r=0x7fffffffddf0, params_i=0x7fffffffde10) at src/stan/mcmc/hmc_base.hpp:153

#11 0x0000000000425eaf in stan::mcmc::nuts_diag<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::nuts_diag (this=0x7fffffffd770, model=..., maxdepth=10, epsilon=-1, epsilon_pm=0, epsilon_adapt=true, delta=0.5, 

    gamma=0.050000000000000003, base_rng=..., params_r=0x7fffffffddf0, params_i=0x7fffffffde10) at src/stan/mcmc/nuts_diag.hpp:127

#12 0x000000000041e251 in stan::gm::nuts_command<oob_model_namespace::oob_model> (argc=1, argv=0x7fffffffe2b8) at src/stan/gm/command.hpp:479

#13 0x000000000040e6ae in main (argc=1, argv=0x7fffffffe2b8) at /tmp/SU/oob/oob.cpp:206

So, it looks like it is checking for nan outside of the range. 

But if I compile with -DNDEBUG -g and O0 and run under gdb:

goodrich@CYBERPOWERPC:/tmp/SU/oob$ gdb ./oob

GNU gdb (GDB) 7.4.1-debian

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /tmp/SU/oob/oob...done.

(gdb) catch throw

Catchpoint 1 (throw)

(gdb) run

Starting program: /tmp/SU/oob/oob 

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Catchpoint 1 (exception thrown), 0x00007ffff791db90 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

(gdb) bt

#0  0x00007ffff791db90 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

#1  0x0000000000408221 in stan::math::(anonymous namespace)::raise_range_error (max=1, i=2, msg=0x48d2c8 "y", idx=1) at src/stan/math/matrix.hpp:156

#2  0x00000000004082a2 in stan::math::(anonymous namespace)::check_range (max=1, i=2, msg=0x48d2c8 "y", idx=1) at src/stan/math/matrix.hpp:166

#3  0x000000000041c58c in stan::math::get_base1<stan::agrad::var> (x=..., i=2, error_msg=0x48d2c8 "y", idx=1) at src/stan/math/matrix.hpp:192

#4  0x0000000000417c68 in oob_model_namespace::oob_model::log_prob (this=0x7fffffffd610, params_r__=..., params_i__=..., pstream__=0x6e81a0) at /tmp/SU/oob/oob.cpp:101

#5  0x00000000004172cd in stan::model::prob_grad_ad::grad_log_prob (this=0x7fffffffd610, params_r=..., params_i=..., gradient=..., output_stream=0x6e81a0)

    at src/stan/model/prob_grad_ad.hpp:44

#6  0x000000000041d9a8 in stan::gm::nuts_command<oob_model_namespace::oob_model> (argc=1, argv=0x7fffffffe2b8) at src/stan/gm/command.hpp:374

#7  0x000000000040e75a in main (argc=1, argv=0x7fffffffe2b8) at /tmp/SU/oob/oob.cpp:206

It catches the range-error exception.

I need more eyes.

Ben

[Marcus Brubaker]

See math/matrix.hpp line 164.  The test and exception throwing was conditionally disabled out as it turned out to be a surprisingly large performance hit in some cases.  You could replace it with a STAN_DEBUG define which can be either 0 or 1 and is set based on NDEBUG if it's not defined.  That way we can easily set it  ourselves.

Cheers,

Marcus

Ben

--
 
 

[Ben Goodrich]

I guess we have a few options here. But I still think that the range bounds in users' models should always be checked on the first iteration only. I guess that would entail passing the iteration number to log_prob or defining a bool in the class or something.

Ben

[Marcus Brubaker]

One option is to template out the range checking in the indexing functions and percolate that up to debug and non-debug versions of log_prob.  Then the debug version is called once at the beginning and everything else uses the non-debug versions.

Cheers,

Marcus

--
 
 

[Bob Carpenter]

On 10/28/12 3:04 PM, Marcus Brubaker wrote:
> One option is to template out the range checking in the indexing functions and percolate that up to debug and non-debug
> versions of log_prob. Then the debug version is called once at the beginning and everything else uses the non-debug
> versions.
>
> Cheers,
> Marcus
>

> On Sun, Oct 28, 2012 at 2:44 PM, Ben Goodrich <goodri...@gmail.com <mailto:goodri...@gmail.com>> wrote:
>
> I guess we have a few options here. But I still think that the range bounds in users' models should always be
> checked on the first iteration only. I guess that would entail passing the iteration number to log_prob or defining
> a bool in the class or something.

I'm afraid we need to keep the range checks in place
all the time by default. Otherwise, there's no way to
stop segfaults from taking down R or the command.

Stan is not like JAGS or BUGS where the loops are all
determined when the data is read in. In Stan, loops can
have bounds that depend on random quantities being sampled:

for (n in 1:floor(Z)) { ... }

I'm OK with a power-user mode that turns all the checking
off. I think that'll be easier to write and understand,
too.

Then we just follow Matt's water-wing metaphor, which,
language being a virus, is now attached lamprey-like to
my poor mind. (And no, mailer, I'm not going to attach
a lamprey to the message, thank you very much.)

- Bob

[Bob Carpenter]

On 10/28/12 12:06 PM, Marcus Brubaker wrote:
> See math/matrix.hpp line 164. The test and exception throwing was conditionally disabled out as it turned out to be a
> surprisingly large performance hit in some cases. You could replace it with a STAN_DEBUG define which can be either 0
> or 1 and is set based on NDEBUG if it's not defined. That way we can easily set it ourselves.

Sorry -- I'm going backwards, but yes, this is what
I think is the right answer.

I liked Eigen's strategy, which for us would mean a

STAN_NO_DEBUG

property, with bounds checking turned off if

(NDEBUG || STAN_NO_DEBUG).

- Bob

[Bob Carpenter]

On 10/27/12 10:12 PM, Ben Goodrich wrote:
> Perhaps another question is why does it segfault for some compiler options and throw an exception for others?

Stan array indexing is std::vector indexing
with operator[](), not with at().

Turns out that in g++, you can enable exception throwing
for out-of-range std::vector operator[]() indexing with
_GLIBCXX_DEBUG.

Otherwise, you get a bad memory write, the behavior
of which isn't specified (i.e., it can vary wildly
by platform, compiler config, or even from run to run
depending on where the program's loaded into memory).

gdb itself will run the code differently, so error
behavior from out-of-range writing can differ there
from its behavior "in the wild".

C++ is perhaps a little too close to the road
to my own taste.

- Bob

[Marcus Brubaker]

I believe what they do is something like:

#ifndef STAN_NO_DEBUG

#ifdef NDEBUG

#define STAN_NO_DEBUG

#endif

#endif

early on in a header file and then they just do

#ifdef STAN_NO_DEBUG 

...

#endif

which I think is a bit cleaner because the checking code code doesn't have to have so many special cases.

Cheers,

Marcus

[Marcus Brubaker]

On a side note, is it possible to fork() and trap the segfault when it's run in R?  Or run the model in a separate process with an exec() call?  These kinds of bugs are likely to be a semi-recurring issue (because, lets be honest, bugs happen and users will always find creative ways to build models which crash) and it'd be nice if it just printed a sufficiently ominous error rather than take down R.  I'm not sure if this has been discussed before, so forgive me if it's come up and been dismissed for whatever reason.

Cheers,

Marcus

[Ben Goodrich]

On Monday, October 29, 2012 4:34:48 PM UTC-4, Marcus Brubaker wrote:

On a side note, is it possible to fork() and trap the segfault when it's run in R?

Not under Windows because fork() isn't supported.

 

Or run the model in a separate process with an exec() call?  

Maybe. I'll have to look into that.

Ben

[Marcus Brubaker]

In theory it should be possible unless R has some restriction on calling exec().  The only problem is that it makes communicating back to R a little more complicated and it take some work to make it cross platform.

Cheers,

Marcus

[Ben Goodrich]

On Monday, October 29, 2012 4:55:33 PM UTC-4, Marcus Brubaker wrote:

In theory it should be possible unless R has some restriction on calling exec().  The only problem is that it makes communicating back to R a little more complicated and it take some work to make it cross platform.

I think everything is executed through R's .Call() interface, which doesn't spawn a separate process. So, we would have to go outside of that somehow. I think if we utilized the snow package, not necessarily to do things in parallel, but to take advantage of the separate processes in a somewhat platform-independent fashion, this idea could work. Another complication would be to get the printing of the sampler progress back to the main R process.

Ben

[Bob Carpenter]

On 10/29/12 4:29 PM, Marcus Brubaker wrote:
...
> I believe what they [Eigen] do is something like:

>
> #ifndef STAN_NO_DEBUG
> #ifdef NDEBUG
> #define STAN_NO_DEBUG
> #endif
> #endif
>
> early on in a header file and then they just do
>
> #ifdef STAN_NO_DEBUG
> ...
> #endif
>
> which I think is a bit cleaner because the checking code code doesn't have to have so many special cases.

That makes sense to me. And also with if(n)def, I don't
think you can do OR-s!

I try to stay away from leaning on the preprocessor.
Another Java holdover habit.

- Bob

[Bob Carpenter]

On 10/29/12 4:55 PM, Marcus Brubaker wrote:

Once we went that far, it'd probably make sense
just to call the regular command line version and
pass the arguments to it, then collect the results
back in R.

Also, R wouldn't have to load the model to run into
R's own memory, so it could be configured to use any
compiler with any options we wanted.

- Bob