Stan crashes after first chain completes

[James Camac]

Hi everyone,

I'm working with a relatively simple model being applied to a large amount of data (180,000 observations). I need to extract the summary estimates of both true_dbh1 and true_dbh2 (i.e. 180,000 x 2 estimates).

The model works fine on subsets of data (e.g. 20,000 observations). But when I apply it to the entire dataset it seems to crash after the first chain completes.

I thought this may be due to it maxing out the RAM. But throughout it's entire run it was using ~ 3gb (1gb per chain) out of a possible 16 gb of RAM.

The model takes about 7 hours to run so it's difficult to diagnose the problem.

I'm using rstan 2.6 and mclapply to run the chains in parallel via the terminal.

Attached is the console crash report, data and model.

Any advice would be greatly appreciated!

  stan_data <- list(

    n_obs = nrow(data),

    census_length = data$census_interval,

    obs_dbh1 = data$dbh_prev,

    obs_dbh2 = data$dbh)

  

  true_dbh_model <-'

    data {

      int<lower=1> n_obs;

      vector[n_obs] census_length;

      vector[n_obs] obs_dbh1;

      vector[n_obs] obs_dbh2;

    }

  

  parameters {

    real<lower=0> true_dbh1[n_obs];

    real log_mu_true_dbh1;

    real<lower=0> log_sigma_true_dbh1;

    real<lower=0> dbh_increment[n_obs];

  }

  

  model {

    real true_dbh2[n_obs];

    

    for (i in 1:n_obs) {

      // Calculating true growth for fitted data

      true_dbh2[i] <- true_dbh1[i] + (dbh_increment[i] * census_length[i]);

    }

    

    // Observation error

    obs_dbh1 ~ normal(true_dbh1, 0.75);

    obs_dbh2 ~ normal(true_dbh2, 0.75);

    

    // Priors

    log_mu_true_dbh1 ~ normal(0,2.5);

    log_sigma_true_dbh1 ~ cauchy(0,2.5);

    true_dbh1 ~ lognormal(log_mu_true_dbh1, log_sigma_true_dbh1);

    dbh_increment ~ lognormal(0,1);

  }

  

  generated quantities {

    real true_dbh2[n_obs];

    

    for (i in 1:n_obs) {

      // recalculating true growth for fitted data

      true_dbh2[i] <- true_dbh1[i] + (dbh_increment[i] * census_length[i]);

    }

  }'

  

  fit <- sflist2stanfit(mclapply(1:3, mc.cores=3,

                                 function(i) {

                                   stan(model_code = true_dbh_model, 

                                        data= stan_data,

                                        pars = c("log_mu_true_dbh1","log_sigma_true_dbh1",

                                                 "true_dbh1","true_dbh2"), 

                                        chains = 1,

                                        chain_id = i,

                                        iter = 1000, 

                                        control=list(stepsize=0.05, adapt_delta=0.9),

                                        refresh=1,

                                        thin=5) 

                                   }))

[James Camac]

Should have also pointed out. That I either don't get an error message in terminal (R totally crashes) or it's the generic sflist needs to be a list error.

Error in sflist2stanfit(mclapply(1:3, mc.cores = 3, function(i) { : 

  'sflist' must be a list of 'stanfit' objects

[Krzysztof Sakrejda]

There are a couple of issues open with rstan related to passing large amounts of data in or out of it.  I imagine but don't know for sure that this is related.  To find out you could run R with gdb and get a backtrace but that doesn't help fix the issue.  I would say try CmdStan (I've used it when the rstan IO became a problem) and then you'll just need to find a way of dealing with the awkwardly large .csv files.

Krzysztof

[Bob Carpenter]

Sorry about that --- it's rude of Stan to crash R.

Is there a way to run RStan so that it only outputs to
file and doesn't save internally? If so, that would be
an even better alternative that would keep you in R.

Otherwise, I'd suggest running large jobs in CmdStan. It's
easy to export the data from RStan and once you have
RStan installed, you have all the pieces on your machine
you need to run CmdStan. Then you can read the data back into
RStan in one go for analysis and it shouldn't have a problem with
sizing.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[James Camac]

Thanks Krzysztof & Bob, I'm not aware of a way to make stan save the output externally rather than internally.

What I originally was doing was wrapping the stan function within another function that runs stan and automatically saves the output as a .rds.

But then I kept running into this issue. So I decided to run rstan by itself. Still no cigar. ;)

I did look into specifying a sample_file in the stan function. But I couldn't get it to work.. possibly because I wasn't specifying it correctly.

I did something like...

stan(...., sample_file='~/Desktop/sample.file.csv') but it never actually produced the file.

Though I don't think this helps as stan still saves things internally.

I'll look into CmdStan. Hopefully, it's fairly straightforward ;). Though Ideally I'd like to stay within Rstan as this model is part of a broader project that is pretty much all R based.

[James Camac]

Also if you set adapt_delta to 0.1 the model at least runs within about an hour (instead of 7) and the error still occurs.

 

[Ben Goodrich]

On Monday, July 13, 2015 at 6:58:12 PM UTC-4, James Camac wrote:

I'm working with a relatively simple model being applied to a large amount of data (180,000 observations). I need to extract the summary estimates of both true_dbh1 and true_dbh2 (i.e. 180,000 x 2 estimates).

I'm working on trying to reproduce your problem for real (it is still running), but if it were the case that you only need the posterior means of true_dbh1 and true_dbh2, then you can not include those among the pars argument to stan() and subsequently recover their posterior means with get_posterior_means().

Ben

[James Camac]

Running a single chain revealed the error.

Error: segfault from C stack overflow

Error: C stack usage  140730434019532 is too close to the limit

Postdoctoral Research Fellow
Biological Sciences
Macquarie University

+61-2-9850-9258
0400972404
James Camac

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/t2B2W_AjlDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[James Camac]

Thanks Ben!

I didn't know get_posterior_means() would also give you the estimates of unmonitored parameters!

I'll give it a try! Though is there a way to calculate the Rhats/effective sample sizes associated with these posteriors? Just to make sure they have adequately converged?

[Ben Goodrich]

On Tuesday, July 14, 2015 at 12:31:04 AM UTC-4, James Camac wrote:

I didn't know get_posterior_means() would also give you the estimates of unmonitored parameters!

I'll give it a try! Though is there a way to calculate the Rhats/effective sample sizes associated with these posteriors? Just to make sure they have adequately converged?

No, all you get in that case are the means for each chain. You can eyeball them and also look for instances of n_divergent__ to get a sense of whether the chains were well behaved. But you are basically just guessing.

Ben

[Bob Carpenter]

[a bit of x-over with stan-dev]

R-hat's possible to calculate as you go using sufficient
stat accumulators, but split R-hat's trickier unless you know
the chain length ahead of time because otherwise you need
a lot of buffering to anticipate the split.

n_eff isn't possible because it requires the whole chain
for autocorrelations (some approximation would probalby
be possible, with same split-chain issues as R-hat).

The quantiles are also tough --- you need q% of the samples to
properly compute a q% quantile at each end.

We'd probably want to generalize to the ragged version,
which we haven't decided how to do yet (weight by chain, weight
by draw, or somewhere in between), but either way will
reduce to the usual method in the non-ragged case.

-Bob

> --

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[James Camac]

I ran a debugger on the model running with 1 chain and the adapt_delta set to 0.1 (just to speed things up).

Got the following error:

Process 1343 stopped

* thread #1: tid = 0xd9a9, 0x00000001000e8eb4 libR.dylib`RecursiveRelease(object=0x0000000109e64188, list=0x0000000106fe1bc8) + 4 at memory.c:3179, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7fff5f3ffff8)

    frame #0: 0x00000001000e8eb4 libR.dylib`RecursiveRelease(object=0x0000000109e64188, list=0x0000000106fe1bc8) + 4 at memory.c:3179

Similar to the console error.

But it also produced a report. I don't understand it, and am not sure if it has anything new that isn't documented in the console report I attached earlier... but it might be use to you guys.

[James Camac]

Looks like the model will run if I don't monitor true_dbh1 and true_dbh2.

Also the segfault still occurs if I move the code in the generated quantities to the transformed parameters block - didn't think it would make a difference but thought I'd try.

Looks like the problem I have is related to this issue:

https://github.com/stan-dev/rstan/issues/76

[Krzysztof Sakrejda]

The report looks similar to this one (https://github.com/stan-dev/rstan/issues/76) which is what I was suggesting, but it does help to have it.    Krzysztof

[Andrew Gelman]

I have an intuition that some sort of super-split R-hat exists, following a FFT principle. But I’ve not been able to come up with anything.
A

[Lisa Mu]

For big data, is it possible to apply R package parallelMCMCcombine as detour?

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0108425

Thanks.

Lisa

[Bob Carpenter]

Doing computations on subsamples of data and combining is complicated
if you want to get the correct posterior.

It looks like the various approaches in this paper are approximate.

It's not something on the short-term horizon for Stan, though
we are looking at EP for doing combinations.

- Bob

[James Camac]

For the time being I can get what I want using get_posterior_mean(). I'm confident the chains have converged as running subsets of data indicate it converges pretty quickly.

Though let me know if and when this bug gets resolved ;).

[Bob Carpenter]

Is there an issue in RStan? If not, that needs to be created
and then you can track the progress there.

- Bob

> On Jul 16, 2015, at 3:55 AM, James Camac <james...@gmail.com> wrote:
>
> For the time being I can get what I want using get_posterior_mean(). I'm confident the chains have converged as running subsets of data indicate it converges pretty quickly.
>
> Though let me know if and when this bug gets resolved ;).
>
>