too many items

[Benjamin Garzon]

Hi all, 

I am getting the following error when fitting the model at the bottom:

Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  : 

  too many items

Calls: stan -> sampling -> sampling -> .local -> scan

The dataset is quite large: S=56, C=35000. The model works fine for e.g. C= 100, but now I need to include all the data. The process is just using a small part of the memory of the server (120 GB) and there's plenty of disk space.

Any suggestions on what the problem is and how to make it work? Should I optimize the code somehow? If the dataset is too large, how can I assess what the limit would be ?

Thanks for developing stan!

Best Benjamin

------------------------------------------------

data {

   int<lower=0> S;          // number of subjects

   int<lower=0> C;          // number of connections

   real values[C, S]; // values

   real age[S]; // age

}

parameters {

   // global parameters governing distribution of connection-specific intercepts and slopes

   real<lower=-1, upper=1> intercept_muc_mu;

   real<lower=0, upper=1> intercept_muc_sigma;

   real<lower=-.01, upper=.01> slope_muc_mu;

   real<lower=0, upper=.1> slope_muc_sigma;

   real<lower=-4, upper=0> log_intercept_sigmac_mu;

   real<lower=0, upper=1> log_intercept_sigmac_sigma;

   real<lower=-.03, upper=.03> log_slope_sigmac_mu;

   real<lower=0, upper=.5> log_slope_sigmac_sigma;

   // connection-specific intercepts and slopes

   real<lower=-3, upper=3> intercept_muc[C];

   real<lower=-.3, upper=.3> slope_muc[C];

   real<lower=-6, upper=1> log_intercept_sigmac[C];

   real<lower=-.3, upper=.3> log_slope_sigmac[C]; 

}

transformed parameters {

   real muc[C, S];

   real<lower=0> sigmac[C, S];

   for (c in 1:C) 

   {

        for (s in 1:S){

          muc[c, s] <- intercept_muc[c] + slope_muc[c]*age[s];

          sigmac[c, s] <- exp(log_intercept_sigmac[c] + log_slope_sigmac[c]*age[s]);

        }

   }   

}

model {

   // priors

   intercept_muc_mu ~ normal(0, .5);

   intercept_muc_sigma ~ uniform(0, 1);

   slope_muc_mu ~ normal(0, .005);

   slope_muc_sigma ~ uniform(0, .1);

   log_intercept_sigmac_mu ~ uniform(-4, 0); //normal(-2, .5);

   log_intercept_sigmac_sigma ~ uniform(0, 1); 

   log_slope_sigmac_mu ~ normal(0, .01);

   log_slope_sigmac_sigma ~ uniform(0, .5);

   intercept_muc ~ normal(intercept_muc_mu, intercept_muc_sigma);

   slope_muc ~ normal(slope_muc_mu, slope_muc_sigma);

   log_intercept_sigmac ~ normal(log_intercept_sigmac_mu, log_intercept_sigmac_sigma);

   log_slope_sigmac ~ normal(log_slope_sigmac_mu, log_slope_sigmac_sigma);

   for (c in 1:C) 

   {

        for (s in 1:S){

          print(sigmac[c, s])

          values[c, s] ~ normal(muc[c, s], sigmac[c, s]);

        }

   }

}

[Ben Goodrich]

On Friday, June 3, 2016 at 10:33:36 AM UTC-4, Benjamin Garzon wrote:

I am getting the following error when fitting the model at the bottom:

Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  : 

  too many items

Calls: stan -> sampling -> sampling -> .local -> scan

So you are calling stan() with sample_file specified, the model runs, and then you get this error at the end when it tries to read in the resulting CSV file?

Ben

[Benjamin Garzon]

HI Ben, 

Exactly. It fails after finishing all the iterations. I'm running it with adapt_delta=0.99, step_size=0.1, max_treedepth=12 to avoid getting divergent transitions. Any ideas?

[Ben Goodrich]

On Saturday, June 4, 2016 at 11:08:55 AM UTC-4, Benjamin Garzon wrote:

Exactly. It fails after finishing all the iterations. I'm running it with adapt_delta=0.99, step_size=0.1, max_treedepth=12 to avoid getting divergent transitions. Any ideas?

The CSV file should be sitting in tempdir(). Is it huge or has corrupted characters or something?

Ben

[Benjamin Garzon]

It 's 1.8 G in size, but it does not look corrupted. Is it too large to be handled by the scan() function ? Is there any way around this ? 

On Friday, June 3, 2016 at 4:33:36 PM UTC+2, Benjamin Garzon wrote:

[Ben Goodrich]

On Saturday, June 4, 2016 at 1:41:57 PM UTC-4, Benjamin Garzon wrote:

It 's 1.8 G in size, but it does not look corrupted. Is it too large to be handled by the scan() function ? Is there any way around this ? 

The scan() function is a base R function that seems to have a limit of INT_MAX / 2

http://docs.rexamine.com/R-devel/scan_8c_source.html

so there is nothing we can do about that in rstan. My guess is that you will have to follow

https://groups.google.com/d/msg/stan-users/urZeEMjjPHM/R0S41-hQbPMJ

Ben