se is NA (loo output)
56 views
Skip to first unread message
Jeffrey Erlich
Sep 29, 2017, 12:37:45 AM9/29/17
to brms-users
I am trying to compare two non-linear models of hyperbolic-discounting that I fit with brms. The basic question is to test whether different treatments cause subjects to behave differently (as in the full model) or whether there is just a population scaling effect (e.g. the whole population shifts between treatments) but individual subjects use essentially the same strategy in different treatments.
"Full" Model:
full_model = bf(choice ~ inv_logit((rewmag/(1 + exp(logk)*delay) - smag)/noise),
noise ~ (1|subjid), logk ~ (treat|subjid), nl = TRUE)
full_fit = brm(full_model,
data = all_trials, family = bernoulli(link='identity'),
prior = c(prior(normal(-3,4), nlpar = 'logk'), prior(normal(1,10),lb = 0, nlpar = 'noise')),
inits = "0",
chains = 10, iter = 10000, warmup=2000, cores = 10)
"Simpler" Model
simple_model = bf(choice ~ inv_logit((rewmag/(1 + exp(logk)*delay) - smag)/noise),
noise ~ (1|subjid), logk ~ (1|treat) + (1|subjid), nl = TRUE)
simple_fit = brm(simple_model,
data = all_trials, family = bernoulli(link='identity'),
prior = c(prior(normal(-3,4), nlpar = 'logk'), prior(normal(1,10),lb = 0, nlpar = 'noise')),
inits = "0",
chains = 20, iter = 25000, warmup=10000, cores = 22)
Both models ran with OK diagnostics (reasonable number of effective samples, Rhat ~ 1).However, when I run `loo` to do model comparison I get this:
> loo::loo(extract_log_lik(simple_fit$fit, parameter_name = 'lp__')) . # is the lp__ the right thing to use here?
Computed from 200000 by 1 log-likelihood matrix
Estimate SE
elpd_loo -13748.1 NA
p_loo 91.9 NA
looic 27496.3 NA
Pareto k diagnostic values:
Count Pct
(-Inf, 0.5] (good) 0 0.0%
(0.5, 0.7] (ok) 0 0.0%
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 1 100.0%
See help('pareto-k-diagnostic') for details.
Warning message:
Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Paul Buerkner
Sep 29, 2017, 2:56:19 AM9/29/17
to Jeffrey Erlich, brms-users
You are not using loo correctly.
Go for LOO(full_fit, simple_fit)
--
You received this message because you are subscribed to the Google Groups "brms-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brms-users+unsubscribe@googlegroups.com.
To post to this group, send email to brms-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brms-users/bf72d029-fc35-42b5-81f4-76322964e30c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jeffrey Erlich
Sep 29, 2017, 6:40:51 AM9/29/17
to brms-users
Thanks. I realized that the "lp__" is not the appropriate paramter
But I get an error with LOO
> brms::LOO(full_fit, simple_fit) Switching to pointwise evaluation to reduce RAM requirements. This will likely increase computation time. Error in draws[["C"]][[cov]] <- as_draws_matrix(C[, cov], dim = c(nsamples, : long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138 | |
|
To unsubscribe from this group and stop receiving emails from it, send an email to brms-users+...@googlegroups.com.
Paul Buerkner
Sep 29, 2017, 6:47:52 AM9/29/17
to Jeffrey Erlich, brms-users
Hmm this is a very strange error. Never seen it before, but I am not a Linux user.
Try setting pointwise = FALSE in LOO and see if this changes something.
Best,Try setting pointwise = FALSE in LOO and see if this changes something.
To unsubscribe from this group and stop receiving emails from it, send an email to brms-users+unsubscribe@googlegroups.com.
To post to this group, send email to brms-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brms-users/64f164e7-4469-4700-9489-5c230eb9cbbd%40googlegroups.com.
Jeffrey Erlich
Sep 29, 2017, 6:52:23 AM9/29/17
to brms-users
I get the same error on a mac. Is my fit too big? (it's around 24MB)
> loo_simplest = LOO(simplest_fit, cores = 3)
Switching to pointwise evaluation to reduce RAM requirements.
This will likely increase computation time.
Error in draws[["C"]][[cov]] <- as_draws_matrix(C[, cov], dim = c(nsamples, :
long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138
> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin16.7.0 (64-bit)
Running under: macOS Sierra 10.12.6
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] brms_1.10.0 ggplot2_2.2.1 Rcpp_0.12.12
loaded via a namespace (and not attached):
[1] mvtnorm_1.0-6 lattice_0.20-35 zoo_1.8-0 gtools_3.5.0 assertthat_0.2.0 digest_0.6.12 mime_0.5
[8] R6_2.2.2 plyr_1.8.4 stats4_3.4.1 coda_0.19-1 colourpicker_0.3 rlang_0.1.2 lazyeval_0.2.0
[15] miniUI_0.1.1 Matrix_1.2-10 DT_0.2 shinythemes_1.1.1 shinyjs_0.9.1 stringr_1.2.0 htmlwidgets_0.9
[22] loo_1.1.0 igraph_1.1.2 munsell_0.4.3 shiny_1.0.5 compiler_3.4.1 httpuv_1.3.5 rstan_2.16.2
[29] pkgconfig_2.0.1 base64enc_0.1-3 rstantools_1.3.0 htmltools_0.3.6 tibble_1.3.4 gridExtra_2.3 threejs_0.3.1
[36] matrixStats_0.52.2 dplyr_0.7.3 grid_3.4.1 nlme_3.1-131 xtable_1.8-2 gtable_0.2.0 magrittr_1.5
[43] StanHeaders_2.16.0-1 scales_0.5.0 stringi_1.1.5 reshape2_1.4.2 bindrcpp_0.2 dygraphs_1.1.1.4 xts_0.10-0
[50] tools_3.4.1 glue_1.1.1 shinystan_2.4.0 markdown_0.8 crosstalk_1.0.0 rsconnect_0.8.5 abind_1.4-5
[57] parallel_3.4.1 inline_0.3.14 colorspace_1.3-2 bridgesampling_0.2-2 bayesplot_1.4.0 bindr_0.1 Brobdingnag_1.2-4
Paul Buerkner
Sep 29, 2017, 7:15:36 AM9/29/17
to Jeffrey Erlich, brms-users
Possible, but I can only tell with a reproducible example.
Can you create an issue on github (https://github.com/paul-buerkner/brms/issues) and provide a reproducible example with fake data?
This would be great!
--
You received this message because you are subscribed to the Google Groups "brms-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brms-users+unsubscribe@googlegroups.com.
To post to this group, send email to brms-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brms-users/36006f8e-1cdb-4d3f-9e91-cf2ffbed7581%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages