Different convergence metrics on M1 vs Intel

[Samuel Johnson]

Hi Gang,

I'm getting some different outputs from the same model, code, and data on different mahcines.

I'm running some retrospective analyses of a fisheries model on two Macs, an M1 Max MBP, and a 2013 garbage can Mac Pro with Intel Xeon processors.

On the M1, I get low gradients during optimisation for point estimates, nice HMC posteriors, etc (albeit with weird gradients after calculating standard errors of MLEs, see https://groups.google.com/g/tmb-users/c/PShnM1ZN7ns/m/oFdQWn2DAAAJ)

On the Intel Mac Pro, there are some cases/peels where the model reaches similar or the same final gradients and parameter estimates, but 1 or all of the SEs are NaN after calculating SD reports.

The two machines are running the same TMB package version, but different R versions (M1 is 4.1.2, Mac Pro is 4.2.0)

Does anyone have any insights into this? Would system architecture be a huge issue here?

SessionInfo() outputs:

M1:

R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin21.1.0 (64-bit)
Running under: macOS Monterey 12.4

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.20/lib/libopenblasp-r0.3.20.dylib
LAPACK: /opt/homebrew/Cellar/r/4.1.2/lib/R/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base    

other attached packages:
 [1] knitr_1.36           corrplot_0.90        devtools_2.4.2      
 [4] usethis_2.1.3        MASS_7.3-54          beepr_1.3          
 [7] scales_1.1.1         wesanderson_0.3.6    Rcpp_1.0.8          
[10] RColorBrewer_1.1-2   here_1.0.1           tmbstan_1.0.2      
[13] rstan_2.21.2         StanHeaders_2.21.0-7 TMB_1.9.0          
[16] reshape2_1.4.4       forcats_0.5.1        stringr_1.4.0      
[19] dplyr_1.0.7          purrr_0.3.4          readr_2.0.2        
[22] tidyr_1.1.4          tibble_3.1.5         ggplot2_3.3.5      
[25] tidyverse_1.3.1      kableExtra_1.3.4     tictoc_1.0.1        
[28] bookdown_0.24        coda_0.19-4        

loaded via a namespace (and not attached):
 [1] matrixStats_0.61.0 fs_1.5.0           lubridate_1.8.0    webshot_0.5.2    
 [5] httr_1.4.2         rprojroot_2.0.2    tools_4.1.2        backports_1.3.0  
 [9] utf8_1.2.2         R6_2.5.1           DBI_1.1.1          colorspace_2.0-2  
[13] withr_2.4.2        tidyselect_1.1.1   gridExtra_2.3      prettyunits_1.1.1
[17] processx_3.5.2     curl_4.3.2         compiler_4.1.2     cli_3.1.0        
[21] rvest_1.0.2        xml2_1.3.2         desc_1.4.0         callr_3.7.0      
[25] systemfonts_1.0.3  digest_0.6.28      rmarkdown_2.11     svglite_2.0.0    
[29] pkgconfig_2.0.3    htmltools_0.5.2    sessioninfo_1.2.1  highr_0.9        
[33] dbplyr_2.1.1       fastmap_1.1.0      rlang_0.4.12       readxl_1.3.1      
[37] rstudioapi_0.13    farver_2.1.0       jquerylib_0.1.4    generics_0.1.1    
[41] jsonlite_1.7.2     inline_0.3.19      magrittr_2.0.1     loo_2.4.1        
[45] Matrix_1.3-4       munsell_0.5.0      fansi_0.5.0        lifecycle_1.0.1  
[49] yaml_2.2.1         stringi_1.7.5      pkgbuild_1.2.0     plyr_1.8.6        
[53] grid_4.1.2         crayon_1.4.2       lattice_0.20-45    haven_2.4.3      
[57] hms_1.1.1          ps_1.6.0           pillar_1.6.4       pkgload_1.2.3    
[61] codetools_0.2-18   stats4_4.1.2       reprex_2.0.1       glue_1.4.2        
[65] evaluate_0.14      V8_3.5.0           remotes_2.4.1      RcppParallel_5.1.4
[69] modelr_0.1.8       vctrs_0.3.8        tzdb_0.2.0         testthat_3.1.0    
[73] cellranger_1.1.0   gtable_0.3.0       assertthat_0.2.1   cachem_1.0.6      
[77] xfun_0.28          broom_0.7.10       viridisLite_0.4.0  audio_0.1-8      
[81] memoise_2.0.0      ellipsis_0.3.2   

Mac Pro:

R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin21.3.0 (64-bit)
Running under: macOS Monterey 12.1

Matrix products: default
BLAS:   /usr/local/Cellar/openblas/0.3.20/lib/libopenblasp-r0.3.20.dylib
LAPACK: /usr/local/Cellar/r/4.2.0/lib/R/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base    

other attached packages:
 [1] corrplot_0.92        devtools_2.4.3       usethis_2.1.6      
 [4] MASS_7.3-56          beepr_1.3            scales_1.2.0        
 [7] wesanderson_0.3.6    Rcpp_1.0.8.3         RColorBrewer_1.1-3  
[10] here_1.0.1           tmbstan_1.0.4        rstan_2.21.5        
[13] StanHeaders_2.21.0-7 TMB_1.9.0            reshape2_1.4.4      
[16] forcats_0.5.1        stringr_1.4.0        dplyr_1.0.9        
[19] purrr_0.3.4          readr_2.1.2          tidyr_1.2.0        
[22] tibble_3.1.7         ggplot2_3.3.6        tidyverse_1.3.1    
[25] kableExtra_1.3.4     tictoc_1.0.1         bookdown_0.26      
[28] coda_0.19-4        

loaded via a namespace (and not attached):
 [1] matrixStats_0.62.0 fs_1.5.2           lubridate_1.8.0    webshot_0.5.3    
 [5] httr_1.4.3         rprojroot_2.0.3    tools_4.2.0        backports_1.4.1  
 [9] utf8_1.2.2         R6_2.5.1           DBI_1.1.2          colorspace_2.0-3  
[13] withr_2.5.0        tidyselect_1.1.2   gridExtra_2.3      prettyunits_1.1.1
[17] processx_3.5.3     compiler_4.2.0     cli_3.3.0          rvest_1.0.2      
[21] xml2_1.3.3         desc_1.4.1         callr_3.7.0        systemfonts_1.0.4
[25] digest_0.6.29      rmarkdown_2.14     svglite_2.1.0      pkgconfig_2.0.3  
[29] htmltools_0.5.2    sessioninfo_1.2.2  dbplyr_2.1.1       fastmap_1.1.0    
[33] rlang_1.0.2        readxl_1.4.0       rstudioapi_0.13    generics_0.1.2    
[37] jsonlite_1.8.0     inline_0.3.19      magrittr_2.0.3     loo_2.5.1        
[41] Matrix_1.4-1       munsell_0.5.0      fansi_1.0.3        lifecycle_1.0.1  
[45] stringi_1.7.6      brio_1.1.3         pkgbuild_1.3.1     plyr_1.8.7        
[49] grid_4.2.0         crayon_1.5.1       lattice_0.20-45    haven_2.5.0      
[53] hms_1.1.1          knitr_1.39         ps_1.7.0           pillar_1.7.0      
[57] pkgload_1.2.4      codetools_0.2-18   stats4_4.2.0       reprex_2.0.1      
[61] glue_1.6.2         evaluate_0.15      remotes_2.4.2      RcppParallel_5.1.5
[65] modelr_0.1.8       vctrs_0.4.1        tzdb_0.3.0         testthat_3.1.4    
[69] cellranger_1.1.0   gtable_0.3.0       assertthat_0.2.1   cachem_1.0.6      
[73] xfun_0.31          broom_0.8.0        viridisLite_0.4.0  audio_0.1-10

[Cole Monnahan - NOAA Federal]

Hi Sam, that's not totally unexpected for HMC via tmbstan, because the autodiff gradients are **slightly** different and that slowly diverges and so usually by the end of the chains two architectures can have completely different samples. Kasper found this when building out tmbstan and comparing to Stan. Stan-dev also talks about this in their manual. But I can't see why that would happen with optimization since it's converging to a stable point. Is your model really unstable? Seems like this and the other issue point to some kind of erratic behavior of the model. Do the SEs compare well to posterior marginal SDs for fixed effects? -Cole