Recommended way to check Matrix version for packages using TMB

[James Thorson]

Hi all,

I'm finally distributing packages using TMB via CRAN (e.g., phylosem), and appreciate all the guidance that's searchable online!

However, I already encounter an issue where users have an old version of Matrix (e.g., 1.5.4), then install phylosem which installs a recent TMB version (e.g., 1.9.10) and the conflict between Matrix and TMB causes an inscrutable user error (in this instance, a "Error in nlminb(start = startpar, objective = fn, gradient = gr, control = nlminb.control,  NA/NaN gradient evaluation" message).

Is there any standard advice to check or enforce an update to Matrix for packages using TMB?

I can imagine three options but I'm not knowledgeable about the potential tradeoffs:

1. phylosem (or other package) edits DESCRIPTION to require "Matrix (>= 1.6.3)" ... but this presumably requires updating phylosem DESCRIPTION periodically.
2. phylosem (or other package) includes an .onAttach() that checks Matrix and throws an error if it isn't sufficiently updated ... but this probably doesn't work on GitHub CI Actions, and might be annoying to users
3. TMB updates its DESCRIPTION ... on github master branch it lists "Matrix (>= 1.0-12), which seems out of date

I don't see options 1-2 in glmmTMB so I'm curious to hear suggestions and whether/how others are dealing with the topic?

Jim

[Ben Bolker]

There has been a lot of back and forth on this.

The Matrix package has added this in the newest release
<https://cran.r-project.org/web/packages/Matrix/news.html>:

New R function Matrix.Version, taking no arguments and returning
list(package, abi, suitesparse), a list containing the numeric versions
of the package, its ABI, and the internal SuiteSparse library. ABI
versioning is new: the version is 1 in this release and will be
incremented by 1 in each future release that changes the ABI. Versions
and their components are defined in a header for use by packages with
LinkingTo: Matrix in ‘DESCRIPTION’. See ‘inst/include/Matrix/version.h’.

There is no support in the DESCRIPTION file for requiring a
particular ABI version; the machinery for adding a 'package version
compiled with' tag (stored in inst/<pkg>-version) is here:

https://github.com/glmmTMB/glmmTMB/blob/master/glmmTMB/src/Makevars

and glmmTMB's machinery for checking is here:

https://github.com/glmmTMB/glmmTMB/blob/master/glmmTMB/R/utils.R#L209-L246

What ought to happen when the Matrix ABI version changes is *one* of
the following:

* all binary repository managers (CRAN, RSPM, GitHub Actions folks,
etc etc etc) rebuild their binary versions of the package
* all downstream/LinkingTo: package maintainers issue a 'bump'
release (ideally, _identical_ to the previous release except for version
number to prevent any possibility of breakage) of their package, to
trigger rebuilding by all of the aforementioned binary repositories

The first is not going to happen, so the second is what we will try
to do going forward.

Hope that helps.

Handling the fallout from the Matrix change has been a bigger hassle
than usual: see


https://github.com/lme4/lme4/issues/763
https://github.com/lme4/lme4/issues/760
https://github.com/lme4/lme4/issues/759
https://github.com/lme4/lme4/issues/758
https://github.com/lme4/lme4/issues/757
https://github.com/lme4/lme4/issues/756
https://github.com/glmmTMB/glmmTMB/issues/965
https://stackoverflow.com/questions/77653712/glmmtmb-giving-error-realted-to-log-likelihood
https://stackoverflow.com/questions/77533611/error-in-glmmtmb-after-matrix-update-negative-log-likelihood-is-nan-at-starting/77534375#77534375
https://stackoverflow.com/questions/77466641/error-in-fittmbtmbstruc-negative-log-likelihood-is-nan-at-starting-parameter/77504843#77504843

et cetera ...

On 2023-12-19 1:52 p.m., James Thorson wrote:
> Hi all,
>
> I'm finally distributing packages using TMB via CRAN (e.g., phylosem

> <https://james-thorson-noaa.github.io/phylosem/>), and appreciate all

> the guidance that's searchable online!
>
> However, I already encounter an issue where users have an old version of
> Matrix (e.g., 1.5.4), then install phylosem which installs a recent TMB
> version (e.g., 1.9.10) and the conflict between Matrix and TMB causes an
> inscrutable user error (in this instance, a "Error in nlminb(start =
> startpar, objective = fn, gradient = gr, control = nlminb.control,
> NA/NaN gradient evaluation" message).
>
> Is there any standard advice to check or enforce an update to Matrix for
> packages using TMB?
>
> I can imagine three options but I'm not knowledgeable about the
> potential tradeoffs:
>

> 1. phylosem (or other package) edits DESCRIPTION to require "Matrix

> (>= 1.6.3)" ... but this presumably requires updating phylosem
> DESCRIPTION periodically.

> 2. phylosem (or other package) includes an .onAttach() that checks

> Matrix and throws an error if it isn't sufficiently updated ... but
> this probably doesn't work on GitHub CI Actions, and might be
> annoying to users

> 3. TMB updates its DESCRIPTION ... on github master branch it lists

> "Matrix (>= 1.0-12), which seems out of date
>
> I don't see options 1-2 in glmmTMB so I'm curious to hear suggestions
> and whether/how others are dealing with the topic?
>
> Jim
>

> --
> To post to this group, send email to us...@tmb-project.org. Before
> posting, please check the wiki and issuetracker at
> https://github.com/kaskr/adcomp/ <https://github.com/kaskr/adcomp/>.
> Please try to create a simple repeatable example to go with your
> question (e.g issues 154, 134, 51). Use the issuetracker to report bugs.
> ---
> You received this message because you are subscribed to the Google
> Groups "TMB Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to tmb-users+...@googlegroups.com
> <mailto:tmb-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tmb-users/CAMDbYvb%2BbgQ24P2UAT%2BFevi-gHuMj%2BnRVQitMveg%3D1zRSde4iQ%40mail.gmail.com <https://groups.google.com/d/msgid/tmb-users/CAMDbYvb%2BbgQ24P2UAT%2BFevi-gHuMj%2BnRVQitMveg%3D1zRSde4iQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
Dr. Benjamin Bolker
Professor, Mathematics & Statistics and Biology, McMaster University
Director, School of Computational Science and Engineering
(Acting) Graduate chair, Mathematics & Statistics
> E-mail is sent at my convenience; I don't expect replies outside of
working hours.

[James Thorson]

Yikes!  I'm empathize while seeing your name in the responses to many of those threads!

So to recap:  it sounds like you suggest:

1. adding the code-block below to Makevars
2. adding a warning to .onLoad by including and calling e.g. `checkDepPackageVersion(dep="TMB", this="phylosem")`

Is that correct?  

A few more questions:

• Why doesn't glmmTMB in .onLoad() also include `checkDepPackageVersion(dep="Matrix", this="TMB")`, as noted is possible in `?checkDepPackageVersion`?
• Could people instead just copy the `TMB:::checkMatrixPackageVersion` to do something similar (or even ask Kasper to make that an exported function)?

Makevars code block

pkgver:
"$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "cat(as.character(packageVersion('TMB')), '\n', sep="", file='../inst/TMB-version')"
"$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "cat(as.character(packageVersion('Matrix')), '\n', sep="", file='../inst/Matrix-version')"

To post to this group, send email to us...@tmb-project.org. Before posting, please check the wiki and issuetracker at https://github.com/kaskr/adcomp/. Please try to create a simple repeatable example to go with your question (e.g issues 154, 134, 51). Use the issuetracker to report bugs.

---
You received this message because you are subscribed to the Google Groups "TMB Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tmb-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tmb-users/126d5698-e36d-424b-94cb-6a18fe0f3998%40gmail.com.

[Ben Bolker]

Yes, I think so.

  * I think we decided to let TMB check the Matrix package itself if it wanted

  * This package-checking is a little bit more flexible than TMB's version (but feel free use that function instead)

  Again, it would be better to check ABI than package version, if/when  that becomes available for each package, to avoid false positives/unnecessary warnings.

[Andrea Havron]

Hi Jim,

Thanks for flagging this issue. I couldn't figure out why my clustTMB package started failing github actions even though everything was passing locally. I applied the recommended changes to the Makevars file and added the version files to my inst folder, but github actions are still loading an older version of Matrix (see here). Have you been able to successfully load version 1.6.4 during github actions? 

Thanks and Happy New Year everyone!

Andrea

To view this discussion on the web visit https://groups.google.com/d/msgid/tmb-users/2c88d8d1-44de-40fd-956a-a74b7e51d404n%40googlegroups.com.