MCMC when treating Fs as parameters (Fmethod=2 or Fmethod=4 and annual Fs estimated for some fleets)

[Shanae Allen]

Hello SS Community,

I’m having difficulty using MCMC with an SS model that treats Fs as parameters (i.e., Fmethod = 2 in SS version 3.30.15 or Fmethod=4 with annual Fs estimated for some fleets in SS version 3.30.22.1). In this case, SS model estimates are well outside of the MCMC posteriors distributions and the logL associated with each MCMC iteration is much higher than the SS model. However, when I use the hybrid F option (Fmethod=3 and landings are fit exactly) the MCMC distributions are in general alignment with the SS model estimates. Is this possibly an issue with MCMC or could it be implying that a model with estimated Fs is unidentifiable?

Thank you,
Shanae Allen

Research Scientist
Marine Fisheries Stock Assessment | Fish and Wildlife Research Institute
Florida Fish and Wildlife Conservation Commission
100 8th Avenue SE | St. Petersburg, FL 33701
814.659.2892 (Cell)
Shanae...@MyFWC.com

[Ian Taylor - NOAA Federal]

Hi Shanae,
Thanks for posting here on the SS3 Forum which has greater visibility than the issues list where you originally posted the problem: https://github.com/nmfs-ost/ss3-source-code/issues/642.

In order to investigate the impact of uncertainty in catch inputs, I would try re-running the models in MLE and MCMC modes with SE = 0.05 for all catch values rather than having higher SEs for the recreational fleets to see if that brings the results of the two methods closer together. Even when we know that catch inputs are uncertain, you may not get plausible results when the input SEs for the catch are high when using F method 2 because the model can just use the extra flexibility in catch to fit other data sources better even when we would not expect those other data types to be informative about catch.
-Ian

--
You received this message because you are subscribed to the Google Groups "SS3 - Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ss3-forum+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ss3-forum/37786958-9693-4b8a-8d0d-e545a99c2a66n%40googlegroups.com.

[Shanae Allen]

Thank you, Ian, for that suggestion! I'm running it now, but the MCMC takes about 2 days to complete. Just to make sure I'm understanding your comment - if the MCMC modes are a closer match to the MLEs using a configuration with low SEs on the landings (0.05 as you recommended), this would suggest that the model with high SEs on landings and Fmethod=2 has too much flexibility? In other words, the landings data (particularly the recreational landings) are not informative enough? 

Thank you again for your help and suggestions!

Shanae

[Kelli Johnson]

If MCMC is a staple in your workflow, I suggest looking into using adnuts, which provides advanced sampling for ADMB models. We use in with the Pacific Hake model and were able to decrease our run times to ~5 hours, though hake is a pretty simple model so yours might take more than five hours but will certainly be less than two days.
Best,

Kelli

[Richard Methot - NOAA Federal]

When F is estimated using hybrid, the fit to the catch will be nearly exact, so the fit to everything else is conditional on a vector of Fs that match the catch.  When F is as a parameter, then the model has more freedom to use Fs that are not so tightly constrained in order to balance the fit to the catch and the fit to everything else.  So it is not surprising that the total logL goes up when doing Fs as parameters because the total now includes a sizable component due to lack of fit to the catch.

I recommend:

1.  look at the fit to each individual component and see what fits better when using Fs as parameters vs the hybrid exact Fs.  You should be able to start by looking at this with the MLE run rather than waiting for the MCMC run.

2.  if something fits much better, then the model configuration has tension between fitting catch closely and fitting that other component.  Are there other aspects of the model fit that indicate patterns in residuals?

3.  Per Ian's suggestion, if the catch se is small, then the parameter run should have a result closer to the hybrid run because the Fs will be closer to the Fs that produce an exact fit.

4.  The hake model that Kelli mentioned uses hybrid F, which may influence run times in MCMC.

On Wednesday, November 13, 2024 at 9:45:13 AM UTC-8 Shanae Allen wrote:

[Richard Methot - NOAA Federal]

A corollary of what I wrote regarding hybrid vs parameter is that hybrid should be preferred when catch se is low because when you use hybrid it will match the retained catch near exactly, so the model is acting as if catch se is low.  This is especially important consideration when there is a similar se on a fleet's retained catch and discarded catch. In that situation, parameter F allows the model to balance the fit to retained and discarded catch (and all other data).

The beauty of F method 4 is that it allows the hybrid vs parameter decision to be fleet-specific.  F method 4 is not a new F method, it is just a way to flexibly use Fmethod 2 (parameters) and Fmethod 3 (hybrid), as was discussed in our May webinar.
Rick