Change going from version 3.30.21 to 3.30.22 and persisting through 3.30.24

[Lisa Ailloud - NOAA Federal]

Hi, 

I ran my old gag grouper assessment model (built using SS version 3.30.16) using version 3.30.24 and noticed a small offset in the early part of the estimated time series. The offset specifically appears when I go from version 3.30.21 to 3.30.22. I looked through the change log but I'm not 100% sure what could be causing the issue.  I thought it might be the updated ADMB compiler but from the release notes it looks like that happened between versions .20 and .21 not .21  and .22, unless I'm mistaken. Any insight into what might have caused the change would be appreciated. (I can't post the SS files on this forum but would be happy to share via email).

When going from version 3.30.21 to 3.30.22 the total_LogL decreases from 15593.9  (0.0942532 is_final_gradient) to 15593.3 (0.0162682 is_final_gradient) and parameter estimates (growth, initF, recdevs, SR_regime...) shift very slightly. 

(I know it's an insignificant change but I like to be able to explain changes if possible :))

Lisa

[Ian Taylor - NOAA Federal]

Hi Lisa,

I don't recall any specific change that would cause this, but here's a suggestion for the first step I would take to diagnose the issue: I would look at starting the estimation from each model using the .par file values from the other model and running both with and without estimation (e.g. .par from 3.30.22 and 3.30.21 executable first with "-stopph 0 -nohess" and then with estimation. This can answer a few questions:

1. Are the dynamics the same for the same set of parameters (when run without estimation)?
2. Is it possible that one of the models did not find the MLE and using the .par from the other version allows it to achieve a better likelihood (which may also match the other version)?

I don't recall whether the .par files will be interchangeable between those versions, but at most you might need to add or subtract a few lines to make them work (after comparing the two files to see if the number of values differs).

-Ian

--
You received this message because you are subscribed to the Google Groups "SS3 - Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ss3-forum+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ss3-forum/946ef5b5-00d7-4be1-a348-671d0406fd37n%40googlegroups.com.

[Richard Methot - NOAA Federal]

Lisa,

I also looked at the change log and see nothing obvious.  I suggest you do exactly as Ian recommends to diagnose possible causes.

Rick

[Lisa Ailloud - NOAA Federal]

Thanks for the advice! It looks like the NLL calculation is just slightly different between the two versions such that the optimal solution from .21 is different from .22

When I run the model with the .22 par file but the .21 exe and nohess the outputs of the model run match the .22 outputs by the likelihood it actually slightly higher so when I run it with hessian it goes back to its original solution :

15594.1 ( .22 par file but the .21 exe no hess  )

15593.9 ( .21 par file with .21 exe with hessian )

15593.3  (.22 par file but the .22 exe with hessian)

If I look at the breakdown at the likelihood by data type (comparing .21 and .21 with .22 par file nohess) there doesn't seem to be a single culprit, some are slightly higher and some slightly lower.. 

[Ian Taylor - NOAA Federal]

Hi Lisa,

Thanks for doing these explorations and sorry that they don't point to any obvious source of the issue. I found an old email thread with folks working on Pacific Sardine where they also found minor changes converting to 3.30.22 where we never could point exactly to the source. At the time we didn't follow your approach of narrowing down to a single version change, so it may have been due to changes prior to 3.30.21. 

Do you think it would be adequate to say something like "A small offset in the early part of the estimated time series was due to changes in Stock Synthesis that could not be pinpointed to a specific change in the software, even after consulting with the SS3 development team"? If not, you could send me the model files and I could try to dig deeper, but I can't promise how quickly I could find an answer or if I will truly be able to do so.

-Ian

To view this discussion visit https://groups.google.com/d/msgid/ss3-forum/a3372c3e-7353-4a0c-84a9-b65a38f1bf05n%40googlegroups.com.

[Kiva Oken - NOAA Federal]

Hi all,

The Init_equ_lambda looks like it is a much larger magnitude than the other non-zero values and would also make sense that it impacts the early trajectory. Could it be that?

-Kiva

Kiva Oken, PhD (she/her)
Research Mathematical Statistician, Northwest Fisheries Science Center
NOAA Fisheries | U.S. Department of Commerce
(564) 225-2297
kiva...@noaa.gov
www.fisheries.noaa.gov

I live and work on the traditional lands of the Coast Salish peoples.

To view this discussion visit https://groups.google.com/d/msgid/ss3-forum/CAOUH9r4DZ5C-HnAkhe%2BMVhpYi9v5o%3D9j0AHerKQoW0E%3DT686NQ%40mail.gmail.com.

[Richard Methot - NOAA Federal]

Good sleuthing Kiva.  I will follow that thread this afternoon.

Rick

Richard  D. Methot Jr.

Stock Assessment Research Scientist (ST)

Northwest Fisheries Science Center

NOAA Fisheries | U.S. Department of Commerce

Office: (425) 666-9893

Mobile: (301) 830-2454

www.fisheries.noaa.gov 

You received this message because you are subscribed to a topic in the Google Groups "SS3 - Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ss3-forum/YO5WzflBAMU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ss3-forum+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ss3-forum/CACsG4K5%2BxYYECMgB70UEMU__UQh%3D30-6Z96X657yo7uptG_7hw%40mail.gmail.com.

[Lisa Ailloud - NOAA Federal]

Rick pointed out that it is likely due to this bug fix related to the se for init_equ catch

Thanks all!

Lisa