Re: [dadi-user] Seeking Help with Model Selection Instability in dadi

[Ryan Gutenkunst]

Hello Fred,

Sorry for the slow reply. I've been on vacation.

I’m not deeply familiar with the Portik pipeline; my group didn’t write it, and we don’t use it.

The most likely issues that 100 optimizations is not enough for your model and data to reliably find the try maximum likelihood parameter values for each model. Local optima are indeed a major problem. You might try using the dadi-cli framework we’ve developed, which has some additional heuristics to aid in optimization.

Best,

Ryan

On Jul 6, 2025, at 7:00 PM, fred jesse <jessef...@gmail.com> wrote:

Hi,

I am writing to seek your expertise regarding an issue I've encountered while using dadi for demographic inference. The data and workflow  from GitHub  dportik/dadi_pipeline: An accessible and flexible tool for fitting demographic models with dadi using custom or published models (available here), conducting goodness of fit tests, and plotting.   I've performed a model selection stability test using the built-in example data and observed concerning inconsistencies in the results.

### Experimental Design:
1. For each candidate model, I ran 100 optimization replicates
2. Selected the top 3 replicates based on log-likelihood
3. Repeated this entire process 8 independent times

### Key Observations:
- The best-fitting model varied across the 8 independent runs
- Even for the same model, parameter estimates showed significant variation
- AIC values of the best models ranged from 955 to 1208 (ΔAIC > 250)

### Specific Examples:
- Run 1: anc_sym_mig_size (AIC=1092)
- Run 2: asym_mig_size (AIC=1097)
- Run 5: sec_contact_sym_mig_size (AIC=955)
- Run 7: sym_mig_size (AIC=1043)

### My Concerns:
1. How can we trust model selection results when the "best" model changes across independent runs?
2. What strategies do you recommend for obtaining stable parameter estimates?
3. Are there known issues with local optima in high-dimensional parameter spaces?

I would greatly appreciate any insights or suggestions ！

Thank you for your time !

Sincerely,
Fred

--
You received this message because you are subscribed to the Google Groups "dadi-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dadi-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dadi-user/6f558af6-48ca-42d6-9842-6c6312fec4e2n%40googlegroups.com.
<dadi_Run_2D_Set.py><optimized_params.txt>