Fmodel and Model Agreement Indices Comparisons

[Charalampos Paraskeva]

Hello group,

Reading some of the model comparison threads in the group I deduce the following:

- It is possible to compare models using the agreement indices, as long as the sample size is the same or about the same.

- Agreement indices are not a fully objective means of comparisons, hence it is necessary to be careful and apply common sense too.

- Agreement indices essentially compare the employed model structure to the null hypothesis model (non-constraint, non-parameterized version of the model) and normalize the index price to somewhat offset sample size.

- It is useful to compare Fmodel indices, as they change with sample size and the parameters/constraints employed.

I have a set of follow-up questions to the discussions:

- It has been suggested that it is good to compare Fmodel indices, but it has not been clarified as to how the latter are compared or comparable! Do higher Fmodel ratios indicate better or worse models in comparison to the null model? For individual likelihood distributions the closer to 1 or above is better, but for entire models this is not defined.

- If Fmodel is affected by sample size (OxCal Documentation says that these "ratios tend to get further and further from 1 for larger number of observations"), then are there moving limits for the associated likelihood ratio (e.g. for 5 likelihood distributions Fmodel ratio limits are between 0.6-1.0, for 10 likelihood distributions the Fmodel ratio limits move to 0.8-1.0, etc.)?

I am trying to understand how model comparisons should more properly be implemented.

Thank you in advance for any response,

Paraskeva Charalambos

PhD Student in Archaeology

The University of Edinburgh

[MILLARD A.R.]

The initial suggestion of using Fmodel was mine, I think, in this post: https://groups.google.com/d/msg/oxcal/mDVhh8FNnGY/6ugk93lyFrEJ But nobody took up my request for comments, so this is not a fully developed idea, nor has it been tested in a situation where the true Bayes factor is known. I do think it is crucial that the radiocarbon data is *exactly the same* in the two models to be compared so the null model denominator of each Fmodel cancels when making the comparison. A ratio of two Fmodel values can then be interpreted along the same lines as a Bayes factor (the summary in Wikipedia is handy: https://en.wikipedia.org/wiki/Bayes_factor#Interpretation).


Best wishes

Andrew
--
 Dr. Andrew Millard 
e: A.R.M...@durham.ac.uk | t: +44 191 334 1147
 w: http://www.dur.ac.uk/archaeology/staff/?id=160
 Senior Lecturer in Archaeology, Durham University, UK

> --
> You received this message because you are subscribed to the Google
> Groups "OxCal" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to oxcal+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Charalampos Paraskeva]

It was indeed a very useful read, but I am a bit worried about applying K values meant for Bayes factor to a pseudo-Bayes factor. Maybe some more experimental work is needed to define values for OxCal. It was very informative to read that the Bayes factor penalties models with too complicated structures to guard against data overfitting.

Thank you for the information Dr. Millard.