sum() and MCMC_sample

[MILLARD A.R.]

There have been extensive discussions about summed probabilities here and in the literature. My view is that they should not be used as (a) their formal statistical meaning is the probability of the data of one of the events chosen at random, (b) they have no uncertainty attached to them, (c) much of the structure is due to the calibration curve, (d) generally there are too few samples to detect changing frequency of events assuming an underlying Poisson distribution of counts.

As well as Steele's paper it is worth looking at these critical evaluations:

Bishop RR. 2015. Did Late Neolithic farming fail or flourish? A Scottish perspective on the evidence for Late Neolithic arable cultivation in the British Isles. World Archaeology 47:834-855.

Chiverrell RC, Thorndycraft VR, Hoffmann TO. 2011. Cumulative probability functions and their role in evaluating the chronology of geomorphological events during the Holocene. Journal of Quaternary Science 26:76-85.

Contreras D, Meadows J. 2014. Summed radiocarbon calibrations as a population proxy: a critical evaluation using a realistic simulation approach. Journal of Archaeological Science 52:591–608.

An MCMC sample is no more valid than the statistical model that underlies it.


Best wishes

Andrew
--
 Dr. Andrew Millard 
e: A.R.M...@durham.ac.uk | t: +44 191 334 1147
 w: http://www.dur.ac.uk/archaeology/staff/?id=160
 Senior Lecturer in Archaeology, & Associate Director
of the Institute of Medieval & Early Modern Studies,
Durham University, UK


> -----Original Message-----
> From: ox...@googlegroups.com [mailto:ox...@googlegroups.com]
> Sent: 23 November 2015 21:01
> To: ox...@googlegroups.com
> Subject: Re: MCMC_sample
>
> Hi Tom,
>
> I have never used the MCMC Sample command so it will be for others to
> comment.
>
> However I would also appreciate comment on the validity of an MCMC
> Sample of a Summed Probability Distribution, given that from the
> manual :
>
> "The 'sum probabilities for calendar dates' no longer provides
> probability range bars since version 4.1.4: "Ranges no longer reported
> for Sum distributions (as these are misleading)".
>
> Best wishes
>
> Ray
>
> In a message dated 23/11/2015 19:48:51 GMT Standard Time,
> tsdy...@gmail.com writes:
>
>
> Aloha Ray,
>
> Many thanks for your response. I hadn't discovered the Wenninger
> et al. paper, but am familiar with the other two.
>
> I'm interested specifically in learning how archaeologists have
> used the output of OxCal's MCMC_sample command. Is Steele 2010 the
> only published attempt so far? The other papers don't address this
> issue specifically, as far as I can tell.
>
> All the best,
> Tom
>
> On Monday, November 23, 2015 at 7:32:10 AM UTC-10, Ray wrote:
>
>
> Hi Tom,
>
> A response to Steele's 2010 paper is at:
>
> Journal of Archaeological Science 38 (2011) 2116-2122
> "A comment on Steele’s (2010) “radiocarbon dates as data:
> quantitative strategies
> for estimating colonization front speeds and event
> densities”
> Briggs Buchanan a,b,1, Marcus Hamilton c,d,e, Kevan
> Edinborough f, Michael J. O’Brien b, Mark Collard a,b,*,1"
>
> Also maybe of interest:
>
> Journal of Archaeological Science 39 (2012) 578-589
> "The use of summed radiocarbon probability distributions in
> archaeology:
> a review of methods
> Alan N. Williams*"
>
> Also of interest:
>
> Documenta Praehistorica XXXVIII (2011)
> "Concepts of probability
> in radiocarbon analysis
> Bernhard Weninger 1, Kevan Edinborough 2, Lee Clare 1 and
> Olaf Jöris 3"
>
> I like the use of 'cum grano salis' in the latter.
>
> Hope it helps
>
> Ray
>
>
>
> In a message dated 22/11/2015 21:39:17 GMT Standard Time,
> tsdy...@gmail.com <javascript:> writes:
>
>
> Aloha all,
>
> I've only found Steele's 2010 paper in JAS that
> discusses use of MCMC_sample. Are there others?
>
> All the best,
> Tom
>
> @Article{steele10:_radioc,
> pages = {2017--2030},
> volume = {37},
> year = {2010},
> journal = {Journal of Archaeological Science},
> title = {Radiocarbon dates as data: quantitative
> strategies for estimating colonization front speeds and event
> densities},
> author = {James Steele}
> }
>
>
>
>
>
> --
> You received this message because you are subscribed to
> the Google Groups "OxCal" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to oxcal+un...@googlegroups.com
> <javascript:> .
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout> .
>
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "OxCal" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to oxcal+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "OxCal" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to oxcal+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Thomas Dye]

What do you think of the CalPal approach?  It appears to minimize the influence of the calbration curve.

All the best,
Tom

[Rayfo...@aol.com]

Hi Andrew, Tom,

 

As to Andrew's four point view ( a,b,c,d) on why Summed Distributions are somewhat fraught, I say Amen to that. 

 

I was going to suggest a further point (e) All Radiocarbon analysis programs do not use the same methodology, so comparisons are further suspect.

 

However, Tom raised the question of CalPal, which I think is allied to my point (e).  The paper I mentioned earlier :

 

" Documenta Praehistorica XXXVIII (2011) 
          "Concepts of probability  in radiocarbon analysis         

Bernhard Weninger 1, Kevan Edinborough 2, Lee Clare 1 and Olaf Jöris 3"

 

is a good (Light on maths) read that addresses the question in great detail, with comparisons of OxCal, Calpal and CALIB. 

 

Best wishes

 

Ray

[MILLARD A.R.]

> In a message dated 26/11/2015 06:17:45 GMT Standard Time,
> tsdy...@gmail.com writes:
>
> What do you think of the CalPal approach? It appears to minimize
> the influence of the calbration curve.

It's taken me some time to get around to reading up on CalPal, and I've waded through the latest paper [1] which claims to describe the underlying methodology and why all Bayesian modelling of the type done in OxCal is wrong. This paper is light on maths as well and I have to say that I am not confident that I could reproduce the CalPal method from the description. The paper is very hard to follow because they insist on adopting non-standard terminology and using standard terminology in their own idiosyncratic way. For example near the beginning of the paper is an extended discussion of commutative operations but what they are actually interested in is invariance of an operation under a transformation. Once I realised that they were sometimes using 'commutative 'when they meant 'invariant' it started to make a bit more sense. Throughout the paper it is not always clear when they are discussing calibration of individual dates and when they are calibrating ensembles.

The conclusion is that CalPal can reproduce OxCal's results for summed probability distributions, but they prefer CalPal's method of adjusting distributions of individual dates on the C14 scale (I think) and summing on the C14 scale followed by calibration of the resulting distribution. There is an extended discussion of Dirac-this and Dirac-that properties of the events and I think what they are ultimately saying is that calibration, even if there was no uncertainty on the measurement and the curve, gives a distribution of possible dates for the event. I think they are then arguing that this makes calibrated distributions (for summed groups, but I'm not clear if they are claiming this for individual calibrated dates) not interpretable in terms of the distribution of events, but they claim there are fundamental reasons why this is inevitable and the CalPal summed distributions are the best we can hope for.

Before I used CalPal I'd like to see the underlying mathematics as then I could at least verify the process by my own calculations. More worryingly I think that this approach tries to correct summed distributions because it makes the same underling mistake about the meaning of summed distributions that many other papers make. That is to assume that the shape of the summed distribution has meaning in terms of the underlying process that generated the dates. I don't know how this would work, and have not seen any paper that establishes the mathematical basis for the claim. Clearly the broad shape is related to the underlying process, but no one has shown that the process can be recovered from a summed distribution which has no uncertainty attached and, as it is usually normalised, has thrown away all information about sample size.

My poster at the 2009 Radiocarbon Conference showed that the 68% or 95% probability range of the summed distribution does not relate in a simple manner to the start and end of a simple uniform generating process, but a Bayesian phase model does recover the start and dates, and even better informs about the uncertainty of those estimates. One possible interpretation of a summed probability distribution, if it is not normalised is the mean of all possible histograms of distributions of dated events. But it is not easy to interpret the mean of a complex multivariate probability distribution without information on the distribution itself.



[1] Weninger B, Clare L, Jöris O, Jung R, Edinborough K. 2015. Quantum theory of radiocarbon calibration. World Archaeology 47:543-566.

[Erik]

Thanks Andrew, that was an informative post.

The discussions on the forum have helped me a lot with this issue, and the papers mentioned above.

Besides the statistical problems, there are also archaeological problems:

• Intra-site sampling.

• Sample size.

• Taphonomic loss.

They are from Williams 2012, who does a nice job of summarizing problems and ways of dealing with them. He has a more positive outlook on summed probability distributions, but is working with a database of 5000 dates from all over Australia. It would be interesting to see this for England (has someone done this?). I do think they can show general, relative population trends in large blocks (Bamforth and Grund 2012). Would you agree, Andrew? Is there any use at all for these curves?

Erik

Bamforth, D. and Grund, B. 2012. Radiocarbon calibration curves, summed probability distributions, and early Paleoindian population trends in North America. Journal of Archaeological Science 39: 1768–1774

Williams, A., 2012. The use of summed radiocarbon probability distributions in archaeology: a review of methods. Journal of Archaeological Science 39: 578–589.

[MILLARD A.R.]

Erik,

In large blocks, yes they can be informative. But Bamforth and Grund didn't use large blocks although they conclude that would be better. Surovell et al 2009 did that and I think it is probably ok to do so as long as the blocks are much larger than the typical calibrated ranges. There is still a problem with sample numbers though. Williams recommendation of 500 seems to have been fairly widely adopted without considering the timespan or number of blocks they are to be divided between. Williams only had 3000 dates spread over 40000 years. If working with 500 year blocks that represents an average of 38 dates per block. If the counts are poisson distributed than there will be a large percentage uncertainty in many of the blocks; an observed count of 38 has a 95% confidence range of 27 to 52 on the underlying true frequency. So it will be hard to see anything other than the most extreme changes. But I have yet to see a paper that actually considers how few events are actually being observed.

Best wishes

Andrew
--
 Dr. Andrew Millard 
e: A.R.M...@durham.ac.uk | t: +44 191 334 1147
 w: http://www.dur.ac.uk/archaeology/staff/?id=160
 Senior Lecturer in Archaeology, & Associate Director
of the Institute of Medieval & Early Modern Studies,
Durham University, UK

[Christopher Ramsey]

Thanks for looking into that Andrew. I’m also very uncertain exactly what CalPal does mathematically.

On the use of Sum - it is certainly true that if you have enough data sampled from a continuous period, the Sum method does tend to a flat distribution. This implies that it is not the Sum approach per se that causes the extra structure but simply the combination of sparse data with the method.

The sum distribution is a convolution of a discrete distribution with a non-normal age uncertainty. Normally if you had a discrete distribution of events and wanted to reconstruct an underlying process you might use a method like kernel-density estimation. The Sum distribution looks deceptively a bit like this, but the use of the calibrated age uncertainty as a kernel is clearly not what you would chose from a kernel-density estimation point-of-view. It may be that the CalPal program is essentially adding an additional Gaussian kernel onto the distribution to smooth it - but this is not how it is expressed.

There would be ways to incorporate more standard kde methods for this purpose and it would be useful to collect people’s views on the value of this and/or what approaches might be most useful.

Best wishes

Christopher

[Thomas Dye]

+1 A statistically defensible estimate of event density would be a useful tool for chronologists.

All the best,
Tom