KDE model

[Andreas Hennius]

Hej
For the last month I  have several interesting articals where  people have been  using KDE-models when calibrating their  radiocarbon dates.
I would like  to try it myself  but can't  find the  function in the  Oxcal program, and  even if I  did I  would  probably need some help about how to do the modelling

Can  someone please give  me a  short  introduction

Kind regards

Andreas hennius

[Christopher Ramsey]

I have updated the manual with some references and basic information.  The paper with the details is at:

http://dx.doi.org/10.1017/rdc.2017.108

or if you cannot access that I attach a preprint.

Best wishes

Christopher

> --
> You received this message because you are subscribed to the Google Groups "OxCal" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to oxcal+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ray Kidd]

Hello Christopher,

Thank you for posting the KDE paper, most instructive.

I've been trying to follow the plot so to speak.  Can you clarify the following two models, I am trying to emulate figure 7 of the paper with the example in the help file.  Why do I get a warning with the first model?  Something to do with the Sequence and Boundaries?

Is figure 7 from the online program?

Best wishes

Ray Kidd


Sequence                                                                                       
        Warning! Model not supported in this context - KDE_Model   
 Plot()
 {
  Sequence()
  {
   Boundary("Start 1");
   KDE_Model()
   {
    KDE_Plot("Plot1", )
    {
    };
    Sum("Sum1")
    {
    };
    R_Date("A",3350,25);
    R_Date("B",3310,25);
    R_Date("C",3320,25);
    R_Date("D",3200,25);
    R_Date("E",3340,25);
    R_Date("F",3260,25);
    R_Date("G",3210,25);
    R_Date("H",3180,25);
    R_Date("I",3350,25);
    R_Date("J",3210,25);
    R_Date("K",3070,25);
    R_Date("L",3200,25);
   };
   Boundary("End 1");
  };
 };


 Plot()
 {
  KDE_Model()
  {
   KDE_Plot("Plot1", )
   {
   };
   Sum("Sum1")
   {
   };
   R_Date("A",3350,25);
   R_Date("B",3310,25);
   R_Date("C",3320,25);
   R_Date("D",3200,25);
   R_Date("E",3340,25);
   R_Date("F",3260,25);
   R_Date("G",3210,25);
   R_Date("H",3180,25);
   R_Date("I",3350,25);
   R_Date("J",3210,25);
   R_Date("K",3070,25);
   R_Date("L",3200,25);
  };
 };

 

 

And from Figure 7:

 

 

[Christopher Ramsey]

Ray

The reason for the error message is that you should not mix KDE_Model with other Bayesian models because it will have confounding effects.  This would be similar to applying a phase model to the same data twice.

So you either want just the KDE_Plot which is an alternative to Sum within another Bayesian model:

 Plot()
 {
  Sequence()
  {
   Boundary("Start 1"){color="green";};
   Phase()
   {
    KDE_Plot("Plot1")

    {
    };
    Sum("Sum1")
    {
    };
    R_Date("A",3350,25);
    R_Date("B",3310,25);
    R_Date("C",3320,25);
    R_Date("D",3200,25);
    R_Date("E",3340,25);
    R_Date("F",3260,25);
    R_Date("G",3210,25);
    R_Date("H",3180,25);
    R_Date("I",3350,25);
    R_Date("J",3210,25);
    R_Date("K",3070,25);
    R_Date("L",3200,25);
   };

   Boundary("End 1"){color="red";};
  };
 };

Or a KDE_Model on its own in place of the uniform phase model as in:

 Plot()
 {
  KDE_Model("Model1")

  {
   R_Date("A",3350,25);
   R_Date("B",3310,25);
   R_Date("C",3320,25);
   R_Date("D",3200,25);
   R_Date("E",3340,25);
   R_Date("F",3260,25);
   R_Date("G",3210,25);
   R_Date("H",3180,25);
   R_Date("I",3350,25);
   R_Date("J",3210,25);
   R_Date("K",3070,25);
   R_Date("L",3200,25);
  };

 };

I attach the two plots that you get from the different methods.  The models are statistically different but the outputs are actually fairly similar.  In the KDE model you can also see the Sum for comparison.

Another new feature that people may wish to try for this and other types of plot is the new plotting function [View > Plot stacks].  The way this works is you select what you want to plot as you would for [View > Plot selected] but each distribution will be superimposed on the previous one until you also tick [Page break].  This way you can build up a more complex plot.  You probably want to turn off [Format > Plot > Title] because these will overwrite.  You might also wish to use color.  Using the first model above, I have selected the first boundary, the KDE plot and then the last boundary (with a Page break) and then finally the Sum.  You then get the third attached plot which shows the start boundary, KDE plot and end boundary in the first line and the Sum plot in the second.

Best wishes

Christopher

> And from Figure 7:

[Ray Kidd]

Hello Christopher,

Most helpful, many thanks.  The workings of the View > Plot Stacks is particularly useful.

In the pdf of the KDE paper, I think there is a typo in the figure 3 text:

"Panel g shows the e ffect
of applying the KDE Model model which uses the KDE distribution as a factor in the likelihood (see
text). Panel g shows a Kernel Density plot of the original calendar dates chosen from the range
AD100-AD500: ideally this is the distribution that the other estimates should reproduce."

I think the first mention of panel g should read 'panel f'?

best wishes

Ray

[Christopher Ramsey]

Ray

Yes - thanks that is right - I'm afraid the same typo has made its way into the printed version. I hope it is clear from the sequence.

Best wishes

Christopher

[Ray Kidd]

Hello Christopher,

Sorry to bother.

If I wanted to export the KDE Model or KDE Plot MCMC data as a csv file, is there a correct method of achieving this?

I can go to table view and click column 2 then highlight and paste the data from there into Excel, subsequently saving as a csv.  Otherwise the save function gives me a .prior file.

Best wishes

Ray

[Christopher Ramsey]

Yes - that is right. The .prior file is just a text file which can be read into Excel if you want too. Alternatively you can use [File > Export...] and save all distributions in the model into one csv file.

Christopher

[Ray Kidd]

Hi Christopher,

Good, many thanks,

Best

Ray

[Andreas Hennius]

hej
Thank you very much  for  your  kind  support,

I will really try to figure this  out, but perhaps  further questions will show up !

Thanks

andreas

[Andreas Hennius]

The  KDE_plot  and KDE_model have  finally  showed  up  on my Oxcal online.

I put in my data and the  analysis  seem to  run, Unfortunatly it crashes in the  end every time.

How  do I  get  out the  curves from the  analysis?

Kind regards Andreas Hennius

 

Den torsdag 1 mars 2018 kl. 18:43:59 UTC+1 skrev Andreas Hennius:

[Christopher Ramsey]

The output files can be quite large in come cases. I would try going back after the run has finished and opening the output file. The distributions can be exported in the same was as for all other distributions (using raw view) or [File > Export...]. There is not currently an easy way to export the ensembles and uncertainties but these can be found as arrays in the .js file. I am working on tools to export these more easily.

Best wishes

Christopher

[Andreas Hennius]

Thanks
But the  diagrams/graphs (with   curves comparable  to the  ones in   in your  article)   do I make them in  Oxcal or in some  other  program?

Kind regards
andreas 

[Christopher Ramsey]

All of the diagrams from my paper were made within OxCal.

Christopher

[Anna Bloxam]

Dear Christopher,

I have followed the thread above but cannot identify a way of exporting the full output of KDE_Model in the latest version of OxCal - is this currently possible?

If there is not an easy export solution, you mention above that the data can be found in the .js file - how do I locate this?

Ideally I would be after a .csv file so I can replot the results against those of other analyses in R.

Thanks & best wishes,

Anna

[Christopher Ramsey]

Anna

Thank you for your email. You can extract this data at the moment only from the .js file. If you look in the .js file you will find terms like:

.kde_ensembles=[];

and then

.kde_ensembles.push([{probNorm:0.000184679,start:-698013,resolution:15.3282,prob:[...]);

the prob array gives the probability histogram, start the date (in astronomical years) of the first term and resolution the spacing.

there is a similar array give for

.kde_mean=

which is the mean value. I have already explained that the main distribution is available from the raw view - and also will be there if you do [File > Export...] which outputs all the prior and posterior dates in one file.

Best wishes

Christopher

[Léa Feyfant]

Dear Christopher,

I am bringing back this conversation to life as I have been trying to use KDE_plot and KDE_model functions. I have first tried to reproduce the plots you posted here in 2018, but I only could arrive to the result below (see attached).

Please could you explain how you obtained these plots ?

Best wishes,

Léa

[Léa Feyfant]

Dear Christopher,

After keeping trying I manage to partially answer my own questions, but I still have some problems.

As Ray, I am trying to reproduce the plots you posted in these conversations (similar to figure 3 and 7 in your 2017 paper, "untitled-14, 15 & 17" attached) using the example from the OxCal Manual.

First, I entered the following model in the program : 

    KDE_Model() 

 { 

 R_Date("A",3350,25); 

 R_Date("B",3310,25); 

 R_Date("C",3320,25); 

 R_Date("D",3200,25); 

 R_Date("E",3340,25); 

 R_Date("F",3260,25); 

 R_Date("G",3210,25); 

 R_Date("H",3180,25); 

 R_Date("I",3350,25); 

 R_Date("J",3210,25); 

 R_Date("K",3070,25); 

 R_Date("L",3200,25); 

 };

And I could only obtain the curves I posted in my previous message (attached here again). I couldn't find how to represent the KDE model with the calibrated and modelled ages (black and gray crosses on your figure), as well as the calibration curve as in Figure 7 of your paper (and the figure "untitled-14&15", attached below, that you previously posted).

Then, I tried the model you posted above to reproduce "untitled-17" (attached):

Plot()
 {
  Sequence()
  {
   Boundary("Start 1"){color="green";};
   Phase()
   {
    KDE_Plot("Plot1")

    {
    };
    Sum("Sum1")
    {
    };
    R_Date("A",3350,25);
    R_Date("B",3310,25);
    R_Date("C",3320,25);
    R_Date("D",3200,25);
    R_Date("E",3340,25);
    R_Date("F",3260,25);
    R_Date("G",3210,25);
    R_Date("H",3180,25);
    R_Date("I",3350,25);
    R_Date("J",3210,25);
    R_Date("K",3070,25);
    R_Date("L",3200,25);
   };

   Boundary("End 1"){color="red";};
  };
 };

I did find how to use the View>>Plot Stacks but apparently not properly. When I tick "Page break" for the Sum, it is the last boundary which appear on the second line. I tried different combinations but never manage to have the Sum alone on the second line.

I have tried many options of plot, view, style and read the entire manual, but I can't find the solution. I am obviously missing something but cannot find what it is.

Please may you help me find the solution?

Many thanks and best wishes,

Léa

[Christopher Ramsey]

Dear Léa

Thanks for your message.  To get the crosses, turn on "Median" in the [Format > Show] menu.

For the stacks you need to select the items that you want in the plot in the order that you want them.  Tick the new page box for the last one on each line.  It should looks something like the attached.

Best wishes

Christopher

To view this discussion on the web visit https://groups.google.com/d/msgid/oxcal/b20f5fe0-c405-4c2c-9253-8c4ebb916784n%40googlegroups.com.
<Untitled-17.pdf><Untitled-14.pdf><Untitled-15.pdf><TestKDEmodel_manual_CurvePlot.jpg><TestKDEmodel_manual_MultiplePlot.jpg>

[Léa Feyfant]

Dear Christopher,

Many thanks for your answer. It was so simple and yet sometimes little things can keep you stuck for hours!

Best wishes,

Léa