Open Source Calibration App

[Lee Drake]

Hi all,

As part of my new position with the Paleoresearch Institute, I wanted to use the freedom to help solve some problems that simply weren't possible while working at Bruker. The big one was calibrations - S1CalProcess was a fantastic app, but on a difficult platform (macros in Excel). Support for this software has been over for some time on Bruker's end, and a long-term solution was needed. 

To that end, I've written an open-source replacement for S1CalProcess. With it, you can create calibrations, modify calibrations, and apply them to new data sets without having to worry about out-of-date software. While it is written in an source language (R), it runs entirely in a web browser. That means it can work on any operating system - even mobile ones like iOS and Android.

The app as it is now has the same functionality as S1CalProcess, but with a few additions.

Changes from S1CalProcess:

• Narrower line definitions (reduces overlap affects)
  
• Plot spectra to check for overlaps & influences
• Custom normalization (can choose normalization for each element separately)
  
• Interactive graphs
  
• Statistical diagnostics (tip of the hat to Arlen Heginbotham)
• New filetype (.quant)

A video for how to use the software can be found here: https://www.youtube.com/watch?v=xm5wiNuQ_7U

(will make a better one later)

What I would like now are for people who would be willing to volunteer as beta testers to find problems with it. It is not 100% ready for everyone yet, but folks on this list serve are welcome to try it out. 

If you would like to try it in its beta phase, please let me know and I'll work with you to set it up on your computer. You can visit the project at https://github.com/leedrake5/CloudCal - instructions for using it are housed there. I'm also attaching a PDF guide for the software to this message. As a warning, it is beta software. It's been through a fair bit of testing already, but I can't guarantee the app and/or your computer won't find creative problems. 

Also a quick note - this app is intended to be free for everyone to use in perpetuity. But, it also represents a whole new platform to work with the XRF - we are working on other apps at PRI to integrate the .quant file standard for core analysis, mapping, and sourcing. In the long term, we'd like to design custom software to handle unique problems. But, this app will remain free & open source to make sure anyone who wants to can build empirical calibrations for XRF, laboratory or portable. 

If you have any questions, please let me know. If you do want to try that app, please let me know as much as you are able to about ease of use and changes you'd like to see. There is currently only one developer (me), but I'd like this to be a convenient solution for everyone. 

Thanks,

---

B. Lee Drake

Department of Anthropology
University of New Mexico
(505) 510.1518
b.lee...@gmail.com

    

[Scott Pike]

Hello Lee,

Thank you for message. I’m very interested in trying your new beta software. I was never that comfortable with S1CalProcess and am glad to try another software. I’ll wait for your marching orders on how best I should access the software.

All the best,

Scott

__________________________

Scott Pike

Associate Professor

Environmental and Earth Sciences Department

Chair, Archaeology Program

Willamette University

900 State Street

Salem, OR  97301

o. 503.370.6587

f.  503.370.6773

sp...@willamette.edu

--
Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<cloudCal.pdf>

[Andy McAlister]

Hi Lee, 

Thanks for putting this software together - it's really impressive and I'm sure a lot of people here will be thankful for it.

I can report that it works just fine in Windows 10 and the graphics make it much more usable than S1CalProcess. The only glitch I've noticed is that the R app running in the background freezes and I need to invoke the dreaded CRT+ALT+DEL routine to close it. 

I'll look more closely at the calibration results over the next few days to see how compares to S1CalProcess. 

One question: is it possible to use Compton normalisation with the csv files generated from Artax? I'm guessing the answer is no because the files contain only net peaks and not entire spectrum data. 

Thanks again, 

Andy

[Lee Drake]

Hi Scott,

Good to hear from you! There are two options - the easy one and the power one. 

Easy: Go to this link: http://54.149.148.150:3838/CloudCal/

Power: Follow the instructions on GitHub: https://github.com/leedrake5/CloudCal

The app should be fine either way, but it is always easier to work with a local copy than a server-based one. What material will you be working with?

Thanks,

---

B. Lee Drake

Department of Anthropology
University of New Mexico
(505) 510.1518
b.lee...@gmail.com

    

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
<cloudCal.pdf>

--
Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

[Glinsman, Lisha]

Lee,

Thank you so much! I am interested (and I'm sure Dylan here at the NGA is as well)

Lisha

---

From: thal...@gmail.com <thal...@gmail.com> on behalf of Lee Drake <b.lee...@gmail.com>
Sent: Tuesday, October 3, 2017 4:02 PM
To: px...@googlegroups.com
Subject: [pxrf] Open Source Calibration App

 

Hi all,

As part of my new position with the Paleoresearch Institute, I wanted to use the freedom to help solve some problems that simply weren't possible while working at Bruker. The big one was calibrations - S1CalProcess was a fantastic app, but on a difficult platform (macros in Excel). Support for this software has been over for some time on Bruker's end, and a long-term solution was needed. 

To that end, I've written an open-source replacement for S1CalProcess. With it, you can create calibrations, modify calibrations, and apply them to new data sets without having to worry about out-of-date software. While it is written in an source language (R), it runs entirely in a web browser. That means it can work on any operating system - even mobile ones like iOS and Android.

The app as it is now has the same functionality as S1CalProcess, but with a few additions.

Changes from S1CalProcess:

• Narrower line definitions (reduces overlap affects)
  
• Plot spectra to check for overlaps & influences
• Custom normalization (can choose normalization for each element separately)
  
• Interactive graphs
  
• Statistical diagnostics (tip of the hat to Arlen Heginbotham)
• New filetype (.quant)

A video for how to use the software can be found here: https://www.youtube.com/watch?v=xm5wiNuQ_7U

Cloud Cal www.youtube.com A new program for building empirical calibrations

(will make a better one later)

What I would like now are for people who would be willing to volunteer as beta testers to find problems with it. It is not 100% ready for everyone yet, but folks on this list serve are welcome to try it out. 

If you would like to try it in its beta phase, please let me know and I'll work with you to set it up on your computer. You can visit the project at https://github.com/leedrake5/CloudCal - instructions for using it are housed there. I'm also attaching a PDF guide for the software to this message. As a warning, it is beta software. It's been through a fair bit of testing already, but I can't guarantee the app and/or your computer won't find creative problems. 

leedrake5/CloudCal github.com CloudCal - Empirical XRF Calibrations

Also a quick note - this app is intended to be free for everyone to use in perpetuity. But, it also represents a whole new platform to work with the XRF - we are working on other apps at PRI to integrate the .quant file standard for core analysis, mapping, and sourcing. In the long term, we'd like to design custom software to handle unique problems. But, this app will remain free & open source to make sure anyone who wants to can build empirical calibrations for XRF, laboratory or portable. 

If you have any questions, please let me know. If you do want to try that app, please let me know as much as you are able to about ease of use and changes you'd like to see. There is currently only one developer (me), but I'd like this to be a convenient solution for everyone. 

Thanks,

---

B. Lee Drake

Department of Anthropology
University of New Mexico
(505) 510.1518
b.lee...@gmail.com

    

--

Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+uns...@googlegroups.com.

[Laura Ware Adlington]

Hi Lee,

The video was great! I would like to try your calibration programme, though I use an Innov-X/Olympus, not a Bruker. It is possible to export the spectra data in a .csv file, so I assume it would translate quite easily. What do you think?

Thanks so much!

Laura

[Scott Pike]

Thanks, Lee.  I’ll try to work with the new software this afternoon once my morning classes are over. For the past several years I have been collecting spectra from clay floor samples from multiple Neolithic structures at the Ness of Brodgar in Scotland (http://www.orkneyjar.com/archaeology/nessofbrodgar/).  I have so much data I’m ready pull out the few remaining strains of hair on my head!  I am using the mud rock calibrations that were set up for my Tracer to quantify the data. Is there any chance you will be at the GSA meeting in Seattle?  I’m presenting a paper where I compare semi-quantitative (normalized counts to the Rh Compton peaks) and quantitative analyses of the same spectra from floors from one of the structures at the site.  I am less interested in being accurate with my counts, and much more interested in being able to compare one archaeological context to another.  

Thanks again for all your support over the years.

Scott

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+uns...@googlegroups.com.

[Lee Drake]

Hi all,

Quick update - good to hear people are able to use the app thus far!

Some specific answers to questions:

How to use the app?

You can try the app pain-free (but with a small chance of a crash) at: http://54.149.148.150:3838/CloudCal/

How to run it more reliably?

You can follow the instructions on the GitHub page for the app: https://github.com/leedrake5/CloudCal. But, in brief:

install.packages("shiny")

shiny::runGitHub("leedrake5/CloudCal")

Can it work on non-Bruker instruments?

It has the possibility of working on everything - even extending beyond XRF. But it is not something I can do now for a variety of reasons. I will look into this in the future, but first step is making sure all Tracers & Artax units are ready to go with this software. 

What materials does it work on?

So far I have heard from folks working on glass & clays. The answer to those is yes. In fact, there is no reason that this cannot work for any material. Because you customize units, you can make it % weight or % volume. Normalization though is key - the more oxygen in the sample (e.g. clays, rocks, and glasses), the more you will want to rely on compton/bremmstrahlung normalization. The more carbon in the sample (plants and organics) ,the more you will want to stick to time normalization. And the more complete the representation of the spectrum - that is to say every major compositional element is present in the spectrum (copper, iron, titanium, etc), the more you will want to normalize to total counts. Keep in mind you can choose any of these options for any elements - sometimes light elements like Mg or Ca are better normalized to time, and sometimes they are better normalized to compton/bremmstrahlung. It is up to you to experiment with it. 

What is the .quant file?

This filetype is an open-source way of holding all the data in a file. If you want to see what it looks like, enter the following code into R:

str(readRDS("location/of/your/file.quant"))

The result will be a lot of information, but here is the basic layout:

$FileType (this defines the kind of csv you upload)
$Units (these are the units you entered into the app - one type for everything) 

$Spectra (this includes all the spectra of the csv files used to make the cal)
$Intensities (these are the intensities of the elements you choose in the 'Counts' tab)
$Values (these are the intensity data you put into the app)

$calList (this is the holder for all the element models defined by you) 

$Element this is all the data for one element

$CalTable (this has all the settings for the element)

$CalType (this is whether or not Lucas-Tooth is used)

$NormType (this is they type of normalization you used)
$Min (this is the minimum normalization energy you chose)
$Max (this is the maximum normalization energy you chose)

$Slope (these are the slope variables you choose if you make a Lucas-Tooth cal)
$Intercept (these are the intercept variables you choose if you make a Lucas-Tooth cal)
$StandardsUsed (this is a list of the standards you used for the regressions) 

$Model (this is the parameters of the model you chose) 

 

I hope this answers some questions - if you need individual help with each Tracer and/or Artax unit you are working with, please let me know - you can email me at either b.lee...@gmail.com or l...@pri.services if you have any specific questions you don't want to publish to the group. 

Thanks,

---

B. Lee Drake

Department of Anthropology
University of New Mexico
(505) 510.1518
b.lee...@gmail.com

    

[Pacea Videos]

Hi Lee,

I was just visiting your xrf.guru website to check whether it was or not a shiny app since we plan to make our own shiny app for something else than XRF. And I then discovered your new application, and this group. I would be glad to beta test your software if still necessary. I work in R a bit, but just a bit, nothing like you... I still have the Spectro XSORT machine but we are going to buy a new one next year. It may be Tracer i5. I can export .csv files that are very simple : channel (or energy) vs. counts. Probably I can thus import them in your shiny app !
I was just improving my calibration on ochres this morning, I could try to use this dataset on your calibration system (some CRMs and some ICP-MS measured samples)
LEt me know if you think this is simple enough to import a .csv fil like that or I have to wait for your app to works for other machines!

[Kristine Martirosyan - Olshansky]

Hi Lee, 

I know I am quite late to the game, but if you are still looking for beta testers, I'd love to try it out. 

I watched the video on obsidian sourcing and am really curious to see how it will work with Southern Caucasian obsidian.  Also, I have over 180 geological samples from Armenia covering most known sources, which I collected and analyzed using the Tracer SD-IV, so if you think these data can enrich the obsidian database you are using for the app, I'll be happy to share the data. 

All the best, 

Kristine 

[Lee Drake]

Hi Kristine,

Absolutely, you can access it here: 

71.205.98.137:3838/CloudCal/

GitHub link to install it locally here: 

https://github.com/leedrake5/CloudCal

Are you at the SAAs? If so, I am the Paleoresearch Booth, across from the main SAA book area. If your not, you can find a video that goes over the instructions here: 

https://youtu.be/xm5wiNuQ_7U

- Lee

--

Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+uns...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--- B. Lee Drake University of New Mexico +1 505.510.1518 b.lee...@gmail.com

[Jacob Thomas]

Hello Lee

I am a bit late to the thread as well, but I have a shiny new Eliot from XGlab, now part of the Bruder family, that I would like to try with your software. It helpfully exports spectra in the artax format. I no longer need to run my Olympic to artax conversion macro for my old xrf! So can I join in on the beta testing as well? I can have access to students as well as traceable standard samples for LA ICP MS as well as ground truth composition for unknowns using the above method or other methods.

Regards

Jacob 

[Kristine Martirosyan - Olshansky]

Thanks Lee! 

I'm excited to give this a try. Alas, I am not at the SAA this year. 

I'll let you know if how it goes. 

Best, 

Kristine 

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--- B. Lee Drake University of New Mexico +1 505.510.1518 b.lee...@gmail.com

--

Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---

You received this message because you are subscribed to a topic in the Google Groups "pXRF for Cultural Heritage" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pxrf/YACRFiUxs3Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best,
Kristine Martirosyan - Olshansky

[Lee Drake]

Hi Jacob,

Yes - you can actually use all three file formats the Elio produces (.spt, .mca, .spx) with the cal app. Because .spx has been added, the cal app now works with Artax units/software output too, both raw spectra and net count following Bayesian deconvolution.  

- Lee

[Aaron Shugar]

Jacob,

I have built some good cals for the artax as well.  With some of the new features Lee is adding in, things are going to be even smoother with building calibrations

Aaron

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--- B. Lee Drake University of New Mexico +1 505.510.1518 b.lee...@gmail.com

--
Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--- B. Lee Drake University of New Mexico +1 505.510.1518 b.lee...@gmail.com

--
Visit the pXRF for Cultural Heritage website at:
https://sites.google.com/site/pxrfgroup/
---
You received this message because you are subscribed to the Google Groups "pXRF for Cultural Heritage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pxrf+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Aaron Shugar
aaron....@gmail.com
shug...@buffalostate.edu

http://flip.it/fCmAh