Re: [PLOTS] Using wavelength fraction absorption ratios to improve substance ID across platforms

[Jeffrey Warren]

Bumping to spectrometry list! Sounds interesting, Gary, thanks!

On May 20, 2013 10:33 PM, "dobrodude" <bark...@comcast.net> wrote:

I have been thinking about the problem of variability in the spectrum of a substance based on different setups and even within the same setup with respect to sample concentration differences.

One way to reduce the effect of variability might be to quantify the absorption of a range of wavelengths (arbitrarily 300-400nm, 401-500nm etc)  by adding a routine that automatically measures the area under the curve of each range and divide each area by the total.

This would produce a ratio or percentage of the total for each set of wavelengths that should be more consistent across concentrations and different spectrometers provided the same light source was used like halogen.
The ratios or percentages for each substance could be stored in the database along with the actual spectrum itself and searched by matching the closest percentage sets with unknowns. The search data could contain perhaps as few as five values or perhaps more depending on the wavelength ranges chosen and automated.

Hypothetical example:
A spectrum of an unknown  substance is run and the absoprtion percentages might be:
300-400nm = 12% of total absorption
401-500nm = 38%  "
501-600nm = 16%
601-700nm = 18%
701-800nm = 26%

A routine could automatically display all posted spectra of knowns with that percentage set or close to it.

Measuring the absorption ratios of various proteins in blood is a standard technique in clinical laboratories.
Let me know what you think!
GB

--
-- Post to this group at publicla...@googlegroups.com. To unsubscribe, email publiclaborato...@googlegroups.com. Options at https://groups.google.com/d/forum/publiclaboratory?hl=en
---
You received this message because you are subscribed to the Google Groups "The Public Laboratory for Open Technology and Science" group.
To unsubscribe from this group and stop receiving emails from it, send an email to publiclaborato...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jeffrey Warren]

Sorry to keep doing this, but we really want to keep spectrometry discussion on the spectrometry list -- please try to reply all to plots-spectrometry only, thanks!

What's the best way to measure area under the curve? It would seem to be pretty easy to do with the JavaScript API: http://publiclab.org/wiki/spectral-workbench-api

I was thinking of just summing the values of all the data points, and divide by the total # of points (for the desired region) -- that'd be the average, not the area under the curve... but easier since we have binned data, not a smooth curve.

Anyhow i'm not so strong on the math... but i can help implement it in JavaScript as a macro.

Jeff

Jeff

On Tue, May 21, 2013 at 9:58 AM, Gary Barker <bark...@comcast.net> wrote:

How do you measure area under the curve with the online software? I have never seen a demo of it.
GB

On 5/21/2013 7:43 AM, Richard Winkel wrote:

I'll throw in a resounding YES vote!

The question I would like to pose, to be answered by the new logic is:

"Looking at wavelength range from x nm to y nm, and with bins z nm wide, what is the percentage of absorption taking place in each bin?"

This is essentially what I would do in Excel if not available in the online software.

Regards,

Dick Winkel

[Sreyanth]

On Tue, May 21, 2013 at 7:46 PM, Jeffrey Warren <je...@publiclab.org> wrote:

Sorry to keep doing this, but we really want to keep spectrometry discussion on the spectrometry list -- please try to reply all to plots-spectrometry only, thanks!

What's the best way to measure area under the curve? It would seem to be pretty easy to do with the JavaScript API: http://publiclab.org/wiki/spectral-workbench-api

If measuring the area under the curve needs to be too accurate, we can have a python script which can do the integration part. Integral of the curve within the limits gives us the area under the curve. This should work cool even without a smooth curve. But, to keep it simple, we can follow a normal area finding methods, though involves few more additions and multiplications, this can be done easily. If you want, I will write a working example in python, which you can easily port into JS. Let me know if you want to see my solution.

--
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.
To post to this group, send email to plots-spe...@googlegroups.com.
Visit this group at http://groups.google.com/group/plots-spectrometry?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Yours Sincerely

Mora Sreyantha Chary

Computer Engineering '14

National Institute of Technology Karnataka

Surathkal, India 575 025

[Jeffrey Warren]

Yes please -- i'm happy to port Python to JS. 

[Richard Winkel]

Just a quick note about integration from graphs in general.

(Integral of ydx) from lower limit to upper limit, is approximately:

[Summation of y*(delta x)] between lower limit and upper limit.

For more discussion regarding variations on the theme see:

http://en.wikipedia.org/wiki/Numerical_integration

By the way, this is a really useful way to get to some solutions when the elegant mathematical closed form solution does not come readily to hand.

Gulp.  Will try to get to the right distribution list.

Regards,

Dick

[barkergk]

Counting pixils under the curve would substitute for area if that is easier. 

If this could be incorporated as part of spectral workbench for all to use and absorption percentages automatically recorded and stored in the database, it could be used as a powerful spectral matching capability. 

I am new to this site so I am still confused about matching the spectrum of an unknown with knowns already posted. The search seems to be restricted to hunting for substances by name instead of by matching spectra. If I am wrong please set me strait.

In any case, storing absorption percentage sets would seem to be a much more straightforward way to search for at least preliminary matches to at least narrow them down to a handful of spectra.

Sent from my Verizon Wireless 4G LTE Smartphone

[Sreyanth]

Jeff, you can find the code here. : http://pastebin.com/kBkgfgmw

Also, find an attached copy if there is some problem with the link. I tried to make it as simpler and detailed as possible in short time. 

​Sreyanth​

On Tue, May 21, 2013 at 10:19 PM, barkergk <bark...@comcast.net> wrote:

Counting pixils under the curve would substitute for area if that is easier. 

If this could be incorporated as part of spectral workbench for all to use and absorption percentages automatically recorded and stored in the database, it could be used as a powerful spectral matching capability. 

I am new to this site so I am still confused about matching the spectrum of an unknown with knowns already posted. The search seems to be restricted to hunting for substances by name instead of by matching spectra. If I am wrong please set me strait.

Yep, counting pixels could be substituted. We can include this when we add the similar-spectra matching module. Till then we can use the Python script I just wrote.

[Gary Barker]

This is great!
Keep in mind that the selection of 5 spectral ranges on the 100's are just arbitrary demarcations, but maybe a good place to start.
Breaking them into small sections so the sets have more value might give a better resolution, but I don't know.
They don't necessarily have to be of equal bandwidth either and breaking them into general colors might be better.

I don't know what or if published research on these questions exists. If so it would be helpful, if not it could provide an interesting group research project for members.
In any case, as long as the bandwidth sections are always the same for an agreed-upon source light, it should be a useful addition to the project.

Can't wait to see this addition to spectral workbench.
GB

[Jeffrey Warren]

Great, thanks Sreyanth! I'll see if it can be readily ported. 

Gary - actually yes, the goal of a spectral matching is one of our top Google Summer of Code proposals:

http://publiclab.org/wiki/gsoc-ideas

I believe Sreyanth submitted a proposal and we'll know final info on the GSoC slots in a couple days.

[Sreyanth]

On Wed, May 22, 2013 at 7:14 PM, Jeffrey Warren <je...@publiclab.org> wrote:

Great, thanks Sreyanth! I'll see if it can be readily ported. 

Gary - actually yes, the goal of a spectral matching is one of our top Google Summer of Code proposals:

http://publiclab.org/wiki/gsoc-ideas

I believe Sreyanth submitted a proposal and we'll know final info on the GSoC slots in a couple days.

Thanks for your reply Jeff.

Also, regarding the code here, if the x-coordinates are in equal intervals, say 1, then the area can be directly calculated without many loops by the formula: (avg of y-coords of first and last points) + (sum of y-coords of all the remaining points).

Its basically the same code, with x2-x1 always equal to 1 (or some constant h, in which case, the above expression needs to multiplied by h)

[dst...@gmail.com]

Jeff,

There are several additional factors to consider:
1 - What about the difference in noise floor between different system results. Broadband noise (ambient light) can easily 'color' all bands?
2 - How do you handle the data differences between systems that are way under, vs way over, saturation in RGB?
2 - What about the case where there are more narrow-band 'peaks' or 'valleys' in the curve -- arbitrary 50-100nm bands could be poor representation?
3 - When the band is wide, integrating the area under the curve tells you nothing about the characteristics at the "top"
4 - As pointed out, comparison of bands, between systems, w/o any form of amplitude calibration is questionable. Yes, requiring the same "type" of light source helps, but that's still far from a calibration.

IF one is limited to a really simple data coordinate storage, you might consider the following concept:
1 - Measure the average level below 400nm and use that as an average of the noise
2 - Keep the bands narrow -- say 20nm wide -- so that's still only about 20 points
3 - Store the mean intensity within a band -- but use the bands below 400nm as avg noise when plotting the data
4 - Take the derivative (average slope) for the band and save that as well
5 - This gives sets of ( wavelength[band-center], mean amplitude above noise, relative slope )
6 - Might consider saving this data for all 3 (RGB) as that is the root data from which the 'intensity' plot is generated. If there is saturation in RGB, the you could automatically detect that and discard the data as invalid for comparison. So:  DataArray( WL, Amp-Noise, Slope, RGB )
7 - Some consideration should be included for signal averaging (smoothing) to limit really noisy data sources

Cheers,

Dave