Using spectrometer to identify rocks in the field

[kent.r...@gmail.com]

Hey all,

First, see this imgur album (was too big to attach the pics on here) http://imgur.com/a/2Qqm8 - it will make the text a bit less confusing!

I am attempting to use the spectrometer to identify rocks (and possibly individual minerals) in the field. I have created a new housing for the spectrometer for use in the field along with a protective "skirt" to block out light. On the inside of this "skirt" are mounted LEDs providing the light for the spectrometer. When using this kit I can get a nice spectra from light bulbs (to calibrate) or from white paper. But when I put in test rocks of differing colors/shapes I get only a very faint spectra. Not nearly enough to use to assist with identifying the material. Is this just a limitation of trying to do reflectance spectroscopy with this kit or am I missing something (in my design or elsewhere) on why I am getting poor detection of spectra?

Thanks,

Kent

[Eduardo Montoya Rossi]

Hey all,

 

did you try to take let´s say a hundred pictures and summ all pixle by pixel? You could gain a sharp spectrum and a factor of ten in signal / noise ratio. The summ picture shoud be of 16 or 32 bits, otherways you will get a white imagen due saturation at a value of 255.

 

Regards,

 

Eduardo.

--
Post to this group at plots-sp...@googlegroups.com
 
Public Lab mailing lists (http://publiclab.org/lists) are great for discussion, but to get attribution, open source your work, and make it easy for others to find and cite your contributions, please publish your work at http://publiclab.org
---
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[kent.r...@gmail.com]

Is there a tutorial somewhere on how to do this?

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrometry+unsub...@googlegroups.com.

[Jeffrey Warren]

Are you using the Desktop kit, in a conduit box? If you're not live-scanning, you could try doing a longer exposure (say, with an SLR or something). Smartphone cameras are also usually more sensitive than the webcam-based specs, so one of those could make things easier. But the easiest thing may just be using a *lot* more light -- esp. if the rocks are quite dark (this sounds dumb i guess :-P)

Ah - i just looked at the photos. Great housing design! I love it because it standardizes the light, distance, etc. So can you add more light? Can you share a link to the spectra you've taken?

If you are interested in open sourcing this design, it'd make a *great* research note! Please consider doing so!

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.

[kent.r...@gmail.com]

The key with the project is that it cannot be much larger than what is shown in the pictures. Minor modifications can be made but it essentially needs to be something that i can place on a surface and save the spectra for analysis at a later time. No extensive messing around of the sample surface or constant adjustment of lighting. (Supposed to be dummied down to "point and shoot" so can be done by someone with minimal knowledge of the system). 

I am really new to spectroscopy so I am getting confused at the spectra produced by different light sources. Currently, the housing holds a strip of LEDs to the light source. I have been reading now that I will need some halogen bulbs to produce a much wider range of spectra for identification. Would switching the lighting source to halogen bulbs allow me to detect a better spectra for identification of material? 

I'd love to open source it but unfortunately this is a work project with work funding and the legal hoops to jump through may prove to be too much. 

Kent

On Wednesday, May 21, 2014 11:01:06 AM UTC-5, Jeffrey Warren wrote:

Are you using the Desktop kit, in a conduit box? If you're not live-scanning, you could try doing a longer exposure (say, with an SLR or something). Smartphone cameras are also usually more sensitive than the webcam-based specs, so one of those could make things easier. But the easiest thing may just be using a *lot* more light -- esp. if the rocks are quite dark (this sounds dumb i guess :-P)

Ah - i just looked at the photos. Great housing design! I love it because it standardizes the light, distance, etc. So can you add more light? Can you share a link to the spectra you've taken?

If you are interested in open sourcing this design, it'd make a *great* research note! Please consider doing so!

Jeff

On Wed, May 21, 2014 at 11:26 AM, <kent.r...@gmail.com> wrote:

Is there a tutorial somewhere on how to do this?

On Wednesday, May 21, 2014 10:09:29 AM UTC-5, eduharmonros wrote:

Hey all,

 

did you try to take let´s say a hundred pictures and summ all pixle by pixel? You could gain a sharp spectrum and a factor of ten in signal / noise ratio. The summ picture shoud be of 16 or 32 bits, otherways you will get a white imagen due saturation at a value of 255.

 

Regards,

 

Eduardo.

On Wed, May 21, 2014 at 9:25 AM, <kent.r...@gmail.com> wrote:

Hey all,

First, see this imgur album (was too big to attach the pics on here) http://imgur.com/a/2Qqm8 - it will make the text a bit less confusing!

I am attempting to use the spectrometer to identify rocks (and possibly individual minerals) in the field. I have created a new housing for the spectrometer for use in the field along with a protective "skirt" to block out light. On the inside of this "skirt" are mounted LEDs providing the light for the spectrometer. When using this kit I can get a nice spectra from light bulbs (to calibrate) or from white paper. But when I put in test rocks of differing colors/shapes I get only a very faint spectra. Not nearly enough to use to assist with identifying the material. Is this just a limitation of trying to do reflectance spectroscopy with this kit or am I missing something (in my design or elsewhere) on why I am getting poor detection of spectra?

Thanks,

Kent

--
Post to this group at plots-sp...@googlegroups.com

 
Public Lab mailing lists (http://publiclab.org/lists) are great for discussion, but to get attribution, open source your work, and make it easy for others to find and cite your contributions, please publish your work at http://publiclab.org
---
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrometry+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[kent.r...@gmail.com]

Woops, here is a link of a spectra of black/brownish soil from out a construction site nearby. 

http://spectralworkbench.org/sets/show/1217

On Wednesday, May 21, 2014 11:01:06 AM UTC-5, Jeffrey Warren wrote:

Are you using the Desktop kit, in a conduit box? If you're not live-scanning, you could try doing a longer exposure (say, with an SLR or something). Smartphone cameras are also usually more sensitive than the webcam-based specs, so one of those could make things easier. But the easiest thing may just be using a *lot* more light -- esp. if the rocks are quite dark (this sounds dumb i guess :-P)

Ah - i just looked at the photos. Great housing design! I love it because it standardizes the light, distance, etc. So can you add more light? Can you share a link to the spectra you've taken?

If you are interested in open sourcing this design, it'd make a *great* research note! Please consider doing so!

Jeff

On Wed, May 21, 2014 at 11:26 AM, <kent.r...@gmail.com> wrote:

Is there a tutorial somewhere on how to do this?

On Wednesday, May 21, 2014 10:09:29 AM UTC-5, eduharmonros wrote:

Hey all,

 

did you try to take let´s say a hundred pictures and summ all pixle by pixel? You could gain a sharp spectrum and a factor of ten in signal / noise ratio. The summ picture shoud be of 16 or 32 bits, otherways you will get a white imagen due saturation at a value of 255.

 

Regards,

 

Eduardo.

On Wed, May 21, 2014 at 9:25 AM, <kent.r...@gmail.com> wrote:

Hey all,

First, see this imgur album (was too big to attach the pics on here) http://imgur.com/a/2Qqm8 - it will make the text a bit less confusing!

I am attempting to use the spectrometer to identify rocks (and possibly individual minerals) in the field. I have created a new housing for the spectrometer for use in the field along with a protective "skirt" to block out light. On the inside of this "skirt" are mounted LEDs providing the light for the spectrometer. When using this kit I can get a nice spectra from light bulbs (to calibrate) or from white paper. But when I put in test rocks of differing colors/shapes I get only a very faint spectra. Not nearly enough to use to assist with identifying the material. Is this just a limitation of trying to do reflectance spectroscopy with this kit or am I missing something (in my design or elsewhere) on why I am getting poor detection of spectra?

Thanks,

Kent

--
Post to this group at plots-sp...@googlegroups.com

 
Public Lab mailing lists (http://publiclab.org/lists) are great for discussion, but to get attribution, open source your work, and make it easy for others to find and cite your contributions, please publish your work at http://publiclab.org
---
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrometry+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jeffrey Warren]

Hi, Kent - it looks like you're getting a lot of stray reflected infrared light inside your device -- perhaps the material is not absorbent in the infrared range? Some plastics suffer from this issue. Do you have a liner or could you paint the inside matte black?

So, if your hardware work builds on the PL spectrometer design, you are actually required to open source your own work, since we use a "sharealike" license, the CERN Open Hardware License. (http://publiclab.org/licenses) I'm happy to help you figure out what that means for your work, and to support you in explaining that requirement to the institution you work for, if that's helpful!

Best,

Jeff

 

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.

[kent.r...@gmail.com]

I've lined the tube with a few layers of black construction paper but the overall orange tint of the plastic gets through anyway. The parts are 3d printed so they are PLA plastic. Would something like PVC piping be a better absorber or would painting the tube work well enough? I may also be able to construct the tube out of other material (any suggestions on the type that may work better?). 

Ah, then I shall speak to my boss. Hopefully he says I can opensource it instead of telling me I can't use the kit anymore :(

Kent

[Jeffrey Warren]

I think spraying it with matte black paint could do the job -- give it a try! I'm not sure if black PVC would work, but it'd probably still be reflective. 

Thanks!

Best,

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.

[jimwelsh]

See:  http://instrumentation.tamu.edu/samples.html and click on "Total Reflectance Plots".

[dst...@gmail.com]

Nice mechanical design and I'm impressed with the 3D printing. However there are some things to remember and think about.

1 - The slit, as shown, is too large. You want a very narrow, sharp-edge slit 10-30 mils wide. The purpose is both for diffraction reasons and second, there are no optics; so the slit is acting like a collumator between the light source and the diffraction grating. This also brings up more points. Try a couple of razor blades carefully positioned edge-to-edge covering the hole. There's also no need to have a long slit -- just makes a wider spectral band (not the nm direction, the other way -- no added resolution).

2 - The slit needs to be positioned at the focus of the camera lens. I can't tell if it is, but my guess is that the slit is too far from the lens to be focused. You'd need to check this.

3 - For reasons of having collumated light, the light source (in this case, reflected light) should not arrive at the slit from a wide angle. That said, the reverse is partially true, that using a slit as the optics (as all the PLab designs do) theoretically means that they(the 'slit optics' are only "looking" at a narrow view-angle of light. You could, perhaps, add a black shield between your light source and sample, and the slit (moving the slit back away from that shield a ways). Then cut a small aperture in the shield, in-line with the slit and camera.

4 - Yes, getting the ambient light level down requires non-reflective black surfaces -- maybe felt, paper, powder-coat flat black paint. Also, by moving the camera/slit assembly further away, you could add a second baffle between the source and camera unit which narrows the angle at which light can enter the slit. The camera/slit chamber needs to be completely non-reflective black inside. The second baffle with aperture hole (like a slit, but say 1/4-in wide just to reduce reflections) might help. The big killer with making measurements is noise and, with spectrometers, ambient light is all noise.

5 - Adding successive images will NOT work to enhance your signal. Signals are all a mix (signal + noise) so you would be simply adding both so the SNR remains the same. Doing an average of 3 or 5 images can sometimes help a little but that is only when the signal looks different from the noise and is consistent in all the images being averaged. This is because noise is mostly Gaussian and so does not correlate (via averaging) as well as a signal. Averaging can improve the SNR a little but it also will remove some 'detail' from the signal due to the 'smoothing' effect of averaging -- which may or may not matter to you.

6 - Past reducing the overall noise (as noted above) the best way is to increase the SNR is to increase the signal. In this case, that is the reflected light from the sample. i.e. so increase the light from the source.

7 - Unless you have specific reasons for looking at LED wavelengths (and I don't know why this would help when looking at rocks) I'd suggest using a broadband light with enough energy to get a bright spectrum -- i.e. lots of signal. Since earth-satellites which do spectral analysis of geology all operate at very low IR wavelengths (much lower than the PLab webcams) I assume you are simply looking at the "color" of the rocks. This suggests that a broadband light source would be best and you would then use an ideal neutral reflector (no color) to establish a baseline (so you know that reflected spectra of the light source) and then measure and calculate the difference for any rock samples. The easiest broadband source to obtain (I believe) are the Solux halogen bulbs which have a known, smooth response -- better than what you get at the hardware stores. Just remember, brighter light will get hot so your housing will have to handle this - but the Solux are 12V which is easier for field use. (Avoid direct reflections -- i.e. us indirect reflection rather then direct -- similar to avoiding bad angles when taking a photo through a window.)

Good luck with your project.

Cheers,
Dave

[kent.r...@gmail.com]

Lots of useful information from you guys! Thank you very much. 

In response to Dave,

1. Just measured the slit - 3mm wide and 20mm long. This seems to be within your recommended range, maybe the slit just appears bigger in the pictures?

2. I have not calculated the focal length of the lense but rather used the dimensions of the housing provided by PL as a guide for the size of the housing I made. Both housings are of the same length but I may have my camera mounted a bit further back than the kit. Will move it forward and test.

3. Does the cylindrical front end of the housing kit of the PL set work as an aperature? I did not realize this. I will design up a part that will work as aperature for the slit. 

4. I will reprint the slit and housing in black material and then either paint flat black or line with felt. 

7. The LED lighting choice was due to my ignorance of how different light sources would affect the reflected spectra. I'm going to replace the LEDs with a halogen light source. 

Most importantly, if I implement all of these changes will the spectra produced be sufficient enough to really identify with any certainty different types of rocks? It will probably take me 3 days or so to implement all the changes but I dont want to do all of them just to find out that the CCD sensor is not able to detect enough for me to really tell the difference between a basalt and a granite (just a random example). 

Thanks a bunch!

Kent 

[Jeffrey Warren]

Maybe i'm just optimistic, but I think that if the lighting source is very consistent, characterizing rocks could work -- you can try Sreyanth's matching code to confirm it: http://publiclab.org/notes/Sreyanth/09-14-2013/finding-closest-match-spectra-from-the-database-gsoc-final-post

> 3. Does the cylindrical front end of the housing kit of the PL set work as an aperature? I did not realize this. I will design up a part that will work as aperature for the slit. 

I think dave is talking about a separate aperature. I don't think the pipe fitting plays a role.

Jeff

--
Post to this group at plots-sp...@googlegroups.com
 
Public Lab mailing lists (http://publiclab.org/lists) are great for discussion, but to get attribution, open source your work, and make it easy for others to find and cite your contributions, please publish your work at http://publiclab.org
---
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.

[Mathew Lippincott]

Kent, you might try throwing one of our optically printed slits in, they're a little easier than fooling with razor blades and give a crisp .09mm slit. The acetate also helps prevent internal reflections.

On Wed, May 21, 2014 at 11:26 AM, <kent.r...@gmail.com> wrote:

--
Post to this group at plots-sp...@googlegroups.com
 
Public Lab mailing lists (http://publiclab.org/lists) are great for discussion, but to get attribution, open source your work, and make it easy for others to find and cite your contributions, please publish your work at http://publiclab.org
---
You received this message because you are subscribed to the Google Groups "plots-spectrometry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to plots-spectrome...@googlegroups.com.

[jimwelsh]

3mm is big enough to drive a truck through.  Well, almost.  But for spectroscopy purposes, it might as well be.  When I google "convert 30 mils to mm", I get an answer of 0.762 millimeters.  Dave's recommendation is basically between 0.25 and 0.75 mm.

[kent.r...@gmail.com]

Ah, I thought he meant mils = millimeters. That explains a lot! 

Thinking I'll get a few of those printed slips, print out a new cap with a larger slit in it and then mount the printed slit on the cap so it's protected and mounted to something that can stabilize it. Also will add something that will work as an aperature. 

Back to the drawing board! 

[Eduardo Montoya Rossi]

Dear All,

Please find attached a small draft of tips that could hopefully work
as a some sort of tutorial. I just found a mistake in TIP6, please
delete the part "then the option Color".

In few hours more I will send a txt file with a Macro listing and some images.

Until soon,

Eduardo.

On 5/21/14, kent.r...@gmail.com <kent.r...@gmail.com> wrote:
> Is there a tutorial somewhere on how to do this?
>
> On Wednesday, May 21, 2014 10:09:29 AM UTC-5, eduharmonros wrote:
>>
>> Hey all,
>>
>> did you try to take let´s say a hundred pictures and summ all pixle by
>> pixel? You could gain a sharp spectrum and a factor of ten in signal /
>> noise ratio. The summ picture shoud be of 16 or 32 bits, otherways you
>> will
>> get a white imagen due saturation at a value of 255.
>>
>> Regards,
>>
>> Eduardo.
>>
>>
>> On Wed, May 21, 2014 at 9:25 AM, <kent.r...@gmail.com

>> <javascript:>>wrote:

>>
>>> Hey all,
>>>
>>> First, see this imgur album (was too big to attach the pics on here)
>>> http://imgur.com/a/2Qqm8 - it will make the text a bit less confusing!
>>>
>>> I am attempting to use the spectrometer to identify rocks (and possibly
>>> individual minerals) in the field. I have created a new housing for the
>>> spectrometer for use in the field along with a protective "skirt" to
>>> block
>>> out light. On the inside of this "skirt" are mounted LEDs providing the
>>> light for the spectrometer. When using this kit I can get a nice spectra
>>>
>>> from light bulbs (to calibrate) or from white paper. But when I put in
>>> test
>>> rocks of differing colors/shapes I get only a very faint spectra. Not
>>> nearly enough to use to assist with identifying the material. Is this
>>> just
>>> a limitation of trying to do reflectance spectroscopy with this kit or am
>>> I
>>> missing something (in my design or elsewhere) on why I am getting poor
>>> detection of spectra?
>>>
>>> Thanks,
>>> Kent
>>>
>>> --

>>> Post to this group at plots-sp...@googlegroups.com <javascript:>

>>>
>>> Public Lab mailing lists (http://publiclab.org/lists) are great for
>>> discussion, but to get attribution, open source your work, and make it
>>> easy
>>> for others to find and cite your contributions, please publish your work
>>> at
>>> http://publiclab.org
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>>
>>> "plots-spectrometry" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>

>>> email to plots-spectrome...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

> --
> Post to this group at plots-sp...@googlegroups.com
>
> Public Lab mailing lists (http://publiclab.org/lists) are great for
> discussion, but to get attribution, open source your work, and make it easy
> for others to find and cite your contributions, please publish your work at
> http://publiclab.org
> ---
> You received this message because you are subscribed to the Google Groups
> "plots-spectrometry" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to plots-spectrome...@googlegroups.com.

[dst...@gmail.com]

Kent,

1 - Yes, mils = thousandths of an inch where 1mm = 40 mils. The largest slit which still gave some reasonable resolution was in the neighborhood of 50 mils. Sure, a film slit could work. The idea with the slit is that a) you want the 2 edges of the slit to be parallel, b) the two edges to be sharp (not fuzzy) and c) there to be zero light leakage anywhere around the slit. You could keep the mechanical 'slot' you have and just cover it with an optical slit. Also, remember that you probably cannot construct an actual optical slit using 3D printing because the slit refines parallel 'edges' -- thin edges with 'zero' depth -- not a long channel of the same width.

2 - The concept of the camera focus is that since the slit is providing the optics, having the camera focused on the slit is a rough analogy to having the camera focused at a star used as a point source where the light really is collimated. Granted the webcam lens has a rather broad DOF, but it still helps the resolution to get it focused on the slit -- not de-focused there.

3 - Sorry for no diagrams, but here's the light sequence: Halogen source : scattered reflected light from the subject : a little of that light is in the direction of the slit : a baffle with an aperture blocks all spurious reflections : remaining light directed toward the slit : that light still spreads a little : add a baffle with another aperture : remaining light is now relatively columated and most spurious reflections blocked : final direct light from slit hits diffraction grating : then immediately the camera lens.

4 - I'm just suggesting two 'baffles' -- the first roughly defines the area of the sample where reflected light will be observed. The second is made closer to the size of the slit (but large enough to make building it easy with no need for difficult alignment) and between slit and camera just to discard spurious reflections so the camera image background field will be a free of light noise as possible -- i.e. to get the max SNR.

5 - Your final question -- Will it work? -- is the $64,000 question. So far, only a few have experimented with reflected spectra techniques and most, including mine, have met with limited success. There are 3 primary issues: a) getting enough broadband light to get a high SNR, b) determining if the visible spectrum of the PLab device (~300nm to ~1000nm) is sufficient and c) characterizing the measurements.

5a - Yes it should be possible to get sufficient light from halogen to illuminate the spectrometer -- though it might take several bulbs to get full-range signal (i.e. near saturation). It will depend largely on the reflectivity of the samples. Dark colors might be the hardest.

5b - Is 300-1000nm sufficient? As I mentioned, geological surveys from space tend to look at wavelengths out past 700-1000nm so for that kind of data, there will be limited results from the visual spectrum. Probably just look at 500nm+ and look only for the general 'slope' of the spectral curve -- AFTER amplitude calibration. I think what you will 'see' with the visible-light PLab spectra is largely the 'color' composition of what the rock 'looks' like as opposed to it's geological makeup such as % iron content -- unless that iron is providing a clear visual effect of orange or brown color.

5c - You would need to a) get a repeatable reference spectra of the system from a neutral reflectance surface and b) subtract all sample measurements from that curve.

The other question is what will you detect within the visible light range. i.e. could you actually tell the difference in visible reflection between sandstone and basalt -- and if so -- between different types of sandstone or different types of basalt.

Perhaps an experiment might be good. Get the slit-camera assembly working and having a dark, low-noise background field. Then point it at a neutral (i.e. 12% grey reflectance) and expose that surface with your halogen source to see if you can get enough light and read a spectrum which looks something like a halogen lamp. Then substitute some rock surface (shielded by a simple black paper baffle with an aperture) and look for a difference. Ultimately your system (computer tools) will need to do a difference between spectra.

Cheers,
Dave

[Eduardo Montoya Rossi]

About the point five: Adding successive images will NOT work to

enhance your signal. Signals
> are all a mix (signal + noise) so you would be simply adding both so the
> SNR remains the same. Doing an average of 3 or 5 images can sometimes help
> a little but that is only when the signal looks different from the noise
> and is consistent in all the images being averaged. This is because noise
> is mostly Gaussian and so does not correlate (via averaging) as well as a
> signal. Averaging can improve the SNR a little but it also will remove some
>
> 'detail' from the signal due to the 'smoothing' effect of averaging --
> which may or may not matter to you.

A short Reflection: Let´s suppose an arbitrary signal of intensity I
=10 together with a Gaussian, non correlated noise of intensity N =
10. This is a somewhat unfavourable case.

Let´s consider a number of replicates n = 100. Then, the total signal
will be Itotal = n * I = 1000 and the total noise will be Ntotal = n *
N / (n^0.5) = N * (n^0.5) = 100.

Thus, for this case the expected improvement in the signal / noise ratio is 10.

Until soon,

Eduardo.

> --
> Post to this group at plots-sp...@googlegroups.com
>
> Public Lab mailing lists (http://publiclab.org/lists) are great for
> discussion, but to get attribution, open source your work, and make it easy
> for others to find and cite your contributions, please publish your work at
> http://publiclab.org
> ---
> You received this message because you are subscribed to the Google Groups
> "plots-spectrometry" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to plots-spectrome...@googlegroups.com.

[dst...@gmail.com]

Perhaps I'm not following your thinking, but this is what your example suggests to me: That a signal of magnitude 10 is mixed with background Gaussian noise of magnitude 10 (which means the signal is completely buried in the noise) and that with only summing 100 separate instances of the signal (S+N) (signal completely buried in the noise) (and we don't have any information ahead of time about the signal -- it's buried in the noise) that you can get a resulting (S+N) SNR of 10 -- and thus out pops the signal out of the noise where you can now see it? It seems like maybe you are considering S and N separately? -- but they are actually mixed and nothing can be assumed known about the signal S within (S+N) and S cannot be extracted from (S+N) without some form of prior knowledge about the nature of just S. This is why enhancing SNR from the collected data is difficult.

Maybe I missed a step in your concept but I've never been able to extract signals from noise by adding multiple instances of (S+N) signal data over time -- especially where the signal was buried completely in the noise. Generally, 6dB SNR is the minimum detectable signal level for collecting any useful data. Filtering or correlation functions are generally the means for improving SNR in noisy signals.

Now it is possible to extract a signal from the noise by means of correlation -- this is how spread-spectrum signals gain the advantage and how NASA is able to extract data from rovers on Mars with 5W transmitters. But, it is not simple addition of signals (S+N) because the two (S and N) are not separate -- their data is combined and no prior knowledge is available about the signal itself w/o the noise. It is because they know the exact signature of the source signal.

I did note that your program reference showed 'mean' and 'averaging' functions which can help reduce the noise when there is some correlation in the signal (signal only) energy between all sample data (S+N) signals. So, maybe I missed something in your description.

Cheers,
Dave

[Eduardo Montoya Rossi]

Hi Dave, hi All,

I would like to defend using the attached pictures, with a part of the spectrum of a CFL. Few pieces of Canson paper were used to dim the light of the CFL. The pictures(n = 100)  were taken in RAW formatat 16-bit grayscale by using the software V4L2 Test Bench in Ubuntu Linux. A cheap webcam with its original lens changed by a the X20 eye piece of a toy telescope was used. The new lens array was not carefully aligned, but the field of view takes probably from about 500 nm about 620 nm. The cam was operated at minimun exposure time and maximun gain in order to allow recording well the noise.

I will not send the stacks because each one takes more than 6 Mb of memory and the message would be too big.

Now:

1) spectrum 1 is a stack of the 100 pictures of the spectrum (not shown here).

2) bg1 is a stack of the 100 pictures of the background (bg, CFL off, not shown here).

3) spectrum1-1 and bg1-1 are the first images of each stack.

4) SUM_spectrum1 and SUM_bg1 are the summ images of the spectrum and bg.

5) SUM spectrum1 minus SUM bg1 and spectrum1-1 minus bg1-1 are the difference images (bg substracted from the spectrum).

6) See also the profile plots of the difference spectra. All the picture (not just a line) were selected to plot the profile.

7) The peaks approximately located at pixels 136, 335, 349 and 556 correspond to the well known peaks at approximately 545 nm, 777 nm, 779 nm and 611 nm.

8) bg subtraction was performed to discard the systematic like noise pattern due uneven behavior of the sensor pixels.

Please tell me what do you think about these pictures.

Now I want to jump and say that a faint signal, eventually lost in a terrible bg will have a systematic contribution to the summation, which should increase linearly with the number of replications of the measure, but the contributions of the noise will distribute symmetrically around its mean value, following a Gaussian curves. This is the basis for the discrimination of the noise, which will increase only as the square root of the number of repetitions of the experiment.

Until soon,

Eduardo.

[Eduardo Montoya Rossi]

Hi Dave,

Please find attached another example, taken at very weak illumintation of the spectrometer slit by the CFL. This time each satck (not shown here because each one takes about 270 Mb) had comprised by n = 10000 pictures. The the increase of the signal / noise ratio by effect of the summation of 10000 pictures is clear.

Cheers,

Eduardo.

[dst...@gmail.com]

Eduardo,

I tried three different photo apps to view the .tif photo images but unfortunately the apps cannot detect photographic data. Seems like there was some issue in the .tif format.  The images, as they appear in google, are blank.

I could see the two graphs in the google post and so am guessing (in the second post) that they represent the before and after images. Just a guess, but it looks like the results are from  photo averaging. If so, yes, averaging does help with Gaussian noise -- especially when there is a large, well- correlated signal like you would find with a CFL spectrum. Just adding image data won't do that and, as you probably known, when adding image data the app doing the calcaultaion must keep the data as type (double) because the sums get very large -- and then the data must be divided and converted to integer before saving as an image file. Maybe it was this last step that had a glitch. Don't know, just guessing.

Cheers,
Dave

[Eduardo Montoya Rossi]

Hi Dave,

The TIF images are 32 bits ones and probably require the use of ImageJ or equivalent software, which is free license and can be downloaded from http://imagej.nih.gov/ij/ Surely you will not appreciate those images with google or MS software.

About summation and averaging, perhaps summation is just the average multiplied by a factor of n? where n is the number of replicates taken for calculating the average. In that case the summation should have just the same information as the average, wityh the advantage of keeping precision and avoiding loss of information beacuse of rounding of decimal figures.

Cheers,

Eduardo.

 

[dst...@gmail.com]

Eduardo,

Ok I can read the images via matlab and just looked at the first frame spectral image and the 'sum' image so I see now what you were describing. Yes, accumulating the noise values average the noise because there is little correlation between frames. Usually the resulting data is divided to keep the same bit depth as the original - thus averaging. I'd thought PLab's code did average a few frames but I'm not sure; if not, I imagine they will add it. I also average sets of 3 pixel 'lines' with the spectrum for the same basic reason.

Cheers,
Dave

On Wednesday, May 21, 2014 7:16:24 PM UTC-7, eduharmonros wrote:

Hi Dave,

[Jeffrey Warren]

Hi, Eduardo - this thread is getting quite long with a lot of files -- perhaps posting a research note at PublicLab.org would be a good way to document and share your work -- you can upload unlimited files and images, and you'll also be open sourcing your research. 

People can comment/tag/respond to it there -- and it's much easier to find and link to! I don't want this great discussion to be lost in the mailing list archives!

Best,

Jeff

[Yagiz Sutcu]

Hi all,

Signal averaging actually increases the strength of a signal relative to noise that is obscuring it. By averaging a set of replicate measurements, the signal-to-noise ratio, S/N, will be increased, ideally in proportion to the square root of the number of measurements.

But the assumptions are:

    Signal and noise are uncorrelated.

    Signal strength is constant in the replicate measurements.

    Noise is random, with a mean of zero and constant variance in the replicate measurements.

If you know the characteristics of the noise pattern, you can do "filtering" in order to denoise the signal.

Cheers,

Yagiz

[dst...@gmail.com]

Yes averaging can help but it's also a tradeoff: Averaging is essentially applying a low-pass filter; filtering out the higher frequency noise 'hash' and leaving low frequency signals alone. So, 'sharp' spikes in the spectra which are down near or in the noise will be both 'spread' or 'broadened' and attenuated by averaging -- thus obscuring some detail in the signal and adding distortion. It's still a very useful technique but nothing is free.
Cheers,
Dave