Bruker CCD data?

[Matt Ginder-Vogel]

Hi,

I was talking with Sam Webb today and he thought you might be willing
to modify you're program to work with bruker data. Would you?

I can send you some data files.

Thanks
Matt

---------------------------------------------
Matt Ginder - Vogel
Post Doctoral Associate
Center for Critical Zone Research
Dept. of Plant and Soil Sciences
University of Delaware
mat...@udel.edu
---------------------------------------------

[joshu...@gmail.com]

Hi Matt. I am always happy to hear new interest in the program. I am
kind of busy right now writing the software manual and adding support
to the program for caking in 2theta and making it so that pixel masks
apply when calibrating. But I would be happy to work on this feature
when I have some free time. Basically, I don't have any bruker data so
if you want my program to read the data in, you or somebody else will
have to give me some actual data in the bruker format to play with.

If you join the discussion group, you can upload a file using the
Google Groups interface. Go to the files tab on the right. Otherwise,
if you create a Google Code account (which can be tied to any email
address or you have gmail you already have one) you can go the Area
Diffraction Machine's Google Code website and upload a file through
the issue tracker. Otherwise, you can just email the file to me (if it
isn't too big) or put it on an FTP.

Also, if you have any information on the bruker file format that would
help me code up the feature, I would be happy to hear about it.
Otherwise, I will go do the digging myself.

Josh

[joshu...@gmail.com]

I added an issue to the issue tracker about this:
http://code.google.com/p/areadiffractionmachine/issues/detail?id=28

Hopefully this won't be too hard to add in.

Josh

[joshu...@gmail.com]

Ok, I have been playing around with the data that you sent me, and I
can mostly get the program to read in the file. I have uploaded a beta
of version 2 of the program (with several other changes) that can
mostly read in the Bruker data you sent me. (http://code.google.com/p/
areadiffractionmachine/downloads/detail?name=Area%20Diffraction
%20Machine%20v2_beta_r194.zip&can=2&q=#makechanges). When I get really
convinced that my code is doing the right thing reading these files
in, I will move the feature down into the more stable version 1 of the
program.

Matt, is there any way you can open my program with a bunch of your
data and see if it causes any weird errors?

I have a couple of concerns about the way the code is now reading data
in. If you or anybody you know knows anything about these bruker
files, I have a couple of questions. First, it looks like Bruker files
can contain 8 bit and 32 bit Bruker data. I have no idea if my program
will properly read this data in because I don't have any data to test.
Do you have any 8 or 32 bit data I can test the program with? Also,
because of the way my program stores the data, any 32 bit data which
is above 2^31-1 would be clipped to 2^31-1. Hopefully, this won't be a
problem.

When I examine the files you sent me, there are 512=32*16 bytes at the
end of the file that I can't figure out what to do with. They are not
part of the regular image. Apperently there is a way to store at the
end of a Bruker file overloaded pixels and each overloaded pixel takes
up 16 bytes. So it sounds like there are 32 overloaded pixels at the
end of the file. But the number of overloaded pixels is supposed to be
set in the header with the "NOVERFL:" value and in both of the files
the value is set to 0. So I am not really sure what is going on. It
might be that the program that wrote the file improperly set the
header value NOVERFL. Right now, the program just doesn't try to read
in any overloaded data. But If you have any data where the header
value NOVERFL is not 0 or any other ideas about what the data at the
end really is, I am all ears.

Finally, I can find the detector to sample distance, the wavelength,
and the x and y center of the image in the header. But I can't find
the pixel length and pixel height of the detector. This is the width
and height of a pixel on the detector (in microns). If you know where
in the header these values are (or at least what these values actually
are for the detector that took the data), that would help me.

Finally, the files you sent me have a .002 extension but do all Bruker
files have a .002 extension. Right now, I have the default extension
of bruker data .bruker but I know this is not right. Do you know what
the default extension should be?

Josh

[aurelien.gourrier]

Hi all,

Bruker file extension is usually .gfrm whatever that means...

Cheers,

Auré

[joshu...@gmail.com]

Ah thanks for the tip Auré. I have the extension properly set in the
latest beta of version 2 of the program.

Josh

On Jun 9, 4:26 am, "aurelien.gourrier" <aurelien.gourr...@yahoo.fr>
wrote:

[alexey.alexeyev]

Hi, Josh.
You doing the great work! I am deeply interested in such software.
I agree with Aurelien, Bruker file extension is usually .gfrm
and .sfrm as well.
If you would like to play with this type of Bruker’s files, I have
downloaded some.

Best regards,
Alexey

[joshu...@gmail.com]

Hi Auré and Alexey. Thank you both for helping me figure out how
bruker data works. I very much appreciate your help. After reading
through the article that Auré sent me, I figured out what I didn't
understand about Matt's data. It turns out that there really were no
overloaded pixels. The reason I know this is because none of the
pixels have an intensity of 65535 (since this is 16-bit data, all
overloaded pixels are stored as 65535). So the extra 512 bytes at the
end of the file are actually for this 'trailer' feature which is
mentioned in the documentation and in the header of the file. I don't
have to worry about that so I am pretty sure I understand Matt's file.

I have been playing around with the files that Alexey uploaded to the
discussion group, and it confuses me a little bit. The first thing
that confuses me is that some of the header data has multiple values.
For example:
* NOVERFL:572 198289
0
* CENTER :506.119995 529.239990 506.119995
529.239990
* DISTANC:5.000000
5.000000
* WAVELEN:0.710730 0.709300 0.713590
0.632290
Do you guys have any idea what the multiple numbers signify. With the
NOVERFL, I am pretty sure that the second number is the number of
overflow values, but I have no idea what the first and third numbers
are. I have no idea why the center and distance are simply repeated
and I don't get why there are 4 wavelengths. Should I just pick the
first, average, or what? I am not really sure.

Also, the file stores 8 bit data so all of the pixels with intensity
equal to 255 need to be replaced with their overloaded value read from
the bottom part of the file. I ran a count though Alexey's file and
there are indeed 198289 of these pixels. So I now believe that NOVERFL
properly reads the number of overloaded pixels. But following the data
array is only 397168 bytes. This is 590 bytes more then having 2 bytes
per overloaded pixel. According to Auré's description of Bruker data,
"The overflow table is stored as ASCII values. Each overflow entry is
16 characters, comprising a 9-character intensity and a 7-character
pixel # offset. The table is padded to a multiple of 512 bytes." Now,
the file doesn't have nearly enough data for this to be true. Alexey,
are you sure that you have uploaded to the website the whole file. Is
it really only 1.4MB on your computer? Furthermore, none of the values
look like ASCII when I open up the file. The file just look like
gibberish. And The number of overloaded pixels are not "padded to a
multiple of 512 bytes" as the documentation describes.

I will keep playing around, I am kind of at a loss. Do either of you
have a suggestion about how to read in these overload pixels?

Thanks for all the help,

Josh

On Jun 10, 2:59 am, "alexey.alexeyev" <alexey.alexe...@gmail.com>
wrote:

[Joshua Lande]

Alternately, if you guys could send me (by attaching to a reply email
or uploading to the discussion group) any other bruker data, that
might help me out what is going on with the files.

Josh

[joshu...@gmail.com]

Hi Alexey and Auré and Matt.

Auré sent me a Bruker data file today which was absolutely beautiful
and exactly to the specification that he posted. And it had all of the
overloaded pixels in the right place. I was really excited because now
I know that there isn't something major I was missing. I was able to
code up the program to properly read in this data now with the
overflow values. I uploaded the new version of the program as
v2_beta_r232. I am still pretty sure that the file that Alexey gave me
doesn't have the overloaded pixels written in it as it should. And so
when you try to load in his file, it will print a warning that it is
not going to read in the overloaded pixels because the format of the
file is wrong. But it will read in the rest of the data and print
warning messages intelegently. I think that my implementation is doing
the right thing.

Finally, if the program finds multiple values in the DISTANCE and
WAVELENGTH header filed, it will use the average value. When it finds
multiple values in the CENTER field, or the NOVERFL field, it will
always take the first number. I hope this is the right thing to do.

If you guys can try reading in some more of your Bruker data with the
new version of the program and report back with success, I will then
migrate the Bruker code down to version 1 of the program (so you guys
don't have to use a possibly buggy beta version of the program.)

Take care and thanks again for all the help,

Josh

On Jun 12, 7:24 pm, Joshua Lande <joshuala...@gmail.com> wrote:
> Alternately, if you guys could send me (by attaching to a reply email  
> or uploading to the discussion group) any other bruker data, that  
> might help me out what is going on with the files.
>
> Josh
>