Re : [area-diffraction-machine] Re: Re : [area-diffraction-machine] Re: Bruker CCD data?
54 views
Skip to first unread message
Auré Gourrier
Jun 14, 2008, 5:57:58 AM6/14/08
to area-diffrac...@googlegroups.com
Hi Josh,
I'm in Poland right now so will make it short. The source of the doc I sent you is indeed a manual for the Bruker Nanostar.
As for the rest of the questions, I'll do my best to find out more when I return at the ESRF. This is anyhow something that we also have to do, so will let you know asap. The fact that you get several figures for some keys in the Bruker header seems puzzling but I vaguely remember something of the like... have to check this out.
I attach a data file, nothing very fancy, this is just a calibration file for SAXS (Silver Behanate). I'm pretty sure that the data is OK.
Cheers,
Auré
I'm in Poland right now so will make it short. The source of the doc I sent you is indeed a manual for the Bruker Nanostar.
As for the rest of the questions, I'll do my best to find out more when I return at the ESRF. This is anyhow something that we also have to do, so will let you know asap. The fact that you get several figures for some keys in the Bruker header seems puzzling but I vaguely remember something of the like... have to check this out.
I attach a data file, nothing very fancy, this is just a calibration file for SAXS (Silver Behanate). I'm pretty sure that the data is OK.
Cheers,
Auré
----- Message d'origine ----
De : "joshu...@gmail.com" <joshu...@gmail.com>
À : Area Diffraction Machine <area-diffrac...@googlegroups.com>
Envoyé le : Vendredi, 13 Juin 2008, 1h46mn 31s
Objet : [area-diffraction-machine] Re: Re : [area-diffraction-machine] Re: Bruker CCD data?
Ah, just for reference, I found an IDL routine which can read in
Bruker data: (http://cars9.uchicago.edu/software/idl/detectors.html).
It is the read_siemens.pro file and although I have never read IDL
code before, I think that it corroborates this documentation on the
Bruker file format.
Also, I Googled parts of your document to try to find where it came
from, and I found a mostly verbatim copy of this document at
http://www.esc.cam.ac.uk/index.php/component/content/article/60-Facilities/229-saxi-area-detector-frame-format.
It says that it describes the "SAXI Area Detector Frame Format."
On Jun 10, 4:27 am, Auré Gourrier <aurelien.gourr...@yahoo.fr> wrote:
> Hi again,
>
> Some time ago, I copied a document from a manual concerning the bruker file format, I attach it below. In particular, there is a description on the table overflow business, hope this helps...
>
> Cheers,
>
> Auré
>
> -----
>
> Appendix
> A. Area Detector Frame Format
>
> The
> Bruker format addresses issues of data portability, speed of image
> I/O and display, and storage of all the data acquisition conditions
> required to reduce the data. Header and overflow information are
> stored in portable ASCII, rather than binary. Critical data, such as
> setting angles, can thus be stored portably as formatted
> floating-point values. The pixel values themselves are binary, but
> they are assured to start on a 512-byte boundary for quick, easy
> access (using Byte IO routines), and they are stored as a byte stream
> without delimiters of any kind. This format is completely
> transportable between all computers and networks, including PCs, NT
> workstations, and SGI systems. The only coding difference on these
> systems is that big-endian systems such as SGI must byte-swap the
> pixel values in the (relatively rare) case where it encounters a
> 16-bit frame.
>
> Up
> to 32 bits/pixel are supported, although frames are stored where
> possible as 8 or 16 bits per pixel with an overflow table like that
> on earlier systems (but this overflow table is ASCII, as described
> below). BYTE ORDER OF PIXEL VALUES IN FRAMES WHICH ARE STORED IN THIS
> FORMAT ON DISK, TAPE, ETC. IS ALWAYS "LITTLE-ENDIAN" (LSB
> first, even though the header contains a flag indicating the pixel
> byte ordering). Bruker does not support big-endian frames that
> otherwise look like this format. The 16-bit format is fully
> implemented. If more than 4096 overflows occur on attempted
> compression to 8 bits, the frame buffer will compress to 16 bits with
> an overflow table (the SAXS HI-STAR and SMART CCD frame buffers
> accumulate 32 bits/pixel internally).
>
> At
> the top level, a frame in Bruker format can be represented as:
>
> Header
> Image
> data
> Overflow
> table
> Trailer
> 1
> .
> .
> .
> Trailer
> N
>
> Header
>
> The
> header is 1) typable ASCII and 2) an even multiple of the 512-byte
> disk block size. Each entry in the header is an 80-byte unterminated
> ASCII line. Thus, the header size is always a multiple of 5 disk
> blocks to hold an integral number of lines. The first eight
> characters of each item contain the item name, with the remaining 72
> characters for ASCII data. Thus, the organization of the header can
> be represented as:
>
> HEADER
> Item_name1
> (8 char): Item_Data1 (72 chars)
> Item_name2
> (8 char): Item_Data2 (72 chars)
> .
> .
> .
> Item_nameN
> (8 char): Item_DataN (72 chars)
>
> This
> organization has several advantages. First, fast random access can be
> performed on the header to pull out or rewrite an individual entry.
> During random access, the item names provide a method for checking
> for corruption of the header block.
> Second,
> frames are actually TYPEable on both PCs and workstations if the
> screen wraps at 80 columns. The padding to a multiple of 5 disk
> blocks contains ASCII dot characters. The last characters of the
> padding are a CTRL+Z and a CTRL+D. This convention provides a
> "warning indicator" on a workstation during which a CTRL+S
> or CTRL+C can be issued, before the binary pixel values start to
> appear. On a PC, an MS-DOS TYPE command stops when it hits the
> CTRL+Z. Presence of the item names makes the output easily
> interpretable.
>
> Third,
> the format is highly portable, because different byte and word
> ordering conventions don't enter into the interpretation of ASCII
> data. Items currently contained in the header are listed below in
> positional order. The format is expandable. New information to be
> added to the header will always be added at the end. A version number
> is present at the start of the header so programs can deal with new
> header information.
>
> The
> header items are:
>
> Mnemonic Offset Version Type Description
> FORMAT
> : 0 1 Int Frame format. Always “86” for Bruker-format frames.
> VERSION: 1 1 Int Header
> version #, such as 1, 2, 3, 4, 5, 7, or 8.
> HDRBLKS: 2 1 Int Header
> size in 512-byte blocks, such as 10 or 15.
> TYPE : 3 1 C String indicating king of data in the frame. Valid primary
> types
> are:DEFAULTAdd
> frameRotation frameSCAN
> FRAMEUNWARPEDUNWARPED
> LINEARFIDUCIAL
> PLATEBACKGROUND<A> & <B> COMBINATIONPOLE
> FIGURE,
> [POLAR | STEREO | WULFF]Sometimes the primary
> type
> is appended with CALIBRATEDFIBERLPAMULT
> _FFPLATESMOOTHEDPole
> figure type may be appended
> with:INTERPOLATEDINVERTEDNORMEDROTATED
> SCHEMESYMMETRIZEDTILTED
> SITE : 4 1 C Site name, from Edit > Configure > User settings
> MODEL : 5 1 C Diffractometer model, from SAXI$ADMODEL
> USER : 6 1 C Username, from Edit > Configure > User settings
> SAMPLE
> : 7 1 C Sample ID, from Project > New/Edit
> SETNAME: 8 1 C Basic
> data set name
> RUN : 9 1 Int Run number within the data set
> SAMPNUM: 10 1 Int Specimen
> number within the data set
> TITLE : 11-18 1 C User comments (8 lines)
> NCOUNTS: 19 1,9 Int Total
> frame counts (followed by reference counts in Ver 9).
> 9:
> Adds direct-beam monitor counts to NCOUNTS line
> NOVERFL: 20 1 Int Number
> of overflows
> MINIMUM: 21 1 Int Minimum
> counts in a pixel
> MAXIMUM: 22 1 Int Maximum
> counts in a pixel
> NONTIME: 23 1 Int Number
> of on-time events
> NLATE : 24 1 Int Number of late events.
> FILENAM: 25 1 C (Original)
> frame filename
> CREATED: 26 1 C Date
> and time of creation
> CUMULAT: 27 1 R Accumulated
> exposure time in seconds
> ELAPSDR: 28 1 R Requested
> time for last exposure in seconds
> ELAPSDA: 29 1 R Actual
> time for last exposure in seconds
> OSCILLA: 30 1 Int Nonzero
> if acquired by oscillation
> NSTEPS
> : 31 1 Int # steps or oscillations in this frame
> RANGE : 32 1 R Scan range in decimal degrees (unsigned)
> START : 33 1 R Starting scan angle value, decimal degrees
> INCREME: 34 1 R Scan
> angle increment between frames (signed)
> NUMBER
> : 35 1 Int Sequence number of this frame in series (@0)
> NFRAMES: 36 1 Int Total
> number of frames in the series
> ANGLES
> : 37 1 4R Diffractometer angles( 2T, OM, PH, CH)
> NOVER64: 38 1 Int Number
> of pixels > 64K
> NPIXELB: 39 1 Int Number
> of bytes/pixel, such as 1, 2, or 4.
> NROWS : 40 1 Int Number of rasters in frame, such as 512, 1024, or 2048.
> NCOLS : 41 1 Int Number of pixels/raster, such as 512, 1024, or 2048
> WORDORD: 42 1 Int Order
> of bytes in word (0=LSB first)
> LONGORD: 43 1 Int Order
> of words in a longword (0=LSW first)
> TARGET
> : 44 1 C X-ray target material: Cu, Mo, Ag, Fe, Cr, Co, or other.
> SOURCEK: 45 1 R X-ray
> source kV
> SOURCEM: 46 1 R X-ray
> source milliamps
> FILTER
> : 47 1 C Filter/monochromator setting: Such as:Parallel, graphiteNi
> FilterC
> FilterZr
> FilterCross coupled Goebel Mirrors
> CELL : 48-49 1 6R Unit cell A,B,C,ALPHA,BETA,GAMMA
> MATRIX
> : 50-51 1 9R Orientation matrix (P3 conventions)
> LOWTEMP: 52 1,8,9 1:
> Int8: Int,3R9: 2Int Low temp flag. Note: Version 8 and 9 headers
> diverge.
> 8: Add sub-mnemonic TEMP: Adds set point, ramp rate,
> hold
> time9: Adds experimental temperature in hundredths C and
> CCD
> sensor temperature in hundredths C
> ZOOM : 53 1 3R Zoom: Xc, Yc, Mag
> CENTER
> : 54 1 2R X, Y of direct beam at 2-theta = 0
> DISTANC: 55 1 R Sample-detector
> distance, cm
> TRAILER: 56 1 Int Byte
> pointer to trailer info
> COMPRES: 57 1 C Compression
> scheme ID, if any. Such as:NONE
> LINEAR
> : 58 1 C Linear scale, offset for pixel values, typically 1.0, 0.0.
> PHD : 59 1 2R Discriminator: Pulse height settings
> PREAMP
> : 60 1 R Preamp gain setting
> CORRECT: 61 1 C Flood
> field correction filename
> WARPFIL: 62 1 C Brass
> plate correction filename
> WAVELEN: 63 1 3R Wavelengths
> (average, a1, a2)
> MAXXY : 64 2 2R X,Y pixel # of maximum counts
> AXIS : 65 3 Int Scan axis (1-4 for 2-theta, omega, phi, chi)
> ENDING
> : 66 3 4R Actual goniometer angles at end of frame
> DETPAR
> : 67-68 4 6R Detector position corrections
> (dX,dY,dDist,Pitch,Roll,Yaw)
> LUT : 69 4 C Recommended display lookup table
> DISPLIM: 70 4 2R Recommended
> display limits
> PROGRAM: 71 4 C Name
> and version of program writing frame, such as:SAXS for
> WNT V4.0.07
> ROTATE
> : 72 5 Int Nonzero if acquired by rotation of phi during scan
> BITMASK: 73 5 C File
> name of active pixel mask associated with this frame or
> $NULL
> OCTMASK: 74-75 5 8Int Octagon
> mask parameters to use if BITMASK=$null
> ESDCELL: 76-77 7 6R Unit
> cell parameter standard deviations
> DETTYPE: 78 78 C2Int Detector
> type (0=multiwire, 1=CCD)MULTIWIRE (default)CCD-
> PXL
> (small CCD)CCD-PXL-2K (Large CCD)CCD-PXL-ARR
> (Mosaic)CCD-PXL-L6000CCD-PXL-KAF1500UNKNOWN
> (imported)OTHER
> (imported)4.0.04: Added sub-mnemonics
> PIXPERCM:
> and CMTOGRID: which are needed for UNKNOWN
> or
> OTHER types.
> NEXP : 79 7,9,10 Int Number of exposures: 1=single, 2=correlated
> sum(followed by
> fixed
> bias ADU in Vers 9).9: Adds per-exposure bias level in ADU
> to
> NEXP line10: Adds baseline offset (normally 32), Adds
> Orientation
> (value of SAXI$INVERTCCD), Adds Overscan flag
> (1
> if overscan used)
> CCDPARM: 80 7 5R CCD
> parameters: readnoise, e/ADU, e/photon, bias, full scale
> CHEM : 81 7 C Chemical formula in CIFTAB string, such as “?”
> MORPH : 82 7 C Crystal morphology in CIFTAB string, such as “?”
> CCOLOR
> : 83 7 C Crystal color in CIFTAB string, such as “?”
> CSIZE : 84 7 C Crystal dimensions (3 ea) in CIFTAB string, such as “?”
> DNSMET
> : 85 7 C Density measurement method in CIFTAB string, such as “?”
> DARK : 86 7 C Name of dark current correction
> AUTORNG: 87 7 5R Auto-ranging:
> gain, high-speed time, scale, offset, full scale
> ZEROADJ: 88 7 4R Goniometer
> zero corrections (refined in least squares)
> XTRANS
> : 89 7 3R Crystal XYZ translations (refined in least squares)
> HKL&XY
> : 90 8 5R HKL and pixel XY for reciprocal space scan
> AXES2 : 91 8 4R Diffractometer setting linear axes (4 ea). (X, Y, Z, Aux)
> ENDING2: 92 8 4R Actual
> goniometer linear axes @ end of frame. (X, Y, Z, Aux)
> 93
> 94
> 95
> Note:
> Items in version 6 headers are obsolete (MICRODIF). Use FRMFIX to
> convert these frame headers to version 8.
> Note:
> Version 5 & 8 items are mainly for GADDS
> Note:
> Version 7 & 9 items are mainly for SMART
> Note:
> Version 9 & 10 items do not add additional lines to the header,
> but instead appends additional values to a previously existing line.
>
> Image
> Data
>
> The
> image data is a simple byte stream (no record delimiters) with one,
> two, or four bytes per pixel, as defined in the header. If the data
> are two bytes per pixel, the least-significant byte is first.
>
> The
> order of the pixels in the file corresponds to the display of pixels
> on a screen, like the raster-scan order of a CRT display. The first
> pixel in the file corresponds to the upper-left corner of the
> display, the 512th to the upper right, and the last pixel to the
> lower right. This data ordering convention is the one used by most
> display devices and by the X-Windows library. Frames display as if
> viewed from the source toward the detector.
>
> Overflow
> Table
>
> The
> overflow table is stored as ASCII values. Each overflow entry is 16
> characters, comprising a 9-character intensity and a 7-character
> pixel # offset. The table is padded to a multiple of 512 bytes. In
> an 8-bit frame, any pixel with a value of 255 in the image needs to
> be looked up in the overflow table to determine its true value (even
> if the true value is 255, to allow overflow table validity checks,
> which could not otherwise be made). In a similar manner, any pixel
> in a 16-bit frame with a value of 65535 must be looked up in the
> overflow table to determine its true value. To look up a pixel
> value, compute its pixel displacement (for example, in a 512x512
> frame, 512*j + k, where j is the zero-based row number and k is the
> zero-based column number), and compare the displacement with that of
> each overflow table entry until a match is found. While the overflow
> table is normally sorted on displacement, it is not guaranteed to be
> sorted, so we recommend that you search the whole table until you
> find a match.
>
> Trailers
>
> The
> header contains a byte pointer (item name "TRAILER", set to
> zero if none) to the location in the file of any "trailer"
> data that has been tacked onto the image. This provision won't be
> used immediately, but provides later expansion capability. For
> example, if a frame is derived from processing some other (parent)
> frame, the header of the parent could be tacked onto the processed
> frame as a trailer to provide processing history. The format of a
> trailer is:
>
> TRAILER
> Trailer
> size in 512-byte blocks (1 byte)
> 3-byte
> pad
> Trailer
> data
>
> ----- Message d'origine ----
> De : "joshuala...@gmail.com" <joshuala...@gmail.com>
> À : Area Diffraction Machine <area-diffrac...@googlegroups.com>
> Envoyé le : Mardi, 10 Juin 2008, 0h18mn 23s
> Objet : [area-diffraction-machine] Re: Bruker CCD data?
>
> Ah thanks for the tip Auré. I have the extension properly set in the
> latest beta of version 2 of the program.
>
> Josh
>
> On Jun 9, 4:26 am, "aurelien.gourrier" <aurelien.gourr...@yahoo.fr>
> wrote:
>
>
>
> > Hi all,
>
> > Bruker file extension is usually .gfrm whatever that means...
>
> > Cheers,
>
> > Auré
>
> > On 5 mai, 23:25, joshuala...@gmail.com wrote:
>
> > > Ok, I have been playing around with the data that you sent me, and I
> > > can mostly get the program to read in the file. I have uploaded a beta
> > > of version 2 of the program (with several other changes) that can
> > > mostly read in the Bruker data you sent me. (http://code.google.com/p/
> > > areadiffractionmachine/downloads/detail?name=Area%20Diffraction
> > > %20Machine%20v2_beta_r194.zip&can=2&q=#makechanges). When I get really
> > > convinced that my code is doing the right thing reading these files
> > > in, I will move the feature down into the more stable version 1 of the
> > > program.
>
> > > Matt, is there any way you can open my program with a bunch of your
> > > data and see if it causes any weird errors?
>
> > > I have a couple of concerns about the way the code is now reading data
> > > in. If you or anybody you know knows anything about these bruker
> > > files, I have a couple of questions. First, it looks like Bruker files
> > > can contain 8 bit and 32 bit Bruker data. I have no idea if my program
> > > will properly read this data in because I don't have any data to test.
> > > Do you have any 8 or 32 bit data I can test the program with? Also,
> > > because of the way my program stores the data, any 32 bit data which
> > > is above 2^31-1 would be clipped to 2^31-1. Hopefully, this won't be a
> > > problem.
>
> > > When I examine the files you sent me, there are 512=32*16 bytes at the
> > > end of the file that I can't figure out what to do with. They are not
> > > part of the regular image. Apperently there is a way to store at the
> > > end of a Bruker file overloaded pixels and each overloaded pixel takes
> > > up 16 bytes. So it sounds like there are 32 overloaded pixels at the
> > > end of the file. But the number of overloaded pixels is supposed to be
> > > set in the header with the "NOVERFL:" value and in both of the files
> > > the value is set to 0. So I am not really sure what is going on. It
> > > might be that the program that wrote the file improperly set the
> > > header value NOVERFL. Right now, the program just doesn't try to read
> > > in any overloaded data. But If you have any data where the header
> > > value NOVERFL is not 0 or any other ideas about what the data at the
> > > end really is, I am all ears.
>
> > > Finally, I can find the detector to sample distance, the wavelength,
> > > and the x and y center of the image in the header. But I can't find
> > > the pixel length and pixel height of the detector. This is the width
> > > and height of a pixel on the detector (in microns). If you know where
> > > in the header these values are (or at least what these values actually
> > > are for the detector that took the data), that would help me.
>
> > > Finally, the files you sent me have a .002 extension but do all Bruker
> > > files have a .002 extension. Right now, I have the default extension
> > > of bruker data .bruker but I know this is not right. Do you know what
> > > the default extension should be?
>
> > > Josh
>
> > > On Mar 30, 11:28 pm, joshuala...@gmail.com wrote:
>
> > > > I added an issue to the issue tracker about this:http://code.google.com/p/areadiffractionmachine/issues/detail?id=28
>
> > > > Hopefully this won't be too hard to add in.
>
> > > > Josh
>
> > > > On Mar 27, 5:51 pm, joshuala...@gmail.com wrote:
>
> > > > > Hi Matt. I am always happy to hear new interest in the program. I am
> > > > > kind of busy right now writing the software manual and adding support
> > > > > to the program for caking in 2theta and making it so that pixel masks
> > > > > apply when calibrating. But I would be happy to work on this feature
> > > > > when I have some free time. Basically, I don't have any bruker data so
> > > > > if you want my program to read the data in, you or somebody else will
> > > > > have to give me some actual data in the bruker format to play with.
>
> > > > > If you join the discussion group, you can upload a file using the
> > > > > Google Groups interface. Go to the files tab on the right. Otherwise,
> > > > > if you create a Google Code account (which can be tied to any email
> > > > > address or you have gmail you already have one) you can go the Area
> > > > > Diffraction Machine's Google Code website and upload a file through
> > > > > the issue tracker. Otherwise, you can just email the file to me (if it
> > > > > isn't too big) or put it on an FTP.
>
> > > > > Also, if you have any information on the bruker file format that would
> > > > > help me code up the feature, I would be happy to hear about it.
> > > > > Otherwise, I will go do the digging myself.
>
> > > > > Josh
>
> > > > > On Mar 27, 9:50 am, Matt Ginder-Vogel <mat...@gmail.com> wrote:
>
> > > > > > Hi,
>
> > > > > > I was talking with Sam Webb today and he thought you might be willing
> > > > > > to modify you're program to work with bruker data. Would you?
>
> > > > > > I can send you some data files.
>
> > > > > > Thanks
> > > > > > Matt
>
> > > > > > ---------------------------------------------
> > > > > > Matt Ginder - Vogel
> > > > > > Post Doctoral Associate
> > > > > > Center for Critical Zone Research
> > > > > > Dept. of Plant and Soil Sciences
> > > > > > University of Delaware
> > > > > > mat...@udel.edu
> > > > > > ---------------------------------------------
>
> _____________________________________________________________________________
> Envoyez avec Yahoo! Mail. Une boite mail plus intelligentehttp://mail.yahoo.fr
Envoyé avec Yahoo! Mail.
Une boite mail plus intelligente.
De : "joshu...@gmail.com" <joshu...@gmail.com>
À : Area Diffraction Machine <area-diffrac...@googlegroups.com>
Envoyé le : Vendredi, 13 Juin 2008, 1h46mn 31s
Objet : [area-diffraction-machine] Re: Re : [area-diffraction-machine] Re: Bruker CCD data?
Ah, just for reference, I found an IDL routine which can read in
Bruker data: (http://cars9.uchicago.edu/software/idl/detectors.html).
It is the read_siemens.pro file and although I have never read IDL
code before, I think that it corroborates this documentation on the
Bruker file format.
Also, I Googled parts of your document to try to find where it came
from, and I found a mostly verbatim copy of this document at
http://www.esc.cam.ac.uk/index.php/component/content/article/60-Facilities/229-saxi-area-detector-frame-format.
It says that it describes the "SAXI Area Detector Frame Format."
On Jun 10, 4:27 am, Auré Gourrier <aurelien.gourr...@yahoo.fr> wrote:
> Hi again,
>
> Some time ago, I copied a document from a manual concerning the bruker file format, I attach it below. In particular, there is a description on the table overflow business, hope this helps...
>
> Cheers,
>
> Auré
>
> -----
>
> Appendix
> A. Area Detector Frame Format
>
> The
> Bruker format addresses issues of data portability, speed of image
> I/O and display, and storage of all the data acquisition conditions
> required to reduce the data. Header and overflow information are
> stored in portable ASCII, rather than binary. Critical data, such as
> setting angles, can thus be stored portably as formatted
> floating-point values. The pixel values themselves are binary, but
> they are assured to start on a 512-byte boundary for quick, easy
> access (using Byte IO routines), and they are stored as a byte stream
> without delimiters of any kind. This format is completely
> transportable between all computers and networks, including PCs, NT
> workstations, and SGI systems. The only coding difference on these
> systems is that big-endian systems such as SGI must byte-swap the
> pixel values in the (relatively rare) case where it encounters a
> 16-bit frame.
>
> Up
> to 32 bits/pixel are supported, although frames are stored where
> possible as 8 or 16 bits per pixel with an overflow table like that
> on earlier systems (but this overflow table is ASCII, as described
> below). BYTE ORDER OF PIXEL VALUES IN FRAMES WHICH ARE STORED IN THIS
> FORMAT ON DISK, TAPE, ETC. IS ALWAYS "LITTLE-ENDIAN" (LSB
> first, even though the header contains a flag indicating the pixel
> byte ordering). Bruker does not support big-endian frames that
> otherwise look like this format. The 16-bit format is fully
> implemented. If more than 4096 overflows occur on attempted
> compression to 8 bits, the frame buffer will compress to 16 bits with
> an overflow table (the SAXS HI-STAR and SMART CCD frame buffers
> accumulate 32 bits/pixel internally).
>
> At
> the top level, a frame in Bruker format can be represented as:
>
> Header
> Image
> data
> Overflow
> table
> Trailer
> 1
> .
> .
> .
> Trailer
> N
>
> Header
>
> The
> header is 1) typable ASCII and 2) an even multiple of the 512-byte
> disk block size. Each entry in the header is an 80-byte unterminated
> ASCII line. Thus, the header size is always a multiple of 5 disk
> blocks to hold an integral number of lines. The first eight
> characters of each item contain the item name, with the remaining 72
> characters for ASCII data. Thus, the organization of the header can
> be represented as:
>
> HEADER
> Item_name1
> (8 char): Item_Data1 (72 chars)
> Item_name2
> (8 char): Item_Data2 (72 chars)
> .
> .
> .
> Item_nameN
> (8 char): Item_DataN (72 chars)
>
> This
> organization has several advantages. First, fast random access can be
> performed on the header to pull out or rewrite an individual entry.
> During random access, the item names provide a method for checking
> for corruption of the header block.
> Second,
> frames are actually TYPEable on both PCs and workstations if the
> screen wraps at 80 columns. The padding to a multiple of 5 disk
> blocks contains ASCII dot characters. The last characters of the
> padding are a CTRL+Z and a CTRL+D. This convention provides a
> "warning indicator" on a workstation during which a CTRL+S
> or CTRL+C can be issued, before the binary pixel values start to
> appear. On a PC, an MS-DOS TYPE command stops when it hits the
> CTRL+Z. Presence of the item names makes the output easily
> interpretable.
>
> Third,
> the format is highly portable, because different byte and word
> ordering conventions don't enter into the interpretation of ASCII
> data. Items currently contained in the header are listed below in
> positional order. The format is expandable. New information to be
> added to the header will always be added at the end. A version number
> is present at the start of the header so programs can deal with new
> header information.
>
> The
> header items are:
>
> Mnemonic Offset Version Type Description
> FORMAT
> : 0 1 Int Frame format. Always “86” for Bruker-format frames.
> VERSION: 1 1 Int Header
> version #, such as 1, 2, 3, 4, 5, 7, or 8.
> HDRBLKS: 2 1 Int Header
> size in 512-byte blocks, such as 10 or 15.
> TYPE : 3 1 C String indicating king of data in the frame. Valid primary
> types
> are:DEFAULTAdd
> frameRotation frameSCAN
> FRAMEUNWARPEDUNWARPED
> LINEARFIDUCIAL
> PLATEBACKGROUND<A> & <B> COMBINATIONPOLE
> FIGURE,
> [POLAR | STEREO | WULFF]Sometimes the primary
> type
> is appended with CALIBRATEDFIBERLPAMULT
> _FFPLATESMOOTHEDPole
> figure type may be appended
> with:INTERPOLATEDINVERTEDNORMEDROTATED
> SCHEMESYMMETRIZEDTILTED
> SITE : 4 1 C Site name, from Edit > Configure > User settings
> MODEL : 5 1 C Diffractometer model, from SAXI$ADMODEL
> USER : 6 1 C Username, from Edit > Configure > User settings
> SAMPLE
> : 7 1 C Sample ID, from Project > New/Edit
> SETNAME: 8 1 C Basic
> data set name
> RUN : 9 1 Int Run number within the data set
> SAMPNUM: 10 1 Int Specimen
> number within the data set
> TITLE : 11-18 1 C User comments (8 lines)
> NCOUNTS: 19 1,9 Int Total
> frame counts (followed by reference counts in Ver 9).
> 9:
> Adds direct-beam monitor counts to NCOUNTS line
> NOVERFL: 20 1 Int Number
> of overflows
> MINIMUM: 21 1 Int Minimum
> counts in a pixel
> MAXIMUM: 22 1 Int Maximum
> counts in a pixel
> NONTIME: 23 1 Int Number
> of on-time events
> NLATE : 24 1 Int Number of late events.
> FILENAM: 25 1 C (Original)
> frame filename
> CREATED: 26 1 C Date
> and time of creation
> CUMULAT: 27 1 R Accumulated
> exposure time in seconds
> ELAPSDR: 28 1 R Requested
> time for last exposure in seconds
> ELAPSDA: 29 1 R Actual
> time for last exposure in seconds
> OSCILLA: 30 1 Int Nonzero
> if acquired by oscillation
> NSTEPS
> : 31 1 Int # steps or oscillations in this frame
> RANGE : 32 1 R Scan range in decimal degrees (unsigned)
> START : 33 1 R Starting scan angle value, decimal degrees
> INCREME: 34 1 R Scan
> angle increment between frames (signed)
> NUMBER
> : 35 1 Int Sequence number of this frame in series (@0)
> NFRAMES: 36 1 Int Total
> number of frames in the series
> ANGLES
> : 37 1 4R Diffractometer angles( 2T, OM, PH, CH)
> NOVER64: 38 1 Int Number
> of pixels > 64K
> NPIXELB: 39 1 Int Number
> of bytes/pixel, such as 1, 2, or 4.
> NROWS : 40 1 Int Number of rasters in frame, such as 512, 1024, or 2048.
> NCOLS : 41 1 Int Number of pixels/raster, such as 512, 1024, or 2048
> WORDORD: 42 1 Int Order
> of bytes in word (0=LSB first)
> LONGORD: 43 1 Int Order
> of words in a longword (0=LSW first)
> TARGET
> : 44 1 C X-ray target material: Cu, Mo, Ag, Fe, Cr, Co, or other.
> SOURCEK: 45 1 R X-ray
> source kV
> SOURCEM: 46 1 R X-ray
> source milliamps
> FILTER
> : 47 1 C Filter/monochromator setting: Such as:Parallel, graphiteNi
> FilterC
> FilterZr
> FilterCross coupled Goebel Mirrors
> CELL : 48-49 1 6R Unit cell A,B,C,ALPHA,BETA,GAMMA
> MATRIX
> : 50-51 1 9R Orientation matrix (P3 conventions)
> LOWTEMP: 52 1,8,9 1:
> Int8: Int,3R9: 2Int Low temp flag. Note: Version 8 and 9 headers
> diverge.
> 8: Add sub-mnemonic TEMP: Adds set point, ramp rate,
> hold
> time9: Adds experimental temperature in hundredths C and
> CCD
> sensor temperature in hundredths C
> ZOOM : 53 1 3R Zoom: Xc, Yc, Mag
> CENTER
> : 54 1 2R X, Y of direct beam at 2-theta = 0
> DISTANC: 55 1 R Sample-detector
> distance, cm
> TRAILER: 56 1 Int Byte
> pointer to trailer info
> COMPRES: 57 1 C Compression
> scheme ID, if any. Such as:NONE
> LINEAR
> : 58 1 C Linear scale, offset for pixel values, typically 1.0, 0.0.
> PHD : 59 1 2R Discriminator: Pulse height settings
> PREAMP
> : 60 1 R Preamp gain setting
> CORRECT: 61 1 C Flood
> field correction filename
> WARPFIL: 62 1 C Brass
> plate correction filename
> WAVELEN: 63 1 3R Wavelengths
> (average, a1, a2)
> MAXXY : 64 2 2R X,Y pixel # of maximum counts
> AXIS : 65 3 Int Scan axis (1-4 for 2-theta, omega, phi, chi)
> ENDING
> : 66 3 4R Actual goniometer angles at end of frame
> DETPAR
> : 67-68 4 6R Detector position corrections
> (dX,dY,dDist,Pitch,Roll,Yaw)
> LUT : 69 4 C Recommended display lookup table
> DISPLIM: 70 4 2R Recommended
> display limits
> PROGRAM: 71 4 C Name
> and version of program writing frame, such as:SAXS for
> WNT V4.0.07
> ROTATE
> : 72 5 Int Nonzero if acquired by rotation of phi during scan
> BITMASK: 73 5 C File
> name of active pixel mask associated with this frame or
> $NULL
> OCTMASK: 74-75 5 8Int Octagon
> mask parameters to use if BITMASK=$null
> ESDCELL: 76-77 7 6R Unit
> cell parameter standard deviations
> DETTYPE: 78 78 C2Int Detector
> type (0=multiwire, 1=CCD)MULTIWIRE (default)CCD-
> PXL
> (small CCD)CCD-PXL-2K (Large CCD)CCD-PXL-ARR
> (Mosaic)CCD-PXL-L6000CCD-PXL-KAF1500UNKNOWN
> (imported)OTHER
> (imported)4.0.04: Added sub-mnemonics
> PIXPERCM:
> and CMTOGRID: which are needed for UNKNOWN
> or
> OTHER types.
> NEXP : 79 7,9,10 Int Number of exposures: 1=single, 2=correlated
> sum(followed by
> fixed
> bias ADU in Vers 9).9: Adds per-exposure bias level in ADU
> to
> NEXP line10: Adds baseline offset (normally 32), Adds
> Orientation
> (value of SAXI$INVERTCCD), Adds Overscan flag
> (1
> if overscan used)
> CCDPARM: 80 7 5R CCD
> parameters: readnoise, e/ADU, e/photon, bias, full scale
> CHEM : 81 7 C Chemical formula in CIFTAB string, such as “?”
> MORPH : 82 7 C Crystal morphology in CIFTAB string, such as “?”
> CCOLOR
> : 83 7 C Crystal color in CIFTAB string, such as “?”
> CSIZE : 84 7 C Crystal dimensions (3 ea) in CIFTAB string, such as “?”
> DNSMET
> : 85 7 C Density measurement method in CIFTAB string, such as “?”
> DARK : 86 7 C Name of dark current correction
> AUTORNG: 87 7 5R Auto-ranging:
> gain, high-speed time, scale, offset, full scale
> ZEROADJ: 88 7 4R Goniometer
> zero corrections (refined in least squares)
> XTRANS
> : 89 7 3R Crystal XYZ translations (refined in least squares)
> HKL&XY
> : 90 8 5R HKL and pixel XY for reciprocal space scan
> AXES2 : 91 8 4R Diffractometer setting linear axes (4 ea). (X, Y, Z, Aux)
> ENDING2: 92 8 4R Actual
> goniometer linear axes @ end of frame. (X, Y, Z, Aux)
> 93
> 94
> 95
> Note:
> Items in version 6 headers are obsolete (MICRODIF). Use FRMFIX to
> convert these frame headers to version 8.
> Note:
> Version 5 & 8 items are mainly for GADDS
> Note:
> Version 7 & 9 items are mainly for SMART
> Note:
> Version 9 & 10 items do not add additional lines to the header,
> but instead appends additional values to a previously existing line.
>
> Image
> Data
>
> The
> image data is a simple byte stream (no record delimiters) with one,
> two, or four bytes per pixel, as defined in the header. If the data
> are two bytes per pixel, the least-significant byte is first.
>
> The
> order of the pixels in the file corresponds to the display of pixels
> on a screen, like the raster-scan order of a CRT display. The first
> pixel in the file corresponds to the upper-left corner of the
> display, the 512th to the upper right, and the last pixel to the
> lower right. This data ordering convention is the one used by most
> display devices and by the X-Windows library. Frames display as if
> viewed from the source toward the detector.
>
> Overflow
> Table
>
> The
> overflow table is stored as ASCII values. Each overflow entry is 16
> characters, comprising a 9-character intensity and a 7-character
> pixel # offset. The table is padded to a multiple of 512 bytes. In
> an 8-bit frame, any pixel with a value of 255 in the image needs to
> be looked up in the overflow table to determine its true value (even
> if the true value is 255, to allow overflow table validity checks,
> which could not otherwise be made). In a similar manner, any pixel
> in a 16-bit frame with a value of 65535 must be looked up in the
> overflow table to determine its true value. To look up a pixel
> value, compute its pixel displacement (for example, in a 512x512
> frame, 512*j + k, where j is the zero-based row number and k is the
> zero-based column number), and compare the displacement with that of
> each overflow table entry until a match is found. While the overflow
> table is normally sorted on displacement, it is not guaranteed to be
> sorted, so we recommend that you search the whole table until you
> find a match.
>
> Trailers
>
> The
> header contains a byte pointer (item name "TRAILER", set to
> zero if none) to the location in the file of any "trailer"
> data that has been tacked onto the image. This provision won't be
> used immediately, but provides later expansion capability. For
> example, if a frame is derived from processing some other (parent)
> frame, the header of the parent could be tacked onto the processed
> frame as a trailer to provide processing history. The format of a
> trailer is:
>
> TRAILER
> Trailer
> size in 512-byte blocks (1 byte)
> 3-byte
> pad
> Trailer
> data
>
> ----- Message d'origine ----
> De : "joshuala...@gmail.com" <joshuala...@gmail.com>
> À : Area Diffraction Machine <area-diffrac...@googlegroups.com>
> Envoyé le : Mardi, 10 Juin 2008, 0h18mn 23s
> Objet : [area-diffraction-machine] Re: Bruker CCD data?
>
> Ah thanks for the tip Auré. I have the extension properly set in the
> latest beta of version 2 of the program.
>
> Josh
>
> On Jun 9, 4:26 am, "aurelien.gourrier" <aurelien.gourr...@yahoo.fr>
> wrote:
>
>
>
> > Hi all,
>
> > Bruker file extension is usually .gfrm whatever that means...
>
> > Cheers,
>
> > Auré
>
> > On 5 mai, 23:25, joshuala...@gmail.com wrote:
>
> > > Ok, I have been playing around with the data that you sent me, and I
> > > can mostly get the program to read in the file. I have uploaded a beta
> > > of version 2 of the program (with several other changes) that can
> > > mostly read in the Bruker data you sent me. (http://code.google.com/p/
> > > areadiffractionmachine/downloads/detail?name=Area%20Diffraction
> > > %20Machine%20v2_beta_r194.zip&can=2&q=#makechanges). When I get really
> > > convinced that my code is doing the right thing reading these files
> > > in, I will move the feature down into the more stable version 1 of the
> > > program.
>
> > > Matt, is there any way you can open my program with a bunch of your
> > > data and see if it causes any weird errors?
>
> > > I have a couple of concerns about the way the code is now reading data
> > > in. If you or anybody you know knows anything about these bruker
> > > files, I have a couple of questions. First, it looks like Bruker files
> > > can contain 8 bit and 32 bit Bruker data. I have no idea if my program
> > > will properly read this data in because I don't have any data to test.
> > > Do you have any 8 or 32 bit data I can test the program with? Also,
> > > because of the way my program stores the data, any 32 bit data which
> > > is above 2^31-1 would be clipped to 2^31-1. Hopefully, this won't be a
> > > problem.
>
> > > When I examine the files you sent me, there are 512=32*16 bytes at the
> > > end of the file that I can't figure out what to do with. They are not
> > > part of the regular image. Apperently there is a way to store at the
> > > end of a Bruker file overloaded pixels and each overloaded pixel takes
> > > up 16 bytes. So it sounds like there are 32 overloaded pixels at the
> > > end of the file. But the number of overloaded pixels is supposed to be
> > > set in the header with the "NOVERFL:" value and in both of the files
> > > the value is set to 0. So I am not really sure what is going on. It
> > > might be that the program that wrote the file improperly set the
> > > header value NOVERFL. Right now, the program just doesn't try to read
> > > in any overloaded data. But If you have any data where the header
> > > value NOVERFL is not 0 or any other ideas about what the data at the
> > > end really is, I am all ears.
>
> > > Finally, I can find the detector to sample distance, the wavelength,
> > > and the x and y center of the image in the header. But I can't find
> > > the pixel length and pixel height of the detector. This is the width
> > > and height of a pixel on the detector (in microns). If you know where
> > > in the header these values are (or at least what these values actually
> > > are for the detector that took the data), that would help me.
>
> > > Finally, the files you sent me have a .002 extension but do all Bruker
> > > files have a .002 extension. Right now, I have the default extension
> > > of bruker data .bruker but I know this is not right. Do you know what
> > > the default extension should be?
>
> > > Josh
>
> > > On Mar 30, 11:28 pm, joshuala...@gmail.com wrote:
>
> > > > I added an issue to the issue tracker about this:http://code.google.com/p/areadiffractionmachine/issues/detail?id=28
>
> > > > Hopefully this won't be too hard to add in.
>
> > > > Josh
>
> > > > On Mar 27, 5:51 pm, joshuala...@gmail.com wrote:
>
> > > > > Hi Matt. I am always happy to hear new interest in the program. I am
> > > > > kind of busy right now writing the software manual and adding support
> > > > > to the program for caking in 2theta and making it so that pixel masks
> > > > > apply when calibrating. But I would be happy to work on this feature
> > > > > when I have some free time. Basically, I don't have any bruker data so
> > > > > if you want my program to read the data in, you or somebody else will
> > > > > have to give me some actual data in the bruker format to play with.
>
> > > > > If you join the discussion group, you can upload a file using the
> > > > > Google Groups interface. Go to the files tab on the right. Otherwise,
> > > > > if you create a Google Code account (which can be tied to any email
> > > > > address or you have gmail you already have one) you can go the Area
> > > > > Diffraction Machine's Google Code website and upload a file through
> > > > > the issue tracker. Otherwise, you can just email the file to me (if it
> > > > > isn't too big) or put it on an FTP.
>
> > > > > Also, if you have any information on the bruker file format that would
> > > > > help me code up the feature, I would be happy to hear about it.
> > > > > Otherwise, I will go do the digging myself.
>
> > > > > Josh
>
> > > > > On Mar 27, 9:50 am, Matt Ginder-Vogel <mat...@gmail.com> wrote:
>
> > > > > > Hi,
>
> > > > > > I was talking with Sam Webb today and he thought you might be willing
> > > > > > to modify you're program to work with bruker data. Would you?
>
> > > > > > I can send you some data files.
>
> > > > > > Thanks
> > > > > > Matt
>
> > > > > > ---------------------------------------------
> > > > > > Matt Ginder - Vogel
> > > > > > Post Doctoral Associate
> > > > > > Center for Critical Zone Research
> > > > > > Dept. of Plant and Soil Sciences
> > > > > > University of Delaware
> > > > > > mat...@udel.edu
> > > > > > ---------------------------------------------
>
> _____________________________________________________________________________
> Envoyez avec Yahoo! Mail. Une boite mail plus intelligentehttp://mail.yahoo.fr
Envoyé avec Yahoo! Mail.
Une boite mail plus intelligente.
Reply all
Reply to author
Forward
0 new messages