Re : [area-diffraction-machine] Re: Bruker CCD data?
Auré Gourrier
Some time ago, I copied a document from a manual concerning the bruker file format, I attach it below. In particular, there is a description on the table overflow business, hope this helps...
Cheers,
Auré
-----
Appendix A. Area Detector Frame Format
The Bruker format addresses issues of data portability, speed of image I/O and display, and storage of all the data acquisition conditions required to reduce the data. Header and overflow information are stored in portable ASCII, rather than binary. Critical data, such as setting angles, can thus be stored portably as formatted floating-point values. The pixel values themselves are binary, but they are assured to start on a 512-byte boundary for quick, easy access (using Byte IO routines), and they are stored as a byte stream without delimiters of any kind. This format is completely transportable between all computers and networks, including PCs, NT workstations, and SGI systems. The only coding difference on these systems is that big-endian systems such as SGI must byte-swap the pixel values in the (relatively rare) case where it encounters a 16-bit frame.
Up to 32 bits/pixel are supported, although frames are stored where possible as 8 or 16 bits per pixel with an overflow table like that on earlier systems (but this overflow table is ASCII, as described below). BYTE ORDER OF PIXEL VALUES IN FRAMES WHICH ARE STORED IN THIS FORMAT ON DISK, TAPE, ETC. IS ALWAYS "LITTLE-ENDIAN" (LSB first, even though the header contains a flag indicating the pixel byte ordering). Bruker does not support big-endian frames that otherwise look like this format. The 16-bit format is fully implemented. If more than 4096 overflows occur on attempted compression to 8 bits, the frame buffer will compress to 16 bits with an overflow table (the SAXS HI-STAR and SMART CCD frame buffers accumulate 32 bits/pixel internally).
At the top level, a frame in Bruker format can be represented as:
Header
Image data
Overflow table
Trailer 1
.
.
.
Trailer N
Header
The header is 1) typable ASCII and 2) an even multiple of the 512-byte disk block size. Each entry in the header is an 80-byte unterminated ASCII line. Thus, the header size is always a multiple of 5 disk blocks to hold an integral number of lines. The first eight characters of each item contain the item name, with the remaining 72 characters for ASCII data. Thus, the organization of the header can be represented as:
HEADER
Item_name1 (8 char): Item_Data1 (72 chars)
Item_name2 (8 char): Item_Data2 (72 chars)
.
.
.
Item_nameN (8 char): Item_DataN (72 chars)
This organization has several advantages. First, fast random access can be performed on the header to pull out or rewrite an individual entry. During random access, the item names provide a method for checking for corruption of the header block.
Second, frames are actually TYPEable on both PCs and workstations if the screen wraps at 80 columns. The padding to a multiple of 5 disk blocks contains ASCII dot characters. The last characters of the padding are a CTRL+Z and a CTRL+D. This convention provides a "warning indicator" on a workstation during which a CTRL+S or CTRL+C can be issued, before the binary pixel values start to appear. On a PC, an MS-DOS TYPE command stops when it hits the CTRL+Z. Presence of the item names makes the output easily interpretable.
Third, the format is highly portable, because different byte and word ordering conventions don't enter into the interpretation of ASCII data. Items currently contained in the header are listed below in positional order. The format is expandable. New information to be added to the header will always be added at the end. A version number is present at the start of the header so programs can deal with new header information.
The header items are:
Mnemonic Offset Version Type Description
FORMAT : 0 1 Int Frame format. Always “86” for Bruker-format frames.
VERSION: 1 1 Int Header version #, such as 1, 2, 3, 4, 5, 7, or 8.
HDRBLKS: 2 1 Int Header size in 512-byte blocks, such as 10 or 15.
TYPE : 3 1 C String indicating king of data in the frame. Valid primary types
are:DEFAULTAdd frameRotation frameSCAN
FRAMEUNWARPEDUNWARPED LINEARFIDUCIAL
PLATEBACKGROUND<A> & <B> COMBINATIONPOLE
FIGURE, [POLAR | STEREO | WULFF]Sometimes the primary
type is appended with CALIBRATEDFIBERLPAMULT
_FFPLATESMOOTHEDPole figure type may be appended
with:INTERPOLATEDINVERTEDNORMEDROTATED
SCHEMESYMMETRIZEDTILTED
SITE : 4 1 C Site name, from Edit > Configure > User settings
MODEL : 5 1 C Diffractometer model, from SAXI$ADMODEL
USER : 6 1 C Username, from Edit > Configure > User settings
SAMPLE : 7 1 C Sample ID, from Project > New/Edit
SETNAME: 8 1 C Basic data set name
RUN : 9 1 Int Run number within the data set
SAMPNUM: 10 1 Int Specimen number within the data set
TITLE : 11-18 1 C User comments (8 lines)
NCOUNTS: 19 1,9 Int Total frame counts (followed by reference counts in Ver 9).
9: Adds direct-beam monitor counts to NCOUNTS line
NOVERFL: 20 1 Int Number of overflows
MINIMUM: 21 1 Int Minimum counts in a pixel
MAXIMUM: 22 1 Int Maximum counts in a pixel
NONTIME: 23 1 Int Number of on-time events
NLATE : 24 1 Int Number of late events.
FILENAM: 25 1 C (Original) frame filename
CREATED: 26 1 C Date and time of creation
CUMULAT: 27 1 R Accumulated exposure time in seconds
ELAPSDR: 28 1 R Requested time for last exposure in seconds
ELAPSDA: 29 1 R Actual time for last exposure in seconds
OSCILLA: 30 1 Int Nonzero if acquired by oscillation
NSTEPS : 31 1 Int # steps or oscillations in this frame
RANGE : 32 1 R Scan range in decimal degrees (unsigned)
START : 33 1 R Starting scan angle value, decimal degrees
INCREME: 34 1 R Scan angle increment between frames (signed)
NUMBER : 35 1 Int Sequence number of this frame in series (@0)
NFRAMES: 36 1 Int Total number of frames in the series
ANGLES : 37 1 4R Diffractometer angles( 2T, OM, PH, CH)
NOVER64: 38 1 Int Number of pixels > 64K
NPIXELB: 39 1 Int Number of bytes/pixel, such as 1, 2, or 4.
NROWS : 40 1 Int Number of rasters in frame, such as 512, 1024, or 2048.
NCOLS : 41 1 Int Number of pixels/raster, such as 512, 1024, or 2048
WORDORD: 42 1 Int Order of bytes in word (0=LSB first)
LONGORD: 43 1 Int Order of words in a longword (0=LSW first)
TARGET : 44 1 C X-ray target material: Cu, Mo, Ag, Fe, Cr, Co, or other.
SOURCEK: 45 1 R X-ray source kV
SOURCEM: 46 1 R X-ray source milliamps
FILTER : 47 1 C Filter/monochromator setting: Such as:Parallel, graphiteNi FilterC
FilterZr FilterCross coupled Goebel Mirrors
CELL : 48-49 1 6R Unit cell A,B,C,ALPHA,BETA,GAMMA
MATRIX : 50-51 1 9R Orientation matrix (P3 conventions)
LOWTEMP: 52 1,8,9 1: Int8: Int,3R9: 2Int Low temp flag. Note: Version 8 and 9 headers
diverge. 8: Add sub-mnemonic TEMP: Adds set point, ramp rate,
hold time9: Adds experimental temperature in hundredths C and
CCD sensor temperature in hundredths C
ZOOM : 53 1 3R Zoom: Xc, Yc, Mag
CENTER : 54 1 2R X, Y of direct beam at 2-theta = 0
DISTANC: 55 1 R Sample-detector distance, cm
TRAILER: 56 1 Int Byte pointer to trailer info
COMPRES: 57 1 C Compression scheme ID, if any. Such as:NONE
LINEAR : 58 1 C Linear scale, offset for pixel values, typically 1.0, 0.0.
PHD : 59 1 2R Discriminator: Pulse height settings
PREAMP : 60 1 R Preamp gain setting
CORRECT: 61 1 C Flood field correction filename
WARPFIL: 62 1 C Brass plate correction filename
WAVELEN: 63 1 3R Wavelengths (average, a1, a2)
MAXXY : 64 2 2R X,Y pixel # of maximum counts
AXIS : 65 3 Int Scan axis (1-4 for 2-theta, omega, phi, chi)
ENDING : 66 3 4R Actual goniometer angles at end of frame
DETPAR : 67-68 4 6R Detector position corrections (dX,dY,dDist,Pitch,Roll,Yaw)
LUT : 69 4 C Recommended display lookup table
DISPLIM: 70 4 2R Recommended display limits
PROGRAM: 71 4 C Name and version of program writing frame, such as:SAXS for
WNT V4.0.07
ROTATE : 72 5 Int Nonzero if acquired by rotation of phi during scan
BITMASK: 73 5 C File name of active pixel mask associated with this frame or
$NULL
OCTMASK: 74-75 5 8Int Octagon mask parameters to use if BITMASK=$null
ESDCELL: 76-77 7 6R Unit cell parameter standard deviations
DETTYPE: 78 78 C2Int Detector type (0=multiwire, 1=CCD)MULTIWIRE (default)CCD-
PXL (small CCD)CCD-PXL-2K (Large CCD)CCD-PXL-ARR
(Mosaic)CCD-PXL-L6000CCD-PXL-KAF1500UNKNOWN
(imported)OTHER (imported)4.0.04: Added sub-mnemonics
PIXPERCM: and CMTOGRID: which are needed for UNKNOWN
or OTHER types.
NEXP : 79 7,9,10 Int Number of exposures: 1=single, 2=correlated sum(followed by
fixed bias ADU in Vers 9).9: Adds per-exposure bias level in ADU
to NEXP line10: Adds baseline offset (normally 32), Adds
Orientation (value of SAXI$INVERTCCD), Adds Overscan flag
(1 if overscan used)
CCDPARM: 80 7 5R CCD parameters: readnoise, e/ADU, e/photon, bias, full scale
CHEM : 81 7 C Chemical formula in CIFTAB string, such as “?”
MORPH : 82 7 C Crystal morphology in CIFTAB string, such as “?”
CCOLOR : 83 7 C Crystal color in CIFTAB string, such as “?”
CSIZE : 84 7 C Crystal dimensions (3 ea) in CIFTAB string, such as “?”
DNSMET : 85 7 C Density measurement method in CIFTAB string, such as “?”
DARK : 86 7 C Name of dark current correction
AUTORNG: 87 7 5R Auto-ranging: gain, high-speed time, scale, offset, full scale
ZEROADJ: 88 7 4R Goniometer zero corrections (refined in least squares)
XTRANS : 89 7 3R Crystal XYZ translations (refined in least squares)
HKL&XY : 90 8 5R HKL and pixel XY for reciprocal space scan
AXES2 : 91 8 4R Diffractometer setting linear axes (4 ea). (X, Y, Z, Aux)
ENDING2: 92 8 4R Actual goniometer linear axes @ end of frame. (X, Y, Z, Aux)
93
94
95
Note: Items in version 6 headers are obsolete (MICRODIF). Use FRMFIX to convert these frame headers to version 8.
Note: Version 5 & 8 items are mainly for GADDS
Note: Version 7 & 9 items are mainly for SMART
Note: Version 9 & 10 items do not add additional lines to the header, but instead appends additional values to a previously existing line.
Image Data
The image data is a simple byte stream (no record delimiters) with one, two, or four bytes per pixel, as defined in the header. If the data are two bytes per pixel, the least-significant byte is first.
The order of the pixels in the file corresponds to the display of pixels on a screen, like the raster-scan order of a CRT display. The first pixel in the file corresponds to the upper-left corner of the display, the 512th to the upper right, and the last pixel to the lower right. This data ordering convention is the one used by most display devices and by the X-Windows library. Frames display as if viewed from the source toward the detector.
Overflow Table
The overflow table is stored as ASCII values. Each overflow entry is 16 characters, comprising a 9-character intensity and a 7-character pixel # offset. The table is padded to a multiple of 512 bytes. In an 8-bit frame, any pixel with a value of 255 in the image needs to be looked up in the overflow table to determine its true value (even if the true value is 255, to allow overflow table validity checks, which could not otherwise be made). In a similar manner, any pixel in a 16-bit frame with a value of 65535 must be looked up in the overflow table to determine its true value. To look up a pixel value, compute its pixel displacement (for example, in a 512x512 frame, 512*j + k, where j is the zero-based row number and k is the zero-based column number), and compare the displacement with that of each overflow table entry until a match is found. While the overflow table is normally sorted on displacement, it is not guaranteed to be sorted, so we recommend that you search the whole table until you find a match.
Trailers
The header contains a byte pointer (item name "TRAILER", set to zero if none) to the location in the file of any "trailer" data that has been tacked onto the image. This provision won't be used immediately, but provides later expansion capability. For example, if a frame is derived from processing some other (parent) frame, the header of the parent could be tacked onto the processed frame as a trailer to provide processing history. The format of a trailer is:
TRAILER
Trailer size in 512-byte blocks (1 byte)
3-byte pad
Trailer data
De : "joshu...@gmail.com" <joshu...@gmail.com>
À : Area Diffraction Machine <area-diffrac...@googlegroups.com>
Envoyé le : Mardi, 10 Juin 2008, 0h18mn 23s
Objet : [area-diffraction-machine] Re: Bruker CCD data?
Ah thanks for the tip Auré. I have the extension properly set in the
latest beta of version 2 of the program.
Josh
On Jun 9, 4:26 am, "aurelien.gourrier" <aurelien.gourr...@yahoo.fr>
wrote:
> Hi all,
>
> Bruker file extension is usually .gfrm whatever that means...
>
> Cheers,
>
> Auré
>
> On 5 mai, 23:25, joshuala...@gmail.com wrote:
>
> > Ok, I have been playing around with the data that you sent me, and I
> > can mostly get the program to read in the file. I have uploaded a beta
> > of version 2 of the program (with several other changes) that can
> > mostly read in the Bruker data you sent me. (http://code.google.com/p/
> > areadiffractionmachine/downloads/detail?name=Area%20Diffraction
> > %20Machine%20v2_beta_r194.zip&can=2&q=#makechanges). When I get really
> > convinced that my code is doing the right thing reading these files
> > in, I will move the feature down into the more stable version 1 of the
> > program.
>
> > Matt, is there any way you can open my program with a bunch of your
> > data and see if it causes any weird errors?
>
> > I have a couple of concerns about the way the code is now reading data
> > in. If you or anybody you know knows anything about these bruker
> > files, I have a couple of questions. First, it looks like Bruker files
> > can contain 8 bit and 32 bit Bruker data. I have no idea if my program
> > will properly read this data in because I don't have any data to test.
> > Do you have any 8 or 32 bit data I can test the program with? Also,
> > because of the way my program stores the data, any 32 bit data which
> > is above 2^31-1 would be clipped to 2^31-1. Hopefully, this won't be a
> > problem.
>
> > When I examine the files you sent me, there are 512=32*16 bytes at the
> > end of the file that I can't figure out what to do with. They are not
> > part of the regular image. Apperently there is a way to store at the
> > end of a Bruker file overloaded pixels and each overloaded pixel takes
> > up 16 bytes. So it sounds like there are 32 overloaded pixels at the
> > end of the file. But the number of overloaded pixels is supposed to be
> > set in the header with the "NOVERFL:" value and in both of the files
> > the value is set to 0. So I am not really sure what is going on. It
> > might be that the program that wrote the file improperly set the
> > header value NOVERFL. Right now, the program just doesn't try to read
> > in any overloaded data. But If you have any data where the header
> > value NOVERFL is not 0 or any other ideas about what the data at the
> > end really is, I am all ears.
>
> > Finally, I can find the detector to sample distance, the wavelength,
> > and the x and y center of the image in the header. But I can't find
> > the pixel length and pixel height of the detector. This is the width
> > and height of a pixel on the detector (in microns). If you know where
> > in the header these values are (or at least what these values actually
> > are for the detector that took the data), that would help me.
>
> > Finally, the files you sent me have a .002 extension but do all Bruker
> > files have a .002 extension. Right now, I have the default extension
> > of bruker data .bruker but I know this is not right. Do you know what
> > the default extension should be?
>
> > Josh
>
> > On Mar 30, 11:28 pm, joshuala...@gmail.com wrote:
>
> > > I added an issue to the issue tracker about this:http://code.google.com/p/areadiffractionmachine/issues/detail?id=28
>
> > > Hopefully this won't be too hard to add in.
>
> > > Josh
>
> > > On Mar 27, 5:51 pm, joshuala...@gmail.com wrote:
>
> > > > Hi Matt. I am always happy to hear new interest in the program. I am
> > > > kind of busy right now writing the software manual and adding support
> > > > to the program for caking in 2theta and making it so that pixel masks
> > > > apply when calibrating. But I would be happy to work on this feature
> > > > when I have some free time. Basically, I don't have any bruker data so
> > > > if you want my program to read the data in, you or somebody else will
> > > > have to give me some actual data in the bruker format to play with.
>
> > > > If you join the discussion group, you can upload a file using the
> > > > Google Groups interface. Go to the files tab on the right. Otherwise,
> > > > if you create a Google Code account (which can be tied to any email
> > > > address or you have gmail you already have one) you can go the Area
> > > > Diffraction Machine's Google Code website and upload a file through
> > > > the issue tracker. Otherwise, you can just email the file to me (if it
> > > > isn't too big) or put it on an FTP.
>
> > > > Also, if you have any information on the bruker file format that would
> > > > help me code up the feature, I would be happy to hear about it.
> > > > Otherwise, I will go do the digging myself.
>
> > > > Josh
>
> > > > On Mar 27, 9:50 am, Matt Ginder-Vogel <mat...@gmail.com> wrote:
>
> > > > > Hi,
>
> > > > > I was talking with Sam Webb today and he thought you might be willing
> > > > > to modify you're program to work with bruker data. Would you?
>
> > > > > I can send you some data files.
>
> > > > > Thanks
> > > > > Matt
>
> > > > > ---------------------------------------------
> > > > > Matt Ginder - Vogel
> > > > > Post Doctoral Associate
> > > > > Center for Critical Zone Research
> > > > > Dept. of Plant and Soil Sciences
> > > > > University of Delaware
> > > > > mat...@udel.edu
> > > > > ---------------------------------------------
Envoyé avec Yahoo! Mail.
Une boite mail plus intelligente.
joshu...@gmail.com
Bruker data: (http://cars9.uchicago.edu/software/idl/detectors.html).
It is the read_siemens.pro file and although I have never read IDL
code before, I think that it corroborates this documentation on the
Bruker file format.
Also, I Googled parts of your document to try to find where it came
from, and I found a mostly verbatim copy of this document at
http://www.esc.cam.ac.uk/index.php/component/content/article/60-Facilities/229-saxi-area-detector-frame-format.
It says that it describes the "SAXI Area Detector Frame Format."
> Envoyez avec Yahoo! Mail. Une boite mail plus intelligentehttp://mail.yahoo.fr