Re: [fiji-devel] metadata from .dm3 files

[Curtis Rueden]

Hi,

> I'm looking for an example in Python for reading metadata from a Gatan

> .dm3 file. I want to extract the pixel dimensions in x and y.

By "pixel dimensions" do you mean the X and Y pixel resolution? Or the calibration in physical space?

Either way, here is a Jython script. It works with Gatan DM3 and many other file formats.

--snip--

from java.io import File

from loci.formats import ImageReader

from loci.formats import MetadataTools

# prompt for file

od = OpenDialog("Choose a file", "")

file = File(od.getDirectory(), od.getFileName())

# parse file header

r = ImageReader()

meta = MetadataTools.createOMEXMLMetadata()

r.setMetadataStore(meta)

r.setId(file.getAbsolutePath())

# get pixel dimensions

sizeX = r.getSizeX()

sizeY = r.getSizeY()

# get physical calibration

pSizeX = meta.getPixelsPhysicalSizeX(0)

pSizeY = meta.getPixelsPhysicalSizeY(0)

# close the file

r.close()

# display results

IJ.showMessage(

  "Dimensions = " + str(sizeX) + " x " + str(sizeY) + "\n" +

  "Calibration (microns) = " + str(pSizeX) + " x " + str(pSizeY))

--snap--

Regards,

Curtis

On Wed, Feb 6, 2013 at 4:51 PM, <ngd...@gmail.com> wrote:

Hello,

I'm looking for an example in Python for reading metadata from a Gatan .dm3 file. I want to extract the pixel dimensions in x and y.

Thanks.

--
--
Please avoid top-posting, and please make sure to reply-to-all!
 
Mailing list web interface: http://groups.google.com/group/fiji-devel
 
---
You received this message because you are subscribed to the Google Groups "Fiji-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Curtis Rueden]

Hi Nuno,

> I was also looking for a flexible tool, in case I need other

> information from the metadata. I guess I can interate 'r' in your

> script for a complete list.

Also be sure to interrogate the "meta" variable, since it contains the bulk of the standardized metadata fields. The "r" variable will give you basic structural details (number of X, Y, Z, T, C, color tables, etc.) but not scientific metadata such as the type of instrument that acquired your data.

If for some reason "meta" doesn't have a method corresponding to a DM3-specific field you know exists and is present, you can probably obtain it by calling "r.getGlobalMetadata()" or "r.getSeriesMetadata()" and inspecting the returned hash for an appropriate key/value pair. But those values are format-specific (i.e., not standardized).

For further details, see:

https://www.openmicroscopy.org/site/support/bio-formats/about/

Regards,

Curtis

On Thu, Feb 7, 2013 at 6:34 AM, <ngd...@gmail.com> wrote:

Hi,

Thanks for the help!
I meant the X and Y pixel resolution.
I was also looking for a flexible tool, in case I need other information from the metadata. I guess I can interate 'r' in your script for a complete list. I'll try and figure out later.

Regards,
Nuno.

[Curtis Rueden]

Hi Nuno,

> I also want to get out of the file the (2) stage X and Y position

> which are format specific values.

Actually, stage position coordinates are one of things standardized by the OME data model. So you can access them in a format-agnostic way:

--snip--

from java.io import File

from loci.formats import ImageReader

from loci.formats import MetadataTools

# prompt for file

od = OpenDialog("Choose a file", "")

file = File(od.getDirectory(), od.getFileName())

# parse file header

r = ImageReader()

meta = MetadataTools.createOMEXMLMetadata()

r.setMetadataStore(meta)

r.setId(file.getAbsolutePath())

# get pixel dimensions

sizeX = r.getSizeX()

sizeY = r.getSizeY()

# close the file

r.close()

# display physical calibration

pSizeX = meta.getPixelsPhysicalSizeX(0)

pSizeY = meta.getPixelsPhysicalSizeY(0)

IJ.log("Dimensions = " + str(sizeX) + " x " + str(sizeY) + "\n" +

  "Calibration (microns) = " + str(pSizeX) + " x " + str(pSizeY))

# display stage positions and plane timings

planeCount = meta.getPlaneCount(0)

for p in range(planeCount):

  z = meta.getPlaneTheZ(0, p)

  c = meta.getPlaneTheC(0, p)

  t = meta.getPlaneTheT(0, p)

  posX = meta.getPlanePositionX(0, p)

  posY = meta.getPlanePositionY(0, p)

  posZ = meta.getPlanePositionZ(0, p)

  exposureTime = meta.getPlaneExposureTime(0, p)

  deltaT = meta.getPlaneDeltaT(0, p)

  IJ.log("Plane Z=" + str(z) + " C=" + str(c) + " T=" + str(t) + ":")

  IJ.log("    position=(" + str(posX) + ", " + str(posY) + ", " + str(posZ) + ")")

  IJ.log("    exposureTime=" + str(exposureTime) + ", deltaT=" + str(deltaT))

--snap--

I strongly suggest avoiding use of the "getGlobalMetadata" and "getSeriesMetadata" format-specific tables unless absolutely necessary. And if they *are* necessary for you, we would like to hear about it: anything generally useful is supposed to get standardized to the OME data model, and if that's not happening, we want to change that!

Regards,

Curtis

On Wed, Feb 13, 2013 at 11:42 AM, <ngd...@gmail.com> wrote:

Okay, reading the image dimensions in pixels is working. Next, I need to read (1) pixel resolution (how big is one pixel); in case of these .dm3 files it's in micrometers / pixel (um/px). I also want to get out of the file the (2) stage X and Y position which are format specific values.

First, I managed to read the contents of the r object like this:

meta_string = str(r.getGlobalMetadata())
f = open('C:\somepath\DM3_metadata.txt', 'w')
meta_list = meta_string.split(', ')
for item in meta_list:
    f.write(item + chr(13))
    print item

Replacing for r.getSeriesMetadata() returned {}

The result is attached. I highlighted the entries I'd like to retrieve.

(1) Pixel resolution is read correctly if I open the .dm3 file in Fiji and also appears in the list Image>Show Info...

root.ImageList.1.ImageData.Calibrations.Dimension.0.Scale = 0.019910393
root.ImageList.1.ImageData.Calibrations.Dimension.1.Scale = 0.019910393
(pixels are squares)

However, I cannot find these values (or 0.02) when I iterate through the 'r' object... I also looked up
http://hudson.openmicroscopy.org.uk/job/BIOFORMATS-trunk/javadoc/
for more methods. I found the suggested .getSizeX() and .getSizeY() under Class ImageReader. There is also one .getResolution() in the same list, but this always gives zero in my tests...
So I don't know how to retrieve the pixel resolution for my script.

I could also work with field of view (yellow highlight), but I would rather work with pixel res if possible. FoV also shows up in FIJI:

root.ImageList.1.ImageTags.Microscope Info.Field of View (µm) = 130.4847533380663

(2) The stage position is the opposite case - FIJI shows the entries but values are NaN:

root.ImageList.1.ImageTags.Microscope Info.Items.5.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.5.Label = xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Tag path = Microscope Info:xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Value = NaN
root.ImageList.1.ImageTags.Microscope Info.Items.6.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.6.Label = yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Tag path = Microscope Info:yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Value = NaN

However you can find the values in the text file, highlighted in green. My question is, what if the best way to read these values (1) and (2) in the script?

On a side note, there are other values that list as NaN in FIJI that can actually be read using these MetadataTools.

Thanks,
Nuno.

[Gregory Jefferis]

Hi Nuno,

On 13 Feb 2013, at 17:42, ngd...@gmail.com wrote:

(2) The stage position is the opposite case - FIJI shows the entries but values are NaN:

root.ImageList.1.ImageTags.Microscope Info.Items.5.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.5.Label = xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Tag path = Microscope Info:xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Value = NaN
root.ImageList.1.ImageTags.Microscope Info.Items.6.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.6.Label = yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Tag path = Microscope Info:yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Value = NaN

I wrote the original DM3_Reader plugin that is bundled with Fiji. Some people still prefer this to Bioformats in some circumstances. If you send me one of your files I can check why xPos and yPos come out as NaN, presumably a DM3_Reader bug that has not previously been noticed.

Best wishes,

Greg.

--

Gregory Jefferis, PhD                      
Division of Neurobiology                   
MRC Laboratory of Molecular Biology,       
Hills Road,                                
Cambridge, CB2 0QH, UK.                    

http://www2.mrc-lmb.cam.ac.uk/group-leaders/h-to-m/g-jefferis

http://www.neuroscience.cam.ac.uk/directory/profile.php?gsxej2
http://flybrain.stanford.edu

[Curtis Rueden]

Hi Nuno,

> and the last 3 'IJ.log' are silent... 

Sounds like Bio-Formats did not detect any stage position or timing metadata in your DM3 file. Otherwise it would have been printed out.

> By the way, I'm not familiar with 'C', 'T' and 'deltaT'; what's the difference between 'z' and 'posZ'?

C, T and Z are the dimensional positions. So e.g. "Z=3 C=0 T=5" means the fourth focal plane, first channel and sixth time point (because all values are 0-indexed).

The deltaT value is the difference in seconds since the last time point's timestamp.

The posX, posY and posZ value are the X, Y and Z stage coordinate positions respectively in the microscope's reference frame (typically microns).

> Please have a look at a .dm3 file I'm using

OK, but where can I find it?

Thanks,

Curtis

On Thu, Feb 14, 2013 at 5:03 PM, <ngd...@gmail.com> wrote:

Thanks again!

However, this new code is only displaying
Dimensions = 1500 x 1500
Calibration (microns) = 0.01418117806315422 x 0.01418117806315422

and the last 3 'IJ.log' are silent...
By the way, I'm not familiar with 'C', 'T' and 'deltaT'; what's the difference between 'z' and 'posZ'?

Please have a look at a .dm3 file I'm using (maybe it's .dm4 format already in disguise). Perhaps there is something out of place?...

Regards,
Nuno.

[Curtis Rueden]

Hi Nuno,

Thanks for the sample file. One way to inspect the metadata parsed by Bio-Formats is to use the Bio-Formats Importer plugin (File > Import > Bio-Formats) and check the "Display metadata" and "Display OME metadata" checkboxes.

The "OME Metadata" window will show the standardized metadata in XML form. Unfortunately, as you say, there are no stage coordinate positions detected with this file.

The "Original Metadata" window (also seen in Image > Show Info) displays all format-specific key/value pairs that Bio-Formats was able to extract from your file.

Do you see the values you want in the Original Metadata list? If so, what are they? Or if not, can you please tell us the exact values you expect to see?

Thanks,

Curtis

On Tue, Feb 19, 2013 at 2:59 PM, <ngd...@gmail.com> wrote:

Hi,

Thanks for the explanations.
I could swear I attached it... see if there is an attachment in this email.
I get this error #340 in Google groups.... this also happened last time. Maybe tha'ts why on the 2nd try the file was forgotten.

[Gregory Jefferis]

On 13 Feb 2013, at 18:42, ngd...@gmail.com wrote:

(2) The stage position is the opposite case - FIJI shows the entries but values are NaN:

root.ImageList.1.ImageTags.Microscope Info.Items.5.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.5.Label = xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Tag path = Microscope Info:xPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.5.Value = NaN
root.ImageList.1.ImageTags.Microscope Info.Items.6.Data Type = 6
root.ImageList.1.ImageTags.Microscope Info.Items.6.Label = yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Tag path = Microscope Info:yPos (µm)
root.ImageList.1.ImageTags.Microscope Info.Items.6.Value = NaN

However you can find the values in the text file, highlighted in green. My question is, what if the best way to read these values (1) and (2) in the script?

On a side note, there are other values that list as NaN in FIJI that can actually be read using these MetadataTools.

v1.4.2 of DM3_Reader fixes this bug (there should be no more NaNs). This is now available in master branch of fiji 

http://fiji.sc/cgi-bin/gitweb.cgi?p=fiji.git;a=commit;h=e0c4299f92649a414091e9b7a3ce20ec54a96c80

and will be available via the fiji updater whenever a new version of the IO_.jar is added (@Dscho?)

It should also be in regular ImageJ soon. Thanks, Nuno for the bug report!

[Johannes Schindelin]

Hi Greg,

On Fri, 22 Feb 2013, Gregory Jefferis wrote:

> v1.4.2 of DM3_Reader fixes this bug (there should be no more NaNs). This
> is now available in master branch of fiji
>
> http://fiji.sc/cgi-bin/gitweb.cgi?p=fiji.git;a=commit;h=e0c4299f92649a414091e9b7a3ce20ec54a96c80
>
> and will be available via the fiji updater whenever a new version of the
> IO_.jar is added (@Dscho?)

I uploaded it.

Thanks!
Dscho

[Curtis Rueden]

Hi Nuno,

> how do I know when the fixes to the OME library are

> implemented/uploaded so that the code that Curtis posted starts

> working?

Sorry for the delay in reply. We have filed a ticket to track this work:

   https://trac.openmicroscopy.org.uk/ome/ticket/10488

You have been CCed, so will receive an automated email whenever the ticket is updated. As for when the code I wrote starts working: after the ticket is closed out and the work is merged, we can upload a new version of Bio-Formats to the Fiji update site. But we do not yet have an automated process for this.

Regards,

Curtis

On Tue, Mar 5, 2013 at 10:21 AM, <ngd...@gmail.com> wrote:

I was just wondering, how do I know when the fixes to the OME library are implemented/uploaded so that the code that Curtis posted starts working?

[Curtis Rueden]

Hi Nuno,

> Maybe you could use a .dm3 file acquired in high vacuum to read the

> pressure values correctly

Thanks, I added the file to our data repository as well, and mentioned it on the ticket.

Regards,

Curtis

On Tue, Mar 5, 2013 at 11:02 AM, <ngd...@gmail.com> wrote:

Thanks. I read the the ticket. Maybe you could use a .dm3 file acquired in high vacuum to read the pressure values correctly... if so, you can find one here:
https://dl.dropbox.com/u/5200940/Prefix_3VBSED_stack_30_slice_0290.zip

Nuno.

[Nuno Dias]

Hello,

I see the ticket is closed and resolved. I didn't notice any updates in FIJI for BioFormats (and the code is still not working). So I was wondering if anything is still missing to have the updated version available in FIJI...

Thanks,
Nuno.

[Curtis Rueden]

Hi Nuno,

> I see the ticket is closed and resolved. I didn't notice any updates

> in FIJI for BioFormats (and the code is still not working).

I have pushed the latest version of Bio-Formats to the Fiji updater. Try updating again, give it a try and let us know how it goes.

Cheers,

Curtis

[Curtis Rueden]

Hi Nuno,

Unfortunately, I just learned that the version of Bio-Formats that I pushed to Fiji still does not integrate the bugfix. I will update Fiji again and let you know as soon as that changes.

Sorry,

Curtis