Error creating dataset with multipage tiffstack larger than 2GB

[JMPP]

I am getting an error when creating a dataset with multipage tiffstacks larger than 2.0 GB.

I am running:

  `dataset = sima.ImagingDataset(sequences, simafolder)`

And the error is:

  `File "C:...lib\site-packages\sima\imaging.py", line 142, in __init__

    str(x) for x in range(self.frame_shape[-1])]

  File "C:...lib\site-packages\sima\imaging.py", line 209, in frame_shape

    self._frame_shape = self.sequences[0].shape[1:]

  File "C:...lib\site-packages\sima\sequence.py", line 167, in shape

    return (len(self),) + self._get_frame(0).shape

  File "C:...lib\site-packages\sima\sequence.py", line 558, in __len__

    self._len = sum(1 for _ in self)

  File "C:...lib\site-packages\sima\sequence.py", line 558, in <genexpr>

    self._len = sum(1 for _ in self)

  File "C:...lib\site-packages\sima\sequence.py", line 509, in __iter__

    for _ in range(self._num_planes)], 0)

  File "C:...lib\site-packages\future\builtins\newnext.py", line 62, in newnext

    iterator.__class__.__name__))

TypeError: 'generator' object is not an iterator`

So I guess it is having issues getting the sequences shape.

If I compress the tiff file to be under 2GB (or cut frames to make it under 2GB) it works fine, but as soon as it is 2.01GB it breaks.

It is not a problem with pillow (as was discussed in a previous git issue) as I can load the tiffstack directly with `Image.open`, and pillow's limit appears to be 4-5GB anyway.

Additionally, if I load a compressed file (~1.3GB) sima can then motion correct it and exports it fine as an uncompressed 2.3GB file, so it seems to be specific to creating the initial dataset.

Compressing each tiff is incredibly time consuming so I would love to fix the error or find a work around. We also don't have a good viewer available for hd5 files yet and we need to be able to open and view the files in ImageJ, so that is also not really an option. Unfortunately I think we are stuck with making tiffs work for now.

Any help/ideas would be greatly appreciated.

Thanks,

Janelle

[Patrick Kaifosh]

Hi Janelle,

Here is some discussion related to the issue (the problem is based in
one of the underlying packages):

https://github.com/losonczylab/sima/issues/195#issuecomment-139054582

From your comments, it sounds like you may have read this already. If
the main problem is that you don't have a viewer for hdf5 files, you
could convert to hdf5, process the data, and then convert back to TIFF
again using the SIMA's export methods. Another solution is that you
could try to find a better TIFF package for python, and then modify SIMA
to use that package).

I'm also aware of some really nice software that's been written to view
SIMA datasets (including those based on HDF5) -- I'll see if I can push
to get that made public soon, but I can't guarantee anything since it's
not my code.

PK

> --
> You received this message because you are subscribed to the Google
> Groups "sima-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sima-users+...@googlegroups.com
> <mailto:sima-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sima-users/670f8173-1983-45a6-ae64-7beca2d4068e%40googlegroups.com
> <https://groups.google.com/d/msgid/sima-users/670f8173-1983-45a6-ae64-7beca2d4068e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Patrick Kaifosh]

Here's the code I mentioned for visualizing SIMA datasets: https://github.com/jbowler06/sima_visualization
It's still under development and probably only of use if you are comfortable really diving into the code base (e.g. it may have some hard-coded lab-specific stuff that would need to be altered). In the words of its creator "Anyone looking for ideas about what can be done might find it informative to checkout, but it's not quite up to a "drop in" solution right now." Perhaps in about a month it will be ready for more straightforward use.

> an email to sima-users+unsubscribe@googlegroups.com
> <mailto:sima-users+unsub...@googlegroups.com>.

[JMPP]

Hi Patrick,

Thanks very much for the info. Indeed I had seen that issue already and it doesn't seem to be the same underlying problem. I haven't found a solution yet, but I now really think it is something specific to the formatting of our tiffs (although still very strange to me that it works for our other files formatted in exactly the same way except they have a file size less than 2GB). 

I will definitely check out the visualization tool and perhaps we can get away from using the tiffs in general. That would be ideal.

Cheers,

Janelle