Save MODIS level 2 data as geoTIFF

[nevermind]

Hi everybody,

I have been using satpy to visualize different bands of the MOD06 cloud product and am now trying to save them as geoTIFFs using the geotiff writer. I am able to generate a TIFF file, but when I try to open it in the Windows file explorer, I get an error message ("file is too large or possibly damaged"). In Python, no errors are displayed. If I try the same with level 1 data (MOD02), I can open the TIFF file without any problems.

In both cases the TIFFs are not georeferenced and I get this warning in Python:

-------------------------------------------------

NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix be returned.
  s = writer(path, mode, driver=driver,

--------------------------------------------------

I believe that this is because the longitude/latitude information is stored in two separate images (or bands or datasets or however you want to call them) and not in the metadata of each band, but this does not explain, why I am unable to open the level 2 data TIFFs. I work in Pycharm using a Python 3.10 Conda environment.

Any help (for both the level 2 data and the georeferencing problem) would be amazing!

  - Jasper

[David Hoese]

Hi Jasper,

1. What L2 products are you reading/processing?
2. Are they category products and if so what is the data type of the data?
3. What is the size of the geotiff that is generated and that Windows
considers corrupt?
4. Are you passing any additional keyword arguments to Scene.save_datasets?

As for the geolocation, this is expected if you don't resample the data.
The MODIS data is a swath of non-uniform pixels and can't be
correctly/easily represented in a geotiff. Once resampled the data will
be on a uniform grid of pixels and can be represented in a geotiff.

Dave

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/29842d4b-3605-401c-96ba-f23c0e01bb95n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/29842d4b-3605-401c-96ba-f23c0e01bb95n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

1. I am working with the data set MOD06_L2.A2021299.1620.061.2021300211624.hdf and am trying to save the "cloud_effective_radius" band as a geoTIFF

2. Not entirely sure what you mean by "category products", the data type is float32

3. The size of the geoTIFF is 1.67 MB

4. My code looks like this:

----------------------------------------------------------

modis_scn.save_datasets(writer="geotiff",

                                                datasets=["cloud_effective_radius"],

                                                filename="{name}_{start_time:%Y%m%d_%H%M%S}.tif",

                                                base_dir=".../MODIS_data/output")

----------------------------------------------------------

Thanks for the advice about the geolocation, I will try to resample the data then.

  - Jasper

[David Hoese]

Jasper,

Is there any chance you could send me the geotiff? There shouldn't be
anything different as far as geotiff creation for L2 versus L1b data so
this is very confusing.

Also, do you have access to another machine (Windows or not) that you
could try opening the geotiff on? You should have GDAL installed in your
python environment as part of satpy and could try running "gdalinfo
your.tif" and if that fails then we have big problems.

Dave

> <https://groups.google.com/d/msgid/pytroll/29842d4b-3605-401c-96ba-f23c0e01bb95n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/29842d4b-3605-401c-96ba-f23c0e01bb95n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/e649185f-e2a8-43a6-9d67-2f43ed8bc0f6n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/e649185f-e2a8-43a6-9d67-2f43ed8bc0f6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

sorry for the late response! I sent the geoTIFF (it is attached to this message) to a different Windows machine via Gmail and was able to open it in the Gmail photo viewer, but after I saved it and tried to open it, I got the same error again. I tried opening the TIFF with different programs (Windows Photos, Photos, Paint etc.) but for some reason only the Gmail photo viewer worked.

Also, I have installed GDAL (it was not installed as a part of satpy) and did not get an error during the installation process, but was not able to import the package in PyCharm. It looks like it installed a folder called "GDAL-3.5.0-py3.10-win-amd64.egg-info" into my "site-packages" folder but did not create a folder called "gdal". I am usually used to packages having an info-folder and a folder that includes all the functions (e.g., "numpy-1.22.4.dist-info" and "numpy"). I manually created a "gdal" folder and filled it with the scripts I found in the "scripts" folder in the "GDAL-3.5.0-py3.10-win-amd64.egg-info" folder, which worked and allowed me to import the package into PyCharm. This "scripts" folder did not include functions like gdalinfo, though, so it looks like I only managed to get parts of GDAL running - not sure how to fix this.

Still, I managed to find the gdalinfo function somewhere in the "pkgs" folder and run it in the OSGeo4W Shell (Anaconda shell could not find the gdalinfo function). I got the following output:

==================================================

Driver: GTiff/GeoTIFF
Files: E:\Jasper\Studium\BA_Thesis\MODIS_data\output\cloud_effective_radius_20211026_162000.tif
Size is 1354, 2030
Metadata:
  TIFFTAG_DATETIME=2021:10:26 16:20:00
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2030.0)
Upper Right ( 1354.0,    0.0)
Lower Right ( 1354.0, 2030.0)
Center      (  677.0, 1015.0)
Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Alpha

===============================================

Thanks again for your help!

  Jasper

[nevermind]

Update: it is possible to open the geoTIFF in QGIS 3.10.9. I still don't understand why it is not working in the photo viewer, though. The MODIS l1b data is not causing any problems.

[David Hoese]

As for GDAL, I was hoping it would have been installed as part of a
satpy conda installation. If it wasn't I would hope that "conda install
-c conda-forge gdal" would have given you everything you needed in your
conda environment. Doing it through PyPI is another story. It sounds
like you somehow got it working so I guess that's fine.

My first guess was going to be that the geotiff generated was big and
would be considered a "BigTIFF", but it seem pretty small.

My next guess was that maybe it was accidentally saved as a non-8-bit
integer type, but it looks like it is.

The next step may be to do gdalinfo for both a working l1b geotiff and
compare it to the non-working l2 geotiff.

Dave

> <https://groups.google.com/d/msgid/pytroll/e649185f-e2a8-43a6-9d67-2f43ed8bc0f6n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/e649185f-e2a8-43a6-9d67-2f43ed8bc0f6n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/224d3d8b-4068-46cb-b0c0-c4fbb45f9755n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/224d3d8b-4068-46cb-b0c0-c4fbb45f9755n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

Here is the output for the MODIS l1b TIFF:

======================================

Driver: GTiff/GeoTIFF
Files: E:\Jasper\Studium\BA_Thesis\MODIS_data\output\natural_color_20060117_085500.tif

Size is 1354, 2030
Metadata:

  TIFFTAG_DATETIME=2006:01:17 08:55:00

Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2030.0)
Upper Right ( 1354.0,    0.0)
Lower Right ( 1354.0, 2030.0)
Center      (  677.0, 1015.0)

Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA
Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha

=======================================

And here for the MODIS l2 TIFF again:

=======================================

Driver: GTiff/GeoTIFF
Files: E:\Jasper\Studium\BA_Thesis\MODIS_data\output\cloud_effective_radius_20211026_162000.tif
Size is 1354, 2030
Metadata:
  TIFFTAG_DATETIME=2021:10:26 16:20:00
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2030.0)
Upper Right ( 1354.0,    0.0)
Lower Right ( 1354.0, 2030.0)
Center      (  677.0, 1015.0)
Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Alpha

=======================================

- Jasper

[David Hoese]

Well they look pretty similar. Can you try saving an L1b single band
('1') instead of an RGB and try opening that? I wonder if your image
viewer doesn't like LA (Luminance + Alpha) tiffs.

Dave

> <https://groups.google.com/d/msgid/pytroll/224d3d8b-4068-46cb-b0c0-c4fbb45f9755n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/224d3d8b-4068-46cb-b0c0-c4fbb45f9755n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/5d259adb-e65f-46a8-9861-ce2937da2e15n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/5d259adb-e65f-46a8-9861-ce2937da2e15n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

I tried this and now I am unable to open this geoTIFF as well - just like with the level 2 data. Do you know of any fix for this? gdalinfo looks normal here as well:

==================================

Driver: GTiff/GeoTIFF
Files: E:\Jasper\Studium\BA_Thesis\MODIS_data\output\1_20060117_085500_1band.tif

Size is 1354, 2030
Metadata:
  TIFFTAG_DATETIME=2006:01:17 08:55:00
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2030.0)
Upper Right ( 1354.0,    0.0)
Lower Right ( 1354.0, 2030.0)
Center      (  677.0, 1015.0)

Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Alpha

===================================

Either way, thanks a lot for all the advice! I will move to working on the georeferencing issue then!

  - Jasper

[David Hoese]

Jasper,

I'm not really sure what's going on. I haven't used a Windows computer
for ~12 years so I don't know what their image viewers should be able to
do as far as viewing TIFF images. You could use `fill_value=0` when
generating this images (save_datasets) to make all transparent/invalid
pixels equal to 0 (black) which would also not creating the alpha band
in the geotiffs. Otherwise, I'm not sure.

Dave

> <https://groups.google.com/d/msgid/pytroll/5d259adb-e65f-46a8-9861-ce2937da2e15n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/5d259adb-e65f-46a8-9861-ce2937da2e15n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/9f9c0ab9-4169-4a18-abda-b7756cd552dcn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/9f9c0ab9-4169-4a18-abda-b7756cd552dcn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

Perfect, this solved the issue! Again, thanks a lot!

  - Jasper