save.datasets() set fill_value to -9999

[nevermind]

Hi everybody,

I am exporting MOD06 datasets as geoTIFFs and I am trying to set the fill value to -9999 instead of "no data". I am using the following code, but for some reason I am still getting the same no data pixels as before:

modis.save_datasets(writer="geotiff",
dtype=np.float32,
enhance=False,
fill_value=-9999,
datasets=["cloud_effective_radius"],
filename="{name}_{start_time:%Y%m%d_%H%M%S}.tif",
base_dir="E:/Jasper/Studium/BA_Thesis/MODIS_data/fill_test")

Did I misunderstand the writer documentation (https://satpy.readthedocs.io/en/stable/api/satpy.writers.geotiff.html#satpy.writers.geotiff.GeoTIFFWriter) or could something else be the problem?

Thanks a lot!

- Jasper

[David Hoese]

Can you turn on debug messages as mentioned in this section of the
documentation:

https://satpy.readthedocs.io/en/stable/quickstart.html#troubleshooting

And paste the output of running your code here? At least at first glance
I would expect your code to work. My best guess otherwise is that the
pixels you think are invalid aren't actually invalid. It has been a
while but from a quick skim of the code I think this line of code should
be replacing any NaN values in your data with your fill value -9999:

https://github.com/pytroll/trollimage/blob/00c14a660d1be4d21c5be413c9b7d799a0ba5d0f/trollimage/xrimage.py#L1047

This is more for my future reference, but I wanted to point it out if
you're curious.

Dave

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

here is my output :

================================================

C:\Users\t22b3\miniconda\envs\my_satpy_env\python.exe E:/Jasper/Studium/BA_Thesis/Python/satpy_conda/fill_value_test.py
[DEBUG: 2022-07-12 11:42:13 : satpy.readers.yaml_reader] Reading ('C:\\Users\\t22b3\\miniconda\\envs\\my_satpy_env\\Lib\\site-packages\\satpy\\etc\\readers\\modis_l2.yaml',)
[DEBUG: 2022-07-12 11:42:13 : satpy.readers.yaml_reader] Assigning to modis_l2: ['E:/Jasper/Studium/BA_Thesis/MODIS_data/MOD06_L2.A2021299.1620.061.2021300211624.hdf']
[DEBUG: 2022-07-12 11:42:13 : satpy.composites.config_loader] Looking for composites config file modis.yaml
[DEBUG: 2022-07-12 11:42:13 : satpy.composites.config_loader] Looking for composites config file visir.yaml
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:14 : satpy.readers.hdfeos_base] Loading longitude
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:14 : satpy.readers.hdfeos_base] Interpolating from 5000 to 1000
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:14 : satpy.readers.yaml_reader] No coordinates found for DataID(name='longitude', resolution=1000, modifiers=())
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:14 : satpy.readers.hdfeos_base] Loading latitude
[DEBUG: 2022-07-12 11:42:14 : satpy.readers.yaml_reader] No coordinates found for DataID(name='latitude', resolution=1000, modifiers=())
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\dask\core.py:119: RuntimeWarning: invalid value encountered in arcsin
  return func(*(_execute_task(a, cache) for a in args))
[WARNING: 2022-07-12 11:42:15 : root] shape found from radius and resolution does not contain only integers: (2316.4616787752584, 2735.0372248527115)
Rounding shape to (2317, 2736) and resolution from (0.009043719999993982, 0.009043719999999977) meters to (0.009040537591786734, 0.009041618823294488) meters
[DEBUG: 2022-07-12 11:42:15 : satpy.scene] Resampling DataID(name='cloud_effective_radius', resolution=1000, modifiers=())
[INFO: 2022-07-12 11:42:15 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-07-12 11:42:15 : satpy.resample] Using default KDTree resampler
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:15 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2022-07-12 11:42:15 : satpy.resample] Resampling array-ef745645ef0c00097d7091570b4c4b6a
[DEBUG: 2022-07-12 11:42:15 : satpy.writers] Reading ['C:\\Users\\t22b3\\miniconda\\envs\\my_satpy_env\\Lib\\site-packages\\satpy\\etc\\writers\\geotiff.yaml']
[DEBUG: 2022-07-12 11:42:16 : satpy.writers] No enhancement being applied to dataset
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\xarray\core\pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  duck_array_version = LooseVersion(duck_array_module.__version__)
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x000002452F252650>
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Starting outermost env
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] No GDAL environment exists
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x000002452F2528F0> created
[DEBUG: 2022-07-12 11:42:16 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2022-07-12 11:42:16 : rasterio._env] PROJ_LIB found in environment.
[DEBUG: 2022-07-12 11:42:16 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000002452F2528F0>.
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x000002452F252650>
[DEBUG: 2022-07-12 11:42:16 : rasterio._io] Path: ParsedPath(path='E:/Jasper/Studium/BA_Thesis/MODIS_data/fill_test/cloud_effective_radius_20211026_162000.tif', archive=None, scheme=None), mode: w, driver: GTiff
[DEBUG: 2022-07-12 11:42:16 : rasterio._io] Option: ('COMPRESS', b'DEFLATE')
[DEBUG: 2022-07-12 11:42:16 : rasterio._io] Option: ('ZLEVEL', b'6')
[DEBUG: 2022-07-12 11:42:16 : rasterio._io] Option: ('TILED', b'TRUE')
[DEBUG: 2022-07-12 11:42:16 : rasterio._base] Nodata success: 1, Nodata value: -9999.000000
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x000002452F252650>
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x000002452F2528F0> options
[DEBUG: 2022-07-12 11:42:16 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x000002452F2528F0>.
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Exiting outermost env
[DEBUG: 2022-07-12 11:42:16 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x000002452F252650>
[INFO: 2022-07-12 11:42:16 : satpy.writers] Computing and writing results...
C:\Users\t22b3\miniconda\envs\my_satpy_env\lib\site-packages\dask\core.py:119: RuntimeWarning: invalid value encountered in arcsin
  return func(*(_execute_task(a, cache) for a in args))

Process finished with exit code 0

================================================

I also looked into the GitHub link you send me, but I honestly did not understand most of it. Or at least I was not able to run the code, since the "Scene" object has not attribute ".fillna" - I suppose I need to import another library, but was not sure, which one.

Thanks!

- Jasper

[David Hoese]

Ignore the github link. It was describing the low-level XRImage class
that satpy uses under the hood which itself uses the xarray DataArray
logic. Don't worry about it.

As for the log, I see:

[DEBUG: 2022-07-12 11:42:16 : rasterio._base] Nodata success: 1, Nodata
value: -9999.000000

So I guess the question is how do you know you aren't getting the
nodata/fill value you expect? This log message suggests that if you did
"gdalinfo your.tif" that it would show a nodata value of -9999.0.

Dave

> <https://satpy.readthedocs.io/en/stable/api/satpy.writers.geotiff.html#satpy.writers.geotiff.GeoTIFFWriter>)

>
> > or could something else be the problem?
> >
> > Thanks a lot!
> > - Jasper
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/599f5622-bba1-4c1c-a0a8-a51fbf8aca19n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/318957e2-6052-4fd1-9374-15faf42e9239n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/318957e2-6052-4fd1-9374-15faf42e9239n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[nevermind]

Hi Dave,

you were right, it turns out that the NoData value was -9999. It was just that when I loaded the TIFF in QGIS, the software automatically turned the -9999 pixels transparent and excluded them from the displayed data range, that is why I got confused - my bad.

Thanks a lot!

- Jasper