Problem with NWCSAF

[Samuele Giampietro]

Hi everyone,

3 days ago, after the update to Satpy 0.52, I seen a bug with NWCSAF data from MSG3. This is the complete debug

[DEBUG: 2024-10-24 18:58:51 : satpy.readers.yaml_reader] Reading ('/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/readers/nwcsaf-geo.yaml',)
[DEBUG: 2024-10-24 18:58:51 : satpy.readers.yaml_reader] Assigning to nwcsaf-geo: ['/mnt/satserver/bas/default/S_NWC_CMA_MSG3_MSG-N-VISIR_20241024T183000Z.nc', '/mnt/satserver/bas/default/S_NWC_CT_MSG3_MSG-N-VISIR_20241024T183000Z.nc', '/mnt/satserver/bas/default/S_NWC_CTTH_MSG3_MSG-N-VISIR_20241024T183000Z.nc']
[DEBUG: 2024-10-24 18:58:51 : rasterio.session] Could not import boto3, continuing with reduced functionality.
[DEBUG: 2024-10-24 18:58:52 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2024-10-24 18:58:52 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/pyorbital/etc
[DEBUG: 2024-10-24 18:58:52 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ct.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ctth_alti.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ctth_alti_pal.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.yaml_reader] No coordinates found for DataID(name='ctth_alti_pal', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ctth_conditions.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ct_conditions.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ctth_status_flag.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ctth_quality.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ct_status_flag.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ct_pal.
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.yaml_reader] No coordinates found for DataID(name='ct_pal', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:52 : satpy.readers.nwcsaf_nc] Reading ct_quality.
[DEBUG: 2024-10-24 18:58:52 : satpy.scene] Unloading dataset: DataID(name='ct', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:52 : satpy.scene] Unloading dataset: DataID(name='ctth_alti', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='cloud_top_height', resolution=3000)
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling add-45d37e02ddfbff041831c6f9ee6c57b9
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ctth_status_flag', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ctth_status_flag
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ctth_conditions', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ctth_conditions
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ctth_quality', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ctth_quality
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='cloudtype', resolution=3000)
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling add-fbd35c44bd5e723d805d4c257979341e
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ct_status_flag', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ct_status_flag
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ct_conditions', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ct_conditions
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Resampling DataID(name='ct_quality', resolution=3000, modifiers=())
[DEBUG: 2024-10-24 18:58:53 : satpy.scene] Data reduction disabled by the user
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Read pre-computed kd-tree parameters
[DEBUG: 2024-10-24 18:58:53 : satpy.resample] Resampling ct_quality
[DEBUG: 2024-10-24 18:58:53 : satpy.writers] Reading ['/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2024-10-24 18:58:53 : satpy.writers] Adding enhancement configuration from file: /home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-10-24 18:58:53 : satpy.writers] Adding enhancement configuration from file: /home/sat/SPStools/userconfig/enhancements/generic.yaml
[DEBUG: 2024-10-24 18:58:53 : satpy.writers] Adding enhancement configuration from file: /home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2024-10-24 18:58:53 : satpy.writers] Data for DataID(name='cloud_top_height', resolution=3000) will be enhanced with options:
[{'name': 'palettize', 'method': <function palettize at 0x7f8078645620>, 'kwargs': {'palettes': [{'dataset': 'ctth_alti_pal', 'color_scale': 255}]}}]
[INFO: 2024-10-24 18:58:53 : satpy.writers] Add coastlines and political borders to image.
[DEBUG: 2024-10-24 18:58:53 : satpy.writers.simple_image] Saving to image: MSG3-cloud_top_height-euro2.png
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing FitsImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Image: failed to import FpxImagePlugin: No module named 'olefile'
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Image: failed to import MicImagePlugin: No module named 'olefile'
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing QoiImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2024-10-24 18:58:53 : PIL.Image] Importing XVThumbImagePlugin
[DEBUG: 2024-10-24 18:58:54 : PIL.PngImagePlugin] STREAM b'IHDR' 16 13
[DEBUG: 2024-10-24 18:58:54 : PIL.PngImagePlugin] STREAM b'IDAT' 41 65536
[INFO: 2024-10-24 18:58:54 : pycoast.cw_base] Using image in cache /home/sat/SPSdata/cache/cmsg3/euro2_1139740863340631669_74b1a6bb37a54872eb2a2ccf440ebb5280e26b7190c3c8d3c7bca3fe5de99197.png
[DEBUG: 2024-10-24 18:58:55 : satpy.writers] Reading ['/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2024-10-24 18:58:55 : satpy.writers] Adding enhancement configuration from file: /home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-10-24 18:58:55 : satpy.writers] Adding enhancement configuration from file: /home/sat/SPStools/userconfig/enhancements/generic.yaml
[DEBUG: 2024-10-24 18:58:55 : satpy.writers] Adding enhancement configuration from file: /home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2024-10-24 18:58:55 : satpy.writers] Data for DataID(name='cloudtype', resolution=3000) will be enhanced with options:
[{'name': 'stretch', 'method': <function stretch at 0x7f8078607060>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0, 0, 0], 'max_stretch': [1, 1, 1]}}]
[DEBUG: 2024-10-24 18:58:55 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0, 0, 0], 'max_stretch': [1, 1, 1]}
Traceback (most recent call last):
  File "/home/sat/SPStools/myscripts/MSG3-NWCSAF-area.py", line 116, in <module>
    height = geo_images(Yea, Mon, Day, Hou, Min, sat, segdir, True, isbulk, 'nwcsaf-geo', composites, area, area_cities,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/SPStools/myscripts/GEOstuff_saf.py", line 1274, in geo_images
    new_scene.save_dataset(composite, sat + '-' + composite + '-' + area +'.png', overlay={'coast_dir': geodata,
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/scene.py", line 1231, in save_dataset
    return writer.save_dataset(self[dataset_id],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/writers/__init__.py", line 883, in save_dataset
    img = get_enhanced_image(dataset.squeeze(), enhance=self.enhancer, overlay=overlay,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/writers/__init__.py", line 445, in get_enhanced_image
    enhancer.apply(img, **dataset.attrs)
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/writers/__init__.py", line 1234, in apply
    fun(img, *args, **kwargs)
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/satpy/enhancements/__init__.py", line 41, in stretch
    return img.stretch(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/trollimage/xrimage.py", line 1007, in stretch
    self.crude_stretch(**kwargs)
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/trollimage/xrimage.py", line 1116, in crude_stretch
    min_stretch = self._check_stretch_value(min_stretch, kind='min')
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/trollimage/xrimage.py", line 1137, in _check_stretch_value
    val = self.xrify_tuples(val)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/trollimage/xrimage.py", line 940, in xrify_tuples
    return xr.DataArray(tup,
           ^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/xarray/core/dataarray.py", line 479, in __init__
    coords, dims = _infer_coords_and_dims(data.shape, coords, dims)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/xarray/core/dataarray.py", line 210, in _infer_coords_and_dims
    _check_coords_dims(shape, new_coords, dims_tuple)
  File "/home/sat/miniconda3/envs/pytroll_test/lib/python3.12/site-packages/xarray/core/dataarray.py", line 140, in _check_coords_dims
    raise ValueError(
ValueError: conflicting sizes for dimension 'bands': length 3 on the data but length 1 on coordinate 'bands'

Can someone help me, please?

Best regards,

Samuele Giampietro

[Raspaud Martin]

Hi Samuele,

I haven't checked in detail the error you have, but I just want to let you know that a new release of satpy has been released in the meantime (v0.52.1) as we also found some bug for the reading of some level 2 products. Could you check if that new version still exhibits the problem you have?

If it does, please provide a minimal program and data so that we can try to reproduce it locally.

Best regards,
Martin
________________________________________
From: pyt...@googlegroups.com <pyt...@googlegroups.com> on behalf of Samuele Giampietro <samu1...@gmail.com>
Sent: 24 October 2024 21:01:46
To: pytroll
Subject: [pytroll] Problem with NWCSAF

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com<mailto:pytroll+u...@googlegroups.com>.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/0ee437a3-cd7f-4051-ba2c-22cb93fdafcbn%40googlegroups.com<https://groups.google.com/d/msgid/pytroll/0ee437a3-cd7f-4051-ba2c-22cb93fdafcbn%40googlegroups.com?utm_medium=email&utm_source=footer>.