NWCSAF cloud data

[lobsiger...@gmail.com]

Hi,

my scripts image Metop and MSG4 NWCSAF cloud data.

The behavior of these has changed in a way which is
*confusing* to me. With satpy 0.36 I could basically
change the pictures "no_data" regions by providing a
fill_value (0 or 255). With satpy 0.40 this does only
work for MetopB/C data of "cloud_top_height". Moreover
for MetopB/C only "cloudmask_extended" and "cloudtype"
do work kind of for multiple passes using MultiScene.
Regarding selecting pixels with weights added by Adam
(PR#2275, available in satpy 0.40) and blending pixels
with weights (PR#2394, work in progress) this basically
works for Metop NWCSAF cloud data in satpy 0.36 but
fails with errors in satpy 0.40 at final image coloring.

I add four images where the top row has been processed with
satpy 0.36 while the bottom row is produced with satpy 0.40.

All full resolution images are available as *.tgz here:

https://we.tl/t-2aug19VTWH

wetransfer 246MB, above link works 1 week starting now

For my application the recent changes  are kind of a setback

though these are not vital. Will there be more changes like

PR#2403 for LEO NWCSAF cloud data in the near future?

Cheers,

Ernst

[lobsiger...@gmail.com]

Dear developers,

after my post above I realized that for all common NWCSAF composites of MSG4 and Metop-B/C the "PaletteCompositor"

has been replaced by the "SingleBandCompositor" EXCEPT for composite "cloud_top_height" that still uses the special

"CloudTopHeightCompositor". This explains the different behavior of composite "cloud_top_height". It also hinted
that this composite for LEO satellites like Metop-B/C might work in satpy 0.40 even including the new blending types.

And indeed it does as shown in the images attached (with the exception of 'stack_no_weights' that just shows the
topmost of three Metop passes). So it appears we have the following issues (and some wished enhancements) here:

Issues in satpy 0.40:
1) Simple stacking of NWCSAF Metop-B/C does not work anymore for 'cloud_top_height', 'cloud_top_pressure' and  'cloud_top_temperature'.

2) Stacking with weights as introduced by Adam in PR#2275 does not work for Metop-B/C NWCSAF composites EXCEPT for 'cloud_top_height'.

3) The use of fill_values changed and is especially unpleasant now for Metop-B/C NWCSAF 'cloudmask' and 'cloudmask_extended'.

Whish list for satpy 0.4x:
A) I would expect that NWCSAF composites with identical names work and look the same for MSG4 and Metop-B/C EUMETCast data.

B) All available composites should also work with my PR#2394 (Adam's intended use of PR#2275 was combining GEO and LEO data).

C) I would like to have the possibility to set the color of the no_data regions to black or white as before. I know that this has been discussed

before but I'm not sure whether the use of the gray palette fill values in 'cloudtype' was originally used for unclassified pixels in the data region.

Attached the output of satpy 0.40 including PR#2394. The new blend types work for composite 'cloud_top_height' only.

All other composites when using the new 'blend_types' fail in the last enhancement phase with the same "KeyError":

...
[DEBUG: 2023-03-18 09:09:55 : satpy.writers] Data for DataID(name='cloudmask_extended') will be enhanced with options:
        [{'name': 'palettize', 'method': <function palettize at 0x7fad2b29dfc0>, 'kwargs': {'palettes': [{'dataset': 'cma_extended_pal', 'color_scale': 255}]}}]
Traceback (most recent call last):
  File "/home/eumetcast/NWCSAF/clouds.py", line 112, in <module>
    new_scn.save_dataset(composite, outdir+'/'+sat+'-'+composite+'-'+area+'-'+blend_type+'-True.png',
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/scene.py", line 1138, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/writers/__init__.py", line 876, in save_dataset
    img = get_enhanced_image(dataset.squeeze(), enhance=self.enhancer, overlay=overlay,
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/writers/__init__.py", line 445, in get_enhanced_image
    enhancer.apply(img, **dataset.attrs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/writers/__init__.py", line 1227, in apply
    fun(img, *args, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/enhancements/__init__.py", line 352, in palettize
    full_cmap = _merge_colormaps(kwargs, img)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/enhancements/__init__.py", line 366, in _merge_colormaps
    cmap = create_colormap(itm, img)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/enhancements/__init__.py", line 480, in create_colormap
    cmap = _create_colormap_from_dataset(img, dataset, color_scale)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/enhancements/__init__.py", line 496, in _create_colormap_from_dataset
    match = find_in_ancillary(img.data, dataset)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/utils.py", line 702, in find_in_ancillary
    matches = [x for x in data.attrs["ancillary_variables"] if x.attrs.get("name") == dataset]
KeyError: 'ancillary_variables'



Best regards,
Ernst