Composites with NOAA20 VIIRS

[lobsiger...@gmail.com]

Hi,

I currently have speeded up my GEO scripts by loading a

list of composites and only reading (decompressing) the
channel datasets needed. This works fine now and I have
turned to my LEO scripts. Here all channels are in every
segment file and it was only for curiosity that I wanted
to know which composite uses what datasets. Maybe there
is a much easier way but I do the following steps.

1) I make a scene with all datasets and print out a list
   using print(scene.available_composite_names())
2) Using above list I generate all possible pictures
   with debug(ON) and capture the datasets used from
   the python text output with grep and another script.

For NOAA20 VIIRS I took all (EUMETCast) 16 MXY channels and
the DNB channel of a night pass. This gave me the following:

viirs_dict = { \
    # Single viirs channels
    'DNB': [DNB],
    'M01': [M01],
    'M02': [M02],
    'M03': [M03],
    'M04': [M04],
    'M05': [M05],
    'M06': [M06],
    'M07': [M07],
    'M08': [M08],
    'M09': [M09],
    'M10': [M10],
    'M11': [M11],
    'M12': [M12],
    'M13': [M13],
    'M14': [M14],
    'M15': [M15],
    'M16': [M16],
    # Defined viirs composites
    'adaptive_dnb': ['DNB'],
    'ash': ['M14', 'M15', 'M16'],
    'cloudtop_daytime': ['M12', 'M15', 'M16'],
    'dust': ['M14', 'M15', 'M16'],
    'dynamic_dnb': ['DNB'],
    'false_color': ['M05', 'M07', 'M11'],
    'fire_temperature': ['M10', 'M11', 'M12'],
    'fire_temperature_39refl': ['M10', 'M11', 'M12', 'M15'],
    'fire_temperature_awips': ['M10', 'M11', 'M12'],
    'fire_temperature_eumetsat': ['M10', 'M11', 'M12'],
    'fog': ['M14', 'M15', 'M16'],
    'green_snow': ['DNB', 'M10', 'M15'],
    'histogram_dnb': ['DNB'],
    'hncc_dnb': ['DNB'],
    'ir108_3d': ['M15'],
    'ir_cloud_day': ['M15'],
    'natural_color': ['M05', 'M07', 'M10'],
    'natural_color_raw': ['DNB', 'M07', 'M10'],
    'natural_color_sun_lowres': ['M05', 'M07', 'M10'],
    'natural_enh': ['DNB', 'M10'],
    'natural_with_night_fog': ['M05', 'M07', 'M10', 'M12', 'M15', 'M16'],
    'night_fog': ['M12', 'M15', 'M16'],
    'night_microphysics': ['DNB', 'M12', 'M15'],
    'night_overview': ['DNB', 'M15'],
    'ocean_color': ['M03', 'M04', 'M05'],
    'overview': ['M05', 'M07', 'M15'],
    'overview_raw': ['DNB', 'M15'],
    'snow_age': ['M07', 'M08', 'M09', 'M10', 'M11'],
    'snow_lowres': ['M07', 'M10', 'M12', 'M15'],
    'true_color': ['M03', 'M04', 'M05'],
    'true_color_crefl': ['M03', 'M04', 'M05'],
    'true_color_lowres': ['M03', 'M04', 'M05'],
    'true_color_lowres_crefl': ['M03', 'M04', 'M05'],
    'true_color_lowres_land': ['M03', 'M04', 'M05'],
    'true_color_lowres_marine_tropical': ['M03', 'M04', 'M05'],
    'true_color_raw': ['M03', 'M04', 'M05']
    }

Most of the night images were all black of course but what strikes me is:

'green_snow': ['DNB', 'M10', 'M15'],                                              (all black)
'natural_enh': ['DNB', 'M10'],                                                          (all black)
'night_microphysics': ['DNB', 'M12', 'M15'],                           (some green)
'overview_raw': ['DNB', 'M15'],     (maybe unenhanced 'night_overview'?)

These composites are not available for day passes (with MXY channels only)
and do not make much sense at night. Is this a satpy bug or a (VIIRS) feature?

Regards,
Ernst

[David Hoese]

Ernst,

I think some of these are being pulled in from the "shared" list of
generic composites and don't make much sense for VIIRS. For example, I
looked at the "green_snow" RGB. It isn't actually defined in the
VIIRS-specific composites here:

https://github.com/pytroll/satpy/blob/main/satpy/etc/composites/viirs.yaml

But it is defined in the parent/inherited composites from visir.yaml
(Visible/IR composites):

https://github.com/pytroll/satpy/blob/15dbf3db78b0f23aa2c8b1acda458212dfca0e84/satpy/etc/composites/visir.yaml#L203-L209

You can see in visir.yaml that green snow depends on a 1.63um input.
Satpy seems to be finding DNB for that, but the "raw" DNB data isn't
really useful as an image without some pre-processing/normalization. So
I think overall this is a "bug" for VIIRS, but I'm not really sure what
the best way to fix it is.

As for optimizing loading of VIIRS data, I think there are some things
that could be changed in the VIIRS reader/file handlers that could help
speed up and/or delay the opening and loading of the file content. If
your current log capture way is "good enough" for you then stick with
that. I know some other people have tried parsing the YAML files
directly (or you could use the lower-level functions for loading them
inside Satpy) and then parse the information contained there. Overall
not the easiest thing so the longer term reader optimizations are
something I look forward to working on...eventually.

Dave

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/22480e23-b1a6-4c7e-8f7e-89817d461999n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/22480e23-b1a6-4c7e-8f7e-89817d461999n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Dave,

regarding "green_snow" it seems to be the 0.635 that DNB is taken
for. Of course satpy cannot know that DNB is only available at
night. So my method to find the datasets used seems to work for all
composites found by scene.available_dataset_names() but VIIRS with
DNB is a special case and not all composites taken from visir.yaml
(or even found in private viirs/visir definitions!) are granted to make
sense. I have no problems with the speed of the reader though.

No action needed! I can use my results to prevent users from asking
for composites using DNB in day passes. Or at night I can search for
DNB/MXY segment files only if those are really needed in composites.

Thanks for your help.

Best regards,
Ernst


visir.yaml:

  green_snow:
    compositor: !!python/name:satpy.composites.GenericCompositor
    prerequisites:
    - 1.63
    - 0.635
    - 10.8
   standard_name: green_snow


The DEBUG: output I used after "| grep WavelengthRange"
(The first line is my marker printed with flush=True) :

green_snow green_snow WavelengthRange
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Resampling DataID(name='DNB', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=743, calibration=<calibration.radiance>, modifiers=())
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Resampling DataID(name='M10', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=742, calibration=<calibration.reflectance>, modifiers=())
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Resampling DataID(name='M15', wavelength=WavelengthRange(min=10.263, central=10.763, max=11.263, unit='µm'), resolution=742, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Unloading dataset: DataID(name='DNB', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=743, calibration=<calibration.radiance>, modifiers=())
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Unloading dataset: DataID(name='M10', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=742, calibration=<calibration.reflectance>, modifiers=())
[DEBUG: 2022-07-19 11:21:29 : satpy.scene] Unloading dataset: DataID(name='M15', wavelength=WavelengthRange(min=10.263, central=10.763, max=11.263, unit='µm'), resolution=742, calibration=<calibration.brightness_temperature>, modifiers=())

[David Hoese]

Sorry for the late reply. I'm just getting back from being sick all last
week.

The DNB (Day/Night Band) is available at day time. I'm not sure if you
were saying these composites weren't available because of the bands
available on EUMETCAST at day time. What I was trying to say in my
previous email is that due to the high dynamic range of the DNB, it is
almost never a good idea for it to be used "raw" with no normalization
done to it. This is/was my guess for why all the images you pointed out
were black.

If what you were saying is that there are composites being shown as
"available" when the dependency channels are not available, then this is
likely a limitation of the VIIRS compact reader (which I assume you're
using) and that should probably be added. The reader may be saying that
a band is available when it isn't actually available.

Dave

> <https://groups.google.com/d/msgid/pytroll/22480e23-b1a6-4c7e-8f7e-89817d461999n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/22480e23-b1a6-4c7e-8f7e-89817d461999n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/6f9e3d16-ce07-4b49-bfd3-ee562ffc5ab5n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/6f9e3d16-ce07-4b49-bfd3-ee562ffc5ab5n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Dave,

before all I wish you good health. The viirs problems I had
seem to be wide spread. YES, I'm using the 'viirs_compact'
reader but DNB channels are only available at *night*. They
show moonlight, auroras and artificial city illumination.
That's at least what I get from EUMETCast: During night
passes I have both DNB + MXY bands. In daylight only MXY.

Therefore viirs 'green_snow', which is a daylight composite,
cannot work if it replaces the 0.635 channel with DNB 0.7.

It's indeed a fact that you have to look twice what report

scn.avaliable_dataset_names()
scn.available_composite_names()

and what data you actually have and what composites you can build.
Here is e.g. what 'modis_l1b' told me when I loaded the EUMETCast
distributed thin_MOD021KM*.hdf files for satellite EOS Aqua/Terra:

# Available channel datasets found in summer 2022 from print(scn.available_dataset_names())
# *****************************************************************************************
# ['1', '10', '11', '12', '13hi', '13lo', '14hi', '14lo', '15', '16', '17', '18', '19', '2',
# '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '3', '30', '31', '32', '33',
# '34', '35', '36', '4', '5', '6', '7', '8', '9']

# Available composite names found in summer 2022 from print(scn.available_composite_names())
# ******************************************************************************************
# ['airmass', 'ash', 'day_microphysics', 'dust', 'fog', 'green_snow', 'ir108_3d', 'ir_cloud_day',
# 'natural_color', 'natural_color_raw', 'natural_with_night_fog', 'night_fog', 'ocean_color',
# 'overview', 'snow', 'true_color', 'true_color_crefl', 'true_color_thin', 'true_color_uncorrected']

I had to find out the hard way that I really had this list of datasets available in my loaded thin_MOD021KM*.HDF:
channels = ['1', '2', '5', '6', '8', '9', '10', '12', '15', '20', '23', '26', '27', '28', '29', '31', '32', '33']

And these composites finally worked while others like 'airmass', 'true_color', ... simply crashed:
composites = ['ash', 'day_microphysics', 'dust', 'fog', 'green_snow', 'ir108_3d', 'ir_cloud_day','natural_color',
              'natural_color_raw', 'natural_with_night_fog', 'night_fog', 'overview', 'snow', 'true_color_thin']

I had similar problems with Metop AVHRR using the reader 'avhrr_l1b_eps'. As you might know

the AVHRR sensor has channels  ['1', '2', '3a', '3b', '4', '5'] where 3a is only ON in daylight
while 3b is only ON during the night. Channels 3a and 3b are NEVER disseminated at the same time.
But method scn.available_composite_names() gives me things like composite 'day_microphysics'
(using ['2', '3b', '4']) and  'snow' (using ['2', '3a', '3b', '4']) which is of course plain BS.

It seems said methods report what datasets the reader is capable to read and what composites
could be combined that way. What datasets are actually available seems to be another story.
Again I don't think anything has to be done. Or should these kind of glitches never happen?


Best regards,

Ernst

P.S.

The documentation hints that some readers can dynamically look what's available though.

That said the 'modis_l1b' reader might just have been the worst example Murphy could find.

[David Hoese]

Ok, thanks Ernst. I think I understand. I'm not really familiar with
EUMETCast's offerings so thanks for that info.

The way it is supposed to work when you ask for
"available_composite_names" is that the Scene checks its readers for
what datasets are available from the readers and then determines what
composites dependencies are "known" (the reader knows what it is, but it
may or may not be available in the currently loaded files) versus which
ones are "available" (the reader knows it can give you the data for that
product...unless something unexpected happens). By default, with no
extra work by the creator of the reader, a reader will consider a
dataset "available" if the file that that dataset comes from has been
loaded (this is "file_type" in the reader YAML). In some readers, the
behavior is customized to check for things like the exact variable being
in the file or checking that the exact resolution (ex 250m) being
available in the file. The VIIRS compact and MODIS L1B/L2 readers both
use the basic functionality. The "viirs_sdr" reader is one example of a
reader that does the fancier technique (mostly because of how
complicated that file format can be with NOAA CLASS ordering options).

So, for this available composite stuff to be more accurate for these
readers, the python code in the readers would need to be updated to
define the "available_datasets" method (as described in the custom
reader documentation) to be smarter about whether or not a dataset is
available from the loaded files.

Dave

> <https://groups.google.com/d/msgid/pytroll/6f9e3d16-ce07-4b49-bfd3-ee562ffc5ab5n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/6f9e3d16-ce07-4b49-bfd3-ee562ffc5ab5n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/82e8a5c0-b318-4d6a-b43d-625e8b1aaa2fn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/82e8a5c0-b318-4d6a-b43d-625e8b1aaa2fn%40googlegroups.com?utm_medium=email&utm_source=footer>.