Composites with BlackMarble background

126 views
Skip to first unread message

lobsiger...@gmail.com

unread,
Jun 6, 2022, 1:42:07 PM6/6/22
to pytroll
Hi,

I recently discovered composites "night_ir_with_background" and "night_ir_with_background_hires". These composites should automatically download respective BlackMarble Geo tif background images from:


I tried the foreground composite 'night_ir_alpha' first using MSG4 seviri data and area 'seviri_0deg'. This worked fine. When I wanted to include the BlackMarble background SatPy could not download the GEO tif files. The error message was something like "neo.sci.gsfc.nasa.gov  IS NOT  neo.gsfc.nasa.gov".  Indeed if you click on the above link you are automatically lead to a site the has no ".sci" in its URL. I changed the URL in the file .../satpy/etc/composites/visir.yaml to


and everything seemed to work. The BlackMarble GEO tif files were downloaded and cached in my user directory ./local/share/satpy/composites/. The only problem I have now is that SatPy (or pyresample?)  keeps complaining:

"Fill value incompatible with integer data using 255 instead."

Indeed the free space background is now white instead of black. Apparently the BlackMarble GEO tif has to be resampled first from "Plate Carrée" to 'seviri_0deg' and in this step a "fill_value" seems missing or is None (?). I tried to add a "fill_value=0" in three different places (composite yaml, resampling step, image saving step) without success.

How can I get a black free space background in 'seviri_0deg'?


Best Regards,
Ernst


Meteosat-11-20220606-SLO-0000-night_ir_alpha-seviri_0deg.jpg
Meteosat-11-20220606-SLO-0000-night_ir_with_background_hires-seviri_0deg.jpg

David Hoese

unread,
Jun 6, 2022, 1:58:19 PM6/6/22
to pyt...@googlegroups.com
Ernst,

A pull request updating the URL would be appreciated. It looks like they
changed the domain, provided an automatic redirect, but didn't include
the old URL in the HTTPS certificate. At least when I go to the URL in
my browser it complains about the certificate and looking at the cert
shows only the one newer domain.

As for the warning, setting `fill_value=0` in the .resample call should
have gotten rid of that. The warning is issued here:

https://github.com/pytroll/pyresample/blob/f4fb7e8a913fd98c24781a9d7a397d75e1a071db/pyresample/kd_tree.py#L1041-L1045

Maybe you could provide a small example of your code that reproduces the
issue and I could try to walk the code to figure out what's going on.

Dave
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/9fcd9477-2ae0-4737-b7ba-b16b914bb5fdn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/9fcd9477-2ae0-4737-b7ba-b16b914bb5fdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 6, 2022, 3:46:49 PM6/6/22
to pytroll
Dave,

thanks for the quick reply. The fill_value warning seems at least in part to be a red herring. When I set fill_value=0 in the resampling step then the warning disappears indeed but the image I get is just the resampled BlackMarble with again white surrounding free space. This is probably the case because 'night_ir_alpha' (the foreground) has an alpha channel to let the background shine through. Setting fill_value=0 also sets the alpha channel to 0 and my foreground is fully transparent. I even tested fill_values 100,  150,  200 and could make the ir clouds more and more reappear. Setting on the other hand fill_value=255 gives a different result from letting the warning come through and having pyresample choose fill_value=255. Surprisingly when I actively set fill_value=255 in the resampling line the free space background turns light gray. I attach DEBUG.txt for this latter case.

Regards,
Ernst
Meteosat-11-20220606-SLO-0000-night_ir_with_background_hires-seviri_0deg-fill_value_0.jpg
Meteosat-11-20220606-SLO-0000-night_ir_with_background_hires-seviri_0deg-fill_value_255.jpg
DEBUG.txt

lobsiger...@gmail.com

unread,
Jun 7, 2022, 5:21:55 AM6/7/22
to pytroll
Hi Dave,
Hi core developers,

I made a minimal script that produces the issues discussed in this thread. I attach the script as *.txt below. You can download via "wetransfer" a *.tgz that contains everything needed (including the MSG4 seviri ir-data) under the link:


This link should stay active for a week or so. Good luck.

Best regards,
Ernst
BlackMarble.txt

lobsiger...@gmail.com

unread,
Jun 7, 2022, 5:27:07 AM6/7/22
to pytroll
Sorry,

I attached the wrong *.txt. Here is the satpy script I was talking about.

Cheers,
Ernst
MSG4-BlackMarble.txt

David Hoese

unread,
Jun 7, 2022, 4:04:15 PM6/7/22
to pyt...@googlegroups.com
Thanks for all the information and the data Ernst.

I don't have a ton of time to debug the whole process, but I have a
similar but slightly different theory. The
"night_ir_with_background_hires" consists of two sub-composites:
night_ir_alpha and _night_background_hires. The _night_background_hires,
as you know, comes from a geotiff that is downloaded. This data is
loaded as an RGB with uint8 data type. The night_ir_alpha composite is
produced as an RGBA of bands from the reader and is of type 32-bit float.

So, what this means is that the Alpha band of night_ir_alpha is in the
range of 180K to 303K in your data case. The space pixels in this alpha
band are set to NaN. So when resampling handles it and is told to use 0
or 255, it isn't that these values are 0 or 255 alpha values but
actually 0 or 255 Kelvin and are then scaled during the enhancement
phase. However, this doesn't explain why the middle of the foreground
image is transparent.

I can't think of a good or easy way to resample these dependencies in
one execution. I guess one question is, what happens to the alpha band
when the two composites are merged to make the
"night_ir_with_background_hires"? You'd think even if the background had
a fill value of 255 (the default resample fill_value for this integer
data) that the alpha band from the foreground would make these
transparent...oh unless the alpha band is explicitly ignored.

Looks like there is some alpha blending going on:

https://github.com/pytroll/satpy/blob/2779675fd72826f8df166c46a1b0d59767ba720f/satpy/composites/__init__.py#L1317-L1328

Dave

On 6/7/22 04:27, lobsiger...@gmail.com wrote:
> Sorry,
>
> I attached the wrong *.txt. Here is the satpy script I was talking about.
>
> Cheers,
> Ernst
>
>
> lobsiger...@gmail.com schrieb am Dienstag, 7. Juni 2022 um 11:21:55 UTC+2:
>
> Hi Dave,
> Hi core developers,
>
> I made a minimal script that produces the issues discussed in this
> thread. I attach the script as *.txt below. You can download via
> "wetransfer" a *.tgz that contains everything needed (including the
> MSG4 seviri ir-data) under the link:
>
> https://we.tl/t-wwckA9DEPY <https://we.tl/t-wwckA9DEPY>
> "neo.sci.gsfc.nasa.gov <http://neo.sci.gsfc.nasa.gov>  IS
> > NOT neo.gsfc.nasa.gov <http://neo.gsfc.nasa.gov>".
> <https://groups.google.com/d/msgid/pytroll/9fcd9477-2ae0-4737-b7ba-b16b914bb5fdn%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/9fcd9477-2ae0-4737-b7ba-b16b914bb5fdn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/353a0199-207e-4df8-aaf8-a1a4b2dfc9a6n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/353a0199-207e-4df8-aaf8-a1a4b2dfc9a6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 7, 2022, 4:59:21 PM6/7/22
to pytroll
Hi Dave,

Thanks again for your time. Christian Peters hinted to use other resamplers. And indeed if I use resampler='gradient_search', method='bilinear' or even the plain and very slow resampler='bilinear' then I can choose the free space surrounding color for *.jpg or *.png output in the new_scene.save_dataset(...) step with fill_value=0 or 255. As a new mystery both these resamplers make the yellow/golden iluminated cities look plain white if I use resolution *_hires. If I open the BlackMarble 3km (_hires) in some other program illumination is yellow/golden as in the 0.1° (lowres) case. It also seems that the enhancement entry (that only exists for the lowres case) has no effect. I can comment it with seemingly no effect. Confusing ...

Cheers,
Ernst

David Hoese

unread,
Jun 7, 2022, 10:15:17 PM6/7/22
to pyt...@googlegroups.com
Ernst,

That enhancement thing is definitely a bug and explains why I saw
differences in some of my tests for the background image when I just
changed the resampling fill_value. The enhancement technically is a
no-op, but without it the default enhancement would be used which
dynamically scales the data between the min and max of the data. I think
the fix would be to change the "standard_name" of the _hires *composite*
to match the lower res version.

The other resamplers working makes me think that they don't support the
same handling of category products that the nearest neighbor resampler
does, but also that the nearest neighbor resampler is broken for
handling alpha bands...I think.

Dave
> <http://neo.sci.gsfc.nasa.gov <http://neo.sci.gsfc.nasa.gov>>  IS
> > > NOT neo.gsfc.nasa.gov <http://neo.gsfc.nasa.gov>
> <http://neo.gsfc.nasa.gov <http://neo.gsfc.nasa.gov>>".
> <https://groups.google.com/d/msgid/pytroll/353a0199-207e-4df8-aaf8-a1a4b2dfc9a6n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/353a0199-207e-4df8-aaf8-a1a4b2dfc9a6n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/c4b1275b-dfda-4405-a3e1-28e81dd75485n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/c4b1275b-dfda-4405-a3e1-28e81dd75485n%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 8, 2022, 10:31:03 AM6/8/22
to pytroll
Dave and All,

I made some more progress. The _hires background has now yellow/golden illumination as well.
I can even force resampler 'nearest' to produce a black free space around the 'seviri_0deg' disk.

The problems/bugs discussed here also affect a lot of other seviri/ahi/abi composites.
I attach my findings and fixes below. To look at resampler 'nearest' goes over my head.

Ernst
night_ir_with_background.txt

David Hoese

unread,
Jun 8, 2022, 3:02:34 PM6/8/22
to pyt...@googlegroups.com
Thanks for the analysis Ernst. If I understood it correctly I think
everything you suggested makes sense. There were no changes to the
generic.yaml enhancements, right?

As for the nearest resampling, this may be a sign of the limitations of
Satpy in general in that it can't handle resampling two different data
types at the same time (the uint8 data of the background and the float
data of the foreground). I had played around with creating two separate
Scenes where I resampled the background separately (with fill_value=0
specified for the background) and then added it to the other Scene. Then
I loaded the combined composite which found the background image already
loaded. However, I got weird results in the output but I think this was
due to the issues you/we found with the enhancements. I'd like to try
again with your changes included. Maybe you could put a pull request
together?

Dave
> <https://groups.google.com/d/msgid/pytroll/c4b1275b-dfda-4405-a3e1-28e81dd75485n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/c4b1275b-dfda-4405-a3e1-28e81dd75485n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/80dfeedd-12d6-4b74-9bfb-52cf61177cbdn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/80dfeedd-12d6-4b74-9bfb-52cf61177cbdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 8, 2022, 4:01:11 PM6/8/22
to pytroll
Dave and All,

NO changes to etc/enhancements/generic.yaml. But there are a couple of additional day/night composites with BlackMarble background that, as far as I understand, have NO corresponding final enhancements. I added those in a P.S. of my text. I attach images of these day/night composites for MSG seviri but they also exist for abi and ahi. No additional changes seem necessary here. But these day/night composites do also benefit from the above fixes for the night/background parts.

Regarding the problem of the white background with resampler='nearest' I just wonder what happens if we set .min instead .max in the function def _get_fill_mask_value(data_dtype). Will this breack masking composites or generate other issues?

def _get_fill_mask_value(data_dtype):
    """Return the maximum value of dtype."""
    if issubclass(data_dtype.type, np.floating):
        fill_value = np.finfo(data_dtype.type).max
    elif issubclass(data_dtype.type, np.integer):
        fill_value = np.iinfo(data_dtype.type).max
    else:
        raise TypeError('Type %s is unsupported for masked fill values' %
                        data_dtype.type)
    return fill_value



I'll think about a PR for these *.yaml files. I remember that I'll have have to fork satpy but I almost certainly have forgotten all the details of how to setup that PR :-\. We'll see ...

Ernst
night_ir_with_background_plus_PS.txt
Meteosat-11-20220606-SLO-1800-natural_enh_with_night_ir_hires-seviri_0deg-fill_value_0.jpg
Meteosat-11-20220606-SLO-0600-natural_enh_with_night_ir_hires-seviri_0deg-fill_value_0.jpg
Meteosat-11-20220606-SLO-1800-natural_color_with_night_ir_hires-seviri_0deg-fill_value_0.jpg
Meteosat-11-20220606-SLO-0600-natural_color_with_night_ir_hires-seviri_0deg-fill_value_0.jpg

David Hoese

unread,
Jun 8, 2022, 9:44:59 PM6/8/22
to pyt...@googlegroups.com
I only faintly remember the discussion Martin and I had about this fill
value stuff and I think we went with 255 for integer fields to avoid the
case where 0 means something valid in a category product (ex. cloud
type, land/sea mask, etc). In these category products there are rarely
255 categories so 255 seemed like a "safe" value to choose by default.

I think in both cases of resampling here we are getting the "expected"
outcome (with your changes included) when only considering the
individual foreground and background images, but when resampling them at
the same time we aren't. I'll keep thinking on this.

Dave
> <https://groups.google.com/d/msgid/pytroll/80dfeedd-12d6-4b74-9bfb-52cf61177cbdn%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/80dfeedd-12d6-4b74-9bfb-52cf61177cbdn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/3b3357dd-e08f-4f6d-a353-f03573f09d44n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/3b3357dd-e08f-4f6d-a353-f03573f09d44n%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 9, 2022, 8:42:47 AM6/9/22
to pytroll
Dave and All,

I have built a second BlackMarbleV2.tgz that includes data for all seviri composites that use BlackMarble
backgrounds. You can get this again with "wetransfer" where the below link should be active for one week.


Interestingly there is no problem with a black free space background for day/night combinations.
For the critical 'night_ir_with_background (_hires)' I adapted .max --> .min as proposed above and
now this also works for resampler='nearest'. I have not seen any side effect so far including my
scripts that generate NWC SAF 'cloudtype' images. I have made the satpy PR you asked for.

Cheers,
Ernst
NWC-cloudtype.png

David Hoese

unread,
Jun 9, 2022, 2:06:31 PM6/9/22
to pyt...@googlegroups.com
Ernst,

Thanks for making the PR. For the NWC SAF cloud type, it makes sense
that you wouldn't see any issues since the 0 value isn't a category in
that product. I'll talk with other satpy devs when Martin gets back from
vacation in July to brainstorm other ideas for a "perfect" solution for
this. It's possible the short-term workaround, (not including hacking
the internal code), would be to load the background in a separate Scene,
resample it, then add it to the original/SEVIRI Scene. Not great, but it
should work.

Dave

P.S. My kid's daycare is closed due to some COVID cases so I'll have
even less time than I originally thought to look at this.
> <https://groups.google.com/d/msgid/pytroll/3b3357dd-e08f-4f6d-a353-f03573f09d44n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/3b3357dd-e08f-4f6d-a353-f03573f09d44n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/f8d2e5a2-ef3e-4bf5-84e0-aa1bb55b7816n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/f8d2e5a2-ef3e-4bf5-84e0-aa1bb55b7816n%40googlegroups.com?utm_medium=email&utm_source=footer>.

lobsiger...@gmail.com

unread,
Jun 9, 2022, 4:05:48 PM6/9/22
to pytroll
Dave,

for me it's O.K. now. Most satpy users will plot the day/night combinations that have no problem anyway.
I'll run with the (integer) .max --> .min hack that -- nobody knows why -- solved my white space problem.
Thanks again for your time.

Ernst
Reply all
Reply to author
Forward
0 new messages