In Satpy 0.42.1 'true_color_with_night_ir_hires' doesn't work anymore

[lobsiger...@gmail.com]

Dear developers,

after updating to SatPy 0.42.1 May 1st 2023 I found that the above mentioned composite does not work anymore. Under cron control I produce hourly images from GOES16 (area caribbean), MSG3 (area westminster) and MSG2 (area afghanistan) that worked flawless with 'true_color_with_night_ir_hires' before.

In addition I find a hell of a lot of warnings about time precision when imaging MSG3/2.

# Name                    Version                   Build  Channel
satpy                     0.42.1             pyhd8ed1ab_0    conda-forge

rasterio                  1.3.6           py311h567e639_0    conda-forge

msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
python                    3.11.3          h2755cc3_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-geotiepoints       1.6.0           py311hcb2cf0a_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge

Any idea?

Cheers,

Ernst

[lobsiger...@gmail.com]

Hi,

could it be that this is related to

https://github.com/pytroll/satpy/pull/2432/commits/0b0ec99b44dcb430733b52ade33adfba473c9d32

the generic image reader that reads in the the background image with artificial illumination?

Ernst

[David Hoese]

Ernst,

> I found that the above mentioned composite does not work anymore

Could you tell us what is actually not working? Error messages?

The warnings about time precision I think is an xarray warning message but I can't be sure without seeing it.

Dave

> --
> You received this message because you are subscribed to the Google Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Hi Dave,

all error messages for the mentioned 3 images attached in my first post above. Last line:

....

ValueError: unrecognized engine rasterio must be one of: ['netcdf4', 'scipy', 'store', 'zarr']

Ernst

[Panu Lahtinen]

On 8.5.2023 6.00, lobsiger...@gmail.com wrote:
> Hi Dave,
>
> all error messages for the mentioned 3 images attached in my first post
> above. Last line:
> ....
> ValueError: unrecognized engine rasterio must be one of: ['netcdf4',
> 'scipy', 'store', 'zarr']

Try installing rioxarray package and see if that helps


P

> https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Finnish Meteorological Institute
Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
Room: 4B15a, Phone: +358 50 380 3261

[lobsiger...@gmail.com]

Panu,

thanks for that. It seems to work again. Reading PR2432 yesterday I figured it has something to do with xarray.

So I *updated* xarray to the latest but with no avail. Now I have the following  three packages in env  (pytroll):

1) rasterio

2) xarray

3) rioxarray

@Martin does PR2432 mean that rasterio is now completely obsolete (replaced by rioxarray) and

can  be deleted? Is there a problem in SatPy 0.42.1 in that rioxarray is not automatically installed?

Best regards,

Ernst

P.S.

I let it now run under cron as usual for some days to see whether everything really works again.

[Martin Raspaud]

Dear Ernst,

Thanks for reporting the issue!

rasterio is still required by rioxarray, so it can't be removed.
Moreover, rasterio is not a hard requirement for satpy, only optional,
and thus was not automatically installed in principle. So the same
applies for rioxarray...

Best regards,
Martin

> https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com>> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer>>>.

> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >

> https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> Finnish Meteorological Institute
> Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland

> Room: 4B15a, Phone: +358 50 380 3261 <tel:+358%2050%203803261>

>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Martin,

thanks for the explanation. I may be wrong but cannot remember to ever have installed rasterio as a separate package.

'true_color_with_night_ir_hires' is a nice composite that works for MSG, GOES and Himawari and is used by a couple

of people with my satpy scripts. I'll make a note about eventually missing rasterio and rioxarray in my documentation.

Remains the tons of warnings

...

/home/eumetcast/miniconda3/envs/pytroll/lib/python3.10/site-packages/satpy/readers/seviri_base.py:386: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.

...

with MSG satellites. Is this something we should care about?

Regards,

Ernst

[martin....@smhi.se]

Ernst,

The warning should now affect the results of the processing. However, it is something we need to take a look at, so it would be nice if you could create an issue for it in satpy's github.

Best regards,
Martin

> https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com>> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com>>> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer>> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/a52d80b5-657c-4ffa-ac82-a9e35eac5a1an%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.

> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "pytroll" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send
> > > an email to pytroll+u...@googlegroups.com
> > > <mailto:pytroll+u...@googlegroups.com>.
> > > To view this discussion on the web, visit
> > >
> >

> https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com>> <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/1632543f-9995-4e00-912e-9b5b9811d5f2n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.

> >
> > --
> > Finnish Meteorological Institute
> > Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
> > Room: 4B15a, Phone: +358 50 380 3261 <tel:+358%2050%203803261>
> <tel:+358%2050%203803261>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >

> https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com> <https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pytroll/ce652da2-d073-4128-a6c2-c492dff703fen%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/8bd632cf-b4d2-4852-ba5a-8be9fc804d9an%40googlegroups.com <https://groups.google.com/d/msgid/pytroll/8bd632cf-b4d2-4852-ba5a-8be9fc804d9an%40googlegroups.com?utm_medium=email&utm_source=footer>.
>

[David Hoese]

Ernst,

Sorry for missing those attachments. It looks like the conda-forge package for trollimage includes rasterio as a hard requirement:

https://github.com/conda-forge/trollimage-feedstock/blob/main/recipe/meta.yaml

which I don't remember doing, but is a happy accident in this case. This means that Satpy depends on trollimage which pulls in rasterio.

In the (near?) future, we hope to switch to rioxarray for writing geotiffs in trollimage and in that case we could update trollimage to depend on rioxarray instead of rasterio. That might solve this inconvenience on your end, but as you're probably already doing you can keep rioxarray as a package to be installed in your environment.

Dave