After an upgrade from Satpy 0.44 composite 'true_color_with_night_ir_hires' fails

[lobsiger...@gmail.com]

Dear developers

For the last 2 years I have been mostly busy with my sundials and offline from PyTROLL/Satpy. I was on C and Python slowly faded away. Now I changed the broken PSU of server "Luna" and started both my combined EumetCast receivers and PyTROLL/Satpy processors Luna and "Kallisto" again.

On Luna I had Satpy 0.44 and on Kallisto I directly updated to Satpy 0.55. Then I noticed that on Kallisto all my GEO scripts with composite 'true_color_with_night_ir_hires' failed
while on Luna these still worked fine. I then updated Kallisto to Satpy 0.56 with no avail.

On Kallisto I always got errors of the kind:
"
....
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/trollimage/xrimage.py", line 1010, in _compute_quantile
    left, right = data_arr.quantile([cutoffs[0], 1. - cutoffs[1]], dim=['x', 'y'])
                                                 ~~~^~~~~~~~~~~~
TypeError: unsupported operand type(s) for -: 'float' and 'list'
"

So I downgraded Kallisto to Satpy 0.44 and 'true_color_with_night_ir_hires' worked again.

Now I incrementally updated Kallisto to Satpy 0.45, 0.46 and composite 'true_color_with_night_ir_hires' still worked fine. It stopped working though when I came to Satpy 0.47. Then my old brain finally remembered that Francis Greaves had a similar problem in 2024 we discussed on MSG-1 here:

https://groups.io/g/MSG-1/topic/104294436#msg36378

This boiled down to a difference of trollimage 1.22 versus 1.21 and Francis opened

https://github.com/pytroll/trollimage/issues/162

This lead to some fixes by Martin that possibly didn't make it into my latest Satpy?

https://github.com/pytroll/satpy/pull/2744

Now I checked my versions of trollimage and I found 1.21 on both servers Luna and Kallisto. So the problem seems to be that when you update Satpy from an old version like 0.44 to 0.46 to a recent version like Satpy 0.55 or 0.56 this will *NOT* update your trollimage that still stays with version 1.21 and then 'true_color_with_night_ir_hires' stopps working.

I do not know whether this is an ERROR or a FEATURE. But I feel that an update of Satpy
should know what trollimage version is needed. I now updated both receivers Luna and
Kallisto to the latest Satpy 0.56 and had to separately update trollimage that went from
1.21 to 1.26. Now everything is fine and 'true_color_with_night_ir_hires' works on both servers.


Best regards,
Ernst

[Martin Raspaud]

Dear Ernst,

Thanks as lot for your detailed analysis of the problem.
I have looked at how satpy defines its dependencies, and as far as I can see the minimal required version of trollimage is 1.24: https://github.com/pytroll/satpy/blob/main/pyproject.toml#L21C1-L21C24

So, as far as I understand, we are doing things right in satpy, and can only assume the problem lies in the tools that manage the updates. Could you describe a bit more how the update is performed? is it using pip, conda, mamba, uv, ...?

Best regards,
Martin
________________________________________
From: pyt...@googlegroups.com <pyt...@googlegroups.com> on behalf of lobsiger...@gmail.com <lobsiger...@gmail.com>
Sent: 29 April 2025 12:28:45
To: pytroll
Subject: [pytroll] After an upgrade from Satpy 0.44 composite 'true_color_with_night_ir_hires' fails

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com<mailto:pytroll+u...@googlegroups.com>.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/6ae0b7e3-88ea-479e-bafc-33ef79ebcfbdn%40googlegroups.com<https://groups.google.com/d/msgid/pytroll/6ae0b7e3-88ea-479e-bafc-33ef79ebcfbdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Dear Martin,

on both servers I have miniconda3 installed and I simply updated
Satpy (while in environment (pytroll)) with "conda update satpy"
from conda-forge. I downgraded with "conda install satpy=0.44" and
incrementally upgraded with "conda install satpy=0.45" etc. ...

These are the current versions of conda I used on my servers:


(base) eumetcast@kallisto:~$ conda list conda
# packages in environment at /home/eumetcast/miniconda3:
#
# Name                    Version                   Build  Channel
conda                     25.3.1          py310h06a4308_0
conda-content-trust       0.1.3           py310h06a4308_0
conda-libmamba-solver     25.3.0             pyhd3eb1b0_0
conda-package-handling    2.2.0           py310h06a4308_0
conda-package-streaming   0.9.0           py310h06a4308_0
(base) eumetcast@kallisto:~$

(base) eumetcast@kallisto:~$ cat .condarc
channels:
  - conda-forge
  - defaults
(base) eumetcast@kallisto:~$




(base) eumetcast@luna:~$ conda list conda
# packages in environment at /home/eumetcast/miniconda3:
#
# Name                    Version                   Build  Channel
conda                     23.10.0         py310h06a4308_0
conda-content-trust       0.1.3           py310h06a4308_0
conda-libmamba-solver     23.11.1         py310h06a4308_0
conda-package-handling    2.2.0           py310h06a4308_0
conda-package-streaming   0.9.0           py310h06a4308_0
(base) eumetcast@luna:~$

(base) eumetcast@luna:~$ cat .condarc
channels:
  - conda-forge
  - defaults
(base) eumetcast@luna:~$


Cheers,
Ernst

[David Hoese]

Looks like the conda-forge recipe fell out of sync with the pyproject.toml description of dependencies. It's stuck at 1.5.1:

https://github.com/conda-forge/satpy-feedstock/blob/e012e0be61e2e341f3d6d84113b2efe7772f78c4/recipe/meta.yaml#L30

I can update that when I get a chance, but without a lot of extra work the old package with the old dependencies will still exist and may be installed if a user only tried updating satpy. It really depends on what the conda dependency solver decides to do.

Dave

[lobsiger...@gmail.com]

Hi Dave,

with the conda forge recipe for Satpy 0.56 you posted, does this mean pyresample and xarray is out of sync as well?

Ernst

[David Hoese]

Yes, technically. Note that typically these version bumps are for one small piece of functionality and most existing functionality will not have changed. So while these limits on conda-forge were wrong, I would guess most of the time the older versions won't show up as an issue for you...except for this time where it did.

I'll try to pay more attention to dependency version changes during releases in the future so hopefully this doesn't happen again. I did merge an update to the conda forge package so you should be able to get access to it if conda lets you update to a new build.

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/50b4cf4a-4c2d-4aef-bb01-7c86de3d657fn%40googlegroups.com.

[lobsiger...@gmail.com]

Dave,

thanks for your work. Obviously it's not an easy task to keep Satpy
and all it's dependencies in a clean state. I noticed that even after
all my up-, down- and  upgrades of Satpy on "Kallisto" I have pyresample
1.27.1 and xarray  2023.10.1. I also had these in spring 2024 and those
are obviously from my first install of PyTROLL/satpy on this server.

I wondered whether there is an easy way to find missing or deprecated
modules compared to the list for 0.56 Martin linked to in his post above.
In the docs I found a function I never used up to now:

Troubleshooting

from satpy.utils import check_satpy
check_satpy()

I gave it a try but the output, apart from many "OKs",  it didn't even mention
pyresample or trollimage and a couple of more stuff from the top of Martin's list.

<CITE>
....

Versions
======
platform: Linux-4.19.0-12-amd64-x86_64-with-glibc2.28
python: 3.11.7

cartopy: not installed
dask: 2023.12.1
fsspec: 2023.12.2
gdal: not installed
geoviews: not installed
h5netcdf: not installed
h5py: 3.9.0
netcdf4: 1.6.4
numpy: 1.26.4
pyhdf: 0.10.5
pyproj: 3.6.0
rasterio: 1.3.7
xarray: 2023.11.0

</CITE>

Is check_satpy() deprecated or did I simply expect too much?

Regards,
Ernst

[David Hoese]

Wow really good point on those libraries not being included. I created an issue for someone to add them to the output:

https://github.com/pytroll/satpy/issues/3116

As for old xarray, I'm realizing that the version of xarray that I updated the conda-forge recipe to (and that is in pyproject.toml) is so old that it doesn't even use xarray's new calendar-based versioning scheme (ex. 2025.1.1). On Satpy's next release we should just bump the required xarray version to something more recent (2024 at least).

The easiest ways to update the dependencies are either:

1. Create a new environment from scratch. This has the major benefit of getting new versions of everything, even things that may not be direct dependencies of Satpy.

2. Do `conda update --all`. This is more likely to cause issues between packages, but conda has been getting better at detecting/handling large updates like this so that packages don't conflict or become incompatible.

3. There is a `--update-deps` for `conda update satpy` that tells it to update all dependencies if possible, but I don't think I've ever used it so I can't guarantee its effectiveness.

Dave