Caching resampler='bilinear' fails

[lobsiger...@gmail.com]

Hi All,

doing a couple of experiments to speed up image generation I also tried different resamplers. Slow resampler 'bilinear' shows huge performance increase once the

caching look up table has been made. But when I want to use it on full disk images it

fails. More exactly: It works a first time and produces the lin_lut_... as usual but

the second time, when it wants to apply the lin_lut_... it stops in error saying:

...

  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/pyresample/bilinear/xarr.py", line 131, in _finalize_output_data
    res = self._reshape_to_target_area(res, data.ndim)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/pyresample/bilinear/xarr.py", line 117, in _reshape_to_target_area
    tmp[self._valid_output_indices] = res[i, :]
AttributeError: 'XArrayBilinearResampler' object has no attribute '_valid_output_indices'

I have no problems with resampler='nearest' that produces look up tables  nn_lut...

Is this a bug or a feature of resampler='bilinear'  that cannot cache free space?

Best regards,

Ernst

[David Hoese]

Could you show us a minimal example of the code you are running that fails? I had guesses for what might be going wrong, but the error message is not what I expected so I'll need more information.

Dave

[lobsiger...@gmail.com]

Hi Dave,

Hi Developers,

I wrote an 11 liner and added data MSG2 VIS006. You can run the script once.

This will make a VIS006.png and a bil_lut *.zarr. Second time it fails. You must

delete the lut to make it rune once again. I have put everything in a *.tgz here:

https://we.tl/t-kT1B6ZMb8p

(about 9 MBytes)

Best regards

Ernst

[lobsiger...@gmail.com]

Hi All,

for those who don't like to click on links like the above (wetransfer, valid one week) here is

the script that you can use with your own data. The script works flawless with areas like

'afghanistan' or  'germ' that are eintirely on the data disk. Interestingly the script even works

 with 'scan1' that seems to have some free space points in the top left corner. But it certainly

does not work (twice) with  'msg_seviri_iodc_9km' that has full lines of free empty black space.

#!/usr/bin/python3
# -*- coding: utf-8 -*-
from glob import glob
from satpy import Scene
area = 'msg_seviri_iodc_9km'
composite='VIS006'
files = glob('./data/H-000-MSG2*')
global_scene = Scene(filenames = files, reader = 'seviri_l1b_hrit')
global_scene.load([composite])
new_scene = global_scene.resample(area, resampler='bilinear', cache_dir='./cache')
new_scene.save_dataset(composite, composite + '.png', fill_value=0)

Regards,

Ernst

Attached: Output of area = 'scan1'

[David Hoese]

Hi Ernst,

I was able to reproduce the issue on my own system with github versions
of the libraries in question. I think this is a bug in Satpy and I'll
point Panu to this thread so he can investigate it further since he made
the bilinear resampler. I think the bug is in this portion of code on Satpy:

https://github.com/pytroll/satpy/blob/bafe54e1d3c598f1d70b616e8d27bb2b25c62190/satpy/resample.py#L872-L879

I think, even after `load_bil_info` loads information from the cache,
the `get_bil_info` function still needs to be called to setup the
_valid_output_indices and other information. We'll have to wait to see
what Panu thinks.

Dave

> https://we.tl/t-kT1B6ZMb8p <https://we.tl/t-kT1B6ZMb8p>

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pytroll" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pytroll/IQPF6XjimcM/unsubscribe
> <https://groups.google.com/d/topic/pytroll/IQPF6XjimcM/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/cb44305d-a39c-46d0-8511-620789831a4en%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/cb44305d-a39c-46d0-8511-620789831a4en%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Panu Lahtinen]

Hi,

it seems that my recent fix for resampling 1D data with the bilinear
resampler broke the cache handling. Using the version just before that
one was merged works fine.

I'll see if I can find a quick fix today, after that it'll be August
after holidays when I get back to work.


P

--
Finnish Meteorological Institute
Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
Room: 4B15a, Phone: +358 50 380 3261

[lobsiger...@gmail.com]

Panu and Dave,

thanks a lot for your help.

Ernst

[Panu Lahtinen]

On 30.6.2022 9.06, Panu Lahtinen wrote:
>
> Hi,
>
> it seems that my recent fix for resampling 1D data with the bilinear
> resampler broke the cache handling. Using the version just before that
> one was merged works fine.
>
> I'll see if I can find a quick fix today, after that it'll be August
> after holidays when I get back to work.

The fix was a pretty simple one in the end:

https://github.com/pytroll/pyresample/pull/438


The big one is to add tests that catch if something breaks the
end-to-end resampling with full cache saving and loading cycle.


P