Testing three resamplers for LEO satellites
34 views
Skip to first unread message
lobsiger...@gmail.com
Sep 11, 2022, 8:40:44 AM9/11/22
to pytroll
Dear All,
I have tested three resamplers with LEO satellites. Only 'nearest'
always behaved as expected. Multiscene seems to be a major problem.
Area 'westminster', DAY single pass, Date 2022/09/10
All satellites use the same composite 'natural_color'
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
-----------------+-----------+------------+------------------
MetopB (avhrr) | O.K. | O.K. | O.K.
-----------------+-----------+------------+------------------
NOAA20 (viirs) | O.K. | bowtie** | fractured
-----------------+-----------+------------+------------------
FY3D (mersi2) | O.K. | O.K. | fractured
-----------------+-----------+------------+------------------
Aqua (MODIS) | O.K. | unusable | interference
-----------------+-----------+------------+------------------
** Bowtie cannot be fixed with radius_of_influence=20000
Area 'westminster', DAY multi passes, Date 2022/09/10
All satellites use the same composite 'natural_color'
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
-----------------+-----------+------------+------------------
MetopB (avhrr) | O.K. | O.K. | O.K.
-----------------+-----------+------------+------------------
NOAA20 (viirs) | O.K. | bowtie** | fractured
-----------------+-----------+------------+------------------
FY3D (mersi2) | O.K. | O.K. | fractured
-----------------+-----------+------------+------------------
Aqua (MODIS) | O.K. | unusable | fractured
-----------------+-----------+------------+------------------
** Bowtie cannot be fixed with radius_of_influence=20000
All satellites use the same composite 'natural_color'
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
-----------------+-----------+------------+------------------
MetopB (avhrr) | O.K. | O.K. | O.K.
-----------------+-----------+------------+------------------
NOAA20 (viirs) | O.K. | bowtie** | fractured
-----------------+-----------+------------+------------------
FY3D (mersi2) | O.K. | O.K. | fractured
-----------------+-----------+------------+------------------
Aqua (MODIS) | O.K. | unusable | interference
-----------------+-----------+------------+------------------
** Bowtie cannot be fixed with radius_of_influence=20000
Area 'westminster', DAY multi passes, Date 2022/09/10
All satellites use the same composite 'natural_color'
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
-----------------+-----------+------------+------------------
MetopB (avhrr) | O.K. | O.K. | O.K.
-----------------+-----------+------------+------------------
NOAA20 (viirs) | O.K. | bowtie** | fractured
-----------------+-----------+------------+------------------
FY3D (mersi2) | O.K. | O.K. | fractured
-----------------+-----------+------------+------------------
Aqua (MODIS) | O.K. | unusable | fractured
-----------------+-----------+------------+------------------
** Bowtie cannot be fixed with radius_of_influence=20000
You can download all generated images from wetransfer (link valid 1 week):
Best regards
Ernst
David Hoese
Sep 11, 2022, 2:50:15 PM9/11/22
to pyt...@googlegroups.com
Hi Ernst,
Would it be possible for you to include some of these images as
attachments (does Google Groups allow inline images?) to your emails?
Having to go to a separate site, download a compressed archive, then
extract it, and finally open the files makes it difficult to start
debugging these types of emails. Also, could you provide more detail in
your email about the issues you are seeing? The tables are really nice
and display what results you're seeing, but what does "unusable" mean in
terms of an image? What does "fractured" mean? interference? Could you
include in the email what your `Scene.resample` calls look like for each
of these?
For the MultiScene execution, are you saying that the results are
different between using a single Scene and the MultiScene even for the
same time periods? My gut reaction is that that shouldn't be possible if
you're calling MultiScene.save_datasets as it calls the Scene methods
underneath. How is the MultiScene created?
Sorry for asking for so much extra information. My hope when opening a
support email is to get as much information from the email as quickly as
I can so I can easily decide whether to answer it now or put it on the
TODO list as something that needs more dedicated time to answer.
Downloading images, scripts, and log files from a remote site makes it
much more likely that I put it in the long term TODO queue.
Dave
On 9/11/22 07:40, lobsiger...@gmail.com wrote:
> *Dear All,*
> *
> *
> *I have tested three resamplers with LEO satellites. Only 'nearest'*
> *always behaved as expected. Multiscene seems to be a major problem.*
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com?utm_medium=email&utm_source=footer>.
Would it be possible for you to include some of these images as
attachments (does Google Groups allow inline images?) to your emails?
Having to go to a separate site, download a compressed archive, then
extract it, and finally open the files makes it difficult to start
debugging these types of emails. Also, could you provide more detail in
your email about the issues you are seeing? The tables are really nice
and display what results you're seeing, but what does "unusable" mean in
terms of an image? What does "fractured" mean? interference? Could you
include in the email what your `Scene.resample` calls look like for each
of these?
For the MultiScene execution, are you saying that the results are
different between using a single Scene and the MultiScene even for the
same time periods? My gut reaction is that that shouldn't be possible if
you're calling MultiScene.save_datasets as it calls the Scene methods
underneath. How is the MultiScene created?
Sorry for asking for so much extra information. My hope when opening a
support email is to get as much information from the email as quickly as
I can so I can easily decide whether to answer it now or put it on the
TODO list as something that needs more dedicated time to answer.
Downloading images, scripts, and log files from a remote site makes it
much more likely that I put it in the long term TODO queue.
Dave
On 9/11/22 07:40, lobsiger...@gmail.com wrote:
> *Dear All,*
> *
> *
> *I have tested three resamplers with LEO satellites. Only 'nearest'*
> *always behaved as expected. Multiscene seems to be a major problem.*
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com?utm_medium=email&utm_source=footer>.
lobsiger...@gmail.com
Sep 11, 2022, 4:35:05 PM9/11/22
to pytroll
Hi Dave,
on the google list are considered to be misuse. I minimized some
problematic images and inline those below. My comment re multipass
multiscene was misleading. I think we basically see the same patterns
as with single pass images but of course twice or three times which
often makes things look worse. My three resampling lines were:
1) resampler='nearest', radius_of_influence=3000
2) resampler='bilinear', reduce_data=True
3) resampler='gradient_search', method='bilinear', reduce_data=True
I used the term interference for a Moiré like look
I used fractured for different pieces in unexpected places
I used bowtie for the bowtie effect known from viirs swath images
This NOAA-20 multipass viirs image was resampled 'gradient_search'. That's what I called fractured.
sorry, to use zipped wetransfers here seems not to be a good idea.
My images are rather big and TBH I don't know what image sizeson the google list are considered to be misuse. I minimized some
problematic images and inline those below. My comment re multipass
multiscene was misleading. I think we basically see the same patterns
as with single pass images but of course twice or three times which
often makes things look worse. My three resampling lines were:
1) resampler='nearest', radius_of_influence=3000
2) resampler='bilinear', reduce_data=True
3) resampler='gradient_search', method='bilinear', reduce_data=True
I used the term interference for a Moiré like look
I used fractured for different pieces in unexpected places
I used bowtie for the bowtie effect known from viirs swath images
None of the images that are not marked O.K. are usable in any way
Unfortunately inline images seem to hide their names. I start with a MODIS multipass resampled 'nearest' to show you what I expect and name O.K.
This is the same MODIS multipass image as above but resampled 'bilinear'. This is what I called 'interference' or 'Moiré'.
Here is a single pass viirs image resampled 'bilinear' that clearly shows the bowtie effect (data reduction towards the limb) normally only seen in swath images.
I noted that the 'bilinear' resampler also has a parameter 'radius_of_influence'. I set 'radius_of_influence'=10000 and later 20000 to no avail.
This NOAA-20 multipass viirs image was resampled 'gradient_search'. That's what I called fractured.
This is the same NOAA-20 viirs image resampled 'gradient_search' as one further up but this time only single pass. Fractured again ...
Hope this helps,
Ernst
lobsiger...@gmail.com
Sep 11, 2022, 7:53:30 PM9/11/22
to pytroll
Dave,
I just noted that with my most simple script that I published in thread "Trouble with MODIS" I can image 'true_color' with 'bilinear' O.K.
The difference is that there I had Terra data and the above problems use Aqua data and my full featured script with all 'features'.
I will investigate Terra data with 'bilinear' with my full featured script tomorrow. I will also try Aqua with my most simple script.
Stay tuned,
Ernst
lobsiger...@gmail.com
Sep 12, 2022, 8:33:10 AM9/12/22
to pytroll
Dave and All,
this is slowly driving me crazy. What started as "Trouble with MODIS"
ended as multiple problems with resamplers and now I'm back to MODIS
again and indeed see that thin MODIS Aqua data behaves differently
from thin MODIS Terra data! I tested that first with my full featured
scripts that use a 50KB LEOstuff.py module (which is suspect from
its sheer size). I then picked out two Aqua segment files and tested
with my very short version of the MODIS script. That's what I get:
Area 'isleofman' (UK), DAY single pass, composites -> generate=True
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
------------------------+-----------+------------+------------------
Terra 'natural_color' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Terra 'true_color_thin' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Aqua 'natural_color' | O.K. | unusable°° | unusable°°
------------------------+-----------+------------+------------------
Aqua 'true_color_thin' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Area 'isleofman' (UK), DAY single pass, composites -> generate=False
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
------------------------+-----------+------------+------------------
Terra 'natural_color' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Terra 'true_color_thin' | ugly** | O.K.++ | many scratches
------------------------+-----------+------------+------------------
Aqua 'natural_color' | O.K. | unusable°° | unusable°°
------------------------+-----------+------------+------------------
Aqua 'true_color_thin' | ugly** | O.K.++ | many scratches
------------------------+-----------+------------+------------------
°° Only parallel scratches all over the place
** Produces ugly pink stains apparently from filler channel 10
++ Only possibility to get 'true_color_thin' O.K. with generate=False
You can download all above images from my receiver/processor Luna:
You can download a *.tgz that has the Aqua/Terra data and the
short scripts that should let you reproduce all above images.
The *.tgz is 300MB, my Uplink is 100MBit/s, maybe it takes a while.
I cannot think of an other way except for not welcome wetransfer.com.
I'll let those links open for some days but will close them as soon
as google plus all the script kiddies and secret services move in ...
Best regards,
Ernst
P.S. Please let me know if there is any problem accessing Luna
this is slowly driving me crazy. What started as "Trouble with MODIS"
ended as multiple problems with resamplers and now I'm back to MODIS
again and indeed see that thin MODIS Aqua data behaves differently
from thin MODIS Terra data! I tested that first with my full featured
scripts that use a 50KB LEOstuff.py module (which is suspect from
its sheer size). I then picked out two Aqua segment files and tested
with my very short version of the MODIS script. That's what I get:
Area 'isleofman' (UK), DAY single pass, composites -> generate=True
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
------------------------+-----------+------------+------------------
Terra 'natural_color' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Terra 'true_color_thin' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Aqua 'natural_color' | O.K. | unusable°° | unusable°°
------------------------+-----------+------------+------------------
Aqua 'true_color_thin' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Area 'isleofman' (UK), DAY single pass, composites -> generate=False
Resampler | 'nearest' | 'bilinear' | 'gradient_search'
------------------------+-----------+------------+------------------
Terra 'natural_color' | O.K. | O.K. | many scratches
------------------------+-----------+------------+------------------
Terra 'true_color_thin' | ugly** | O.K.++ | many scratches
------------------------+-----------+------------+------------------
Aqua 'natural_color' | O.K. | unusable°° | unusable°°
------------------------+-----------+------------+------------------
Aqua 'true_color_thin' | ugly** | O.K.++ | many scratches
------------------------+-----------+------------+------------------
°° Only parallel scratches all over the place
** Produces ugly pink stains apparently from filler channel 10
++ Only possibility to get 'true_color_thin' O.K. with generate=False
You can download all above images from my receiver/processor Luna:
You can download a *.tgz that has the Aqua/Terra data and the
short scripts that should let you reproduce all above images.
The *.tgz is 300MB, my Uplink is 100MBit/s, maybe it takes a while.
I cannot think of an other way except for not welcome wetransfer.com.
I'll let those links open for some days but will close them as soon
as google plus all the script kiddies and secret services move in ...
Best regards,
Ernst
P.S. Please let me know if there is any problem accessing Luna
David Hoese
Sep 12, 2022, 10:01:35 AM9/12/22
to pyt...@googlegroups.com
Ernst,
It isn't that wetransfer isn't welcome. It is that the information isn't
in the email. I understand that for this particular problem that may not
be possible. A lot of the core Satpy/pyresample devs are either on
holiday or getting back from holiday or getting ready to go on holiday
so the "whole team" isn't really available to discuss some of this
stuff. Solving any problem like this will take time and adds a "weight"
to the whole idea of solving the problem.
Just keep doing what you're doing and I'll just have to get over it. I
can't guarantee any timeliness in a proper answer though, especially
since I'm not the expert on the bilinear or gradient search resamplers.
The end result of this thread may be a series of github issues for each
resampler in pyresample and/or another one for the thin MODIS reader in
Satpy.
Dave
> > https://we.tl/t-xgo9Z9E9py <https://we.tl/t-xgo9Z9E9py>
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> <https://groups.google.com/d/msgid/pytroll/281f3b9b-4b08-4ae1-9f13-132074d3e946n%40googlegroups.com?utm_medium=email&utm_source=footer>.
It isn't that wetransfer isn't welcome. It is that the information isn't
in the email. I understand that for this particular problem that may not
be possible. A lot of the core Satpy/pyresample devs are either on
holiday or getting back from holiday or getting ready to go on holiday
so the "whole team" isn't really available to discuss some of this
stuff. Solving any problem like this will take time and adds a "weight"
to the whole idea of solving the problem.
Just keep doing what you're doing and I'll just have to get over it. I
can't guarantee any timeliness in a proper answer though, especially
since I'm not the expert on the bilinear or gradient search resamplers.
The end result of this thread may be a series of github issues for each
resampler in pyresample and/or another one for the thin MODIS reader in
Satpy.
Dave
> Aqua-20220910-DAY-1327-natural_color-westminster-multi-nearest.jpg
>
> Unfortunately inline images seem to hide their names. I start
> with a MODIS multipass resampled 'nearest' to show you what I
> expect and name O.K.
>
>
>
> Unfortunately inline images seem to hide their names. I start
> with a MODIS multipass resampled 'nearest' to show you what I
> expect and name O.K.
>
>
> Aqua-20220910-DAY-1327-natural_color-westminster-multi-bilinear.jpg
>
> This is the same MODIS multipass image as above but resampled
> 'bilinear'. This is what I called 'interference' or 'Moiré'.
>
>
>
> This is the same MODIS multipass image as above but resampled
> 'bilinear'. This is what I called 'interference' or 'Moiré'.
>
>
> NOAA20-20220910-DAY-1228-natural_color-westminster-bilinear.jpg
>
> Here is a single pass viirs image resampled 'bilinear' that
> clearly shows the bowtie effect (data reduction towards the
> limb) normally only seen in swath images.
> I noted that the 'bilinear' resampler also has a parameter
> 'radius_of_influence'. I set 'radius_of_influence'=10000 and
> later 20000 to no avail.
>
>
>
> Here is a single pass viirs image resampled 'bilinear' that
> clearly shows the bowtie effect (data reduction towards the
> limb) normally only seen in swath images.
> I noted that the 'bilinear' resampler also has a parameter
> 'radius_of_influence'. I set 'radius_of_influence'=10000 and
> later 20000 to no avail.
>
>
> NOAA20-20220910-DAY-1228-natural_color-westminster-multi-gradient_search.jpg
>
> This NOAA-20 multipass viirs image was resampled
> 'gradient_search'. That's what I called fractured.
>
>
>
> This NOAA-20 multipass viirs image was resampled
> 'gradient_search'. That's what I called fractured.
>
>
> >
> >
> > Best regards
> > Ernst
> >
> >
> > --
> > You received this message because you are subscribed to
> the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails
> from it, send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > Best regards
> > Ernst
> >
> >
> > --
> > You received this message because you are subscribed to
> the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails
> from it, send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/pytroll/656ae6ee-1ff4-4262-8bab-d7453d27349an%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/281f3b9b-4b08-4ae1-9f13-132074d3e946n%40googlegroups.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> <https://groups.google.com/d/msgid/pytroll/281f3b9b-4b08-4ae1-9f13-132074d3e946n%40googlegroups.com?utm_medium=email&utm_source=footer>.
lobsiger...@gmail.com
Sep 12, 2022, 3:52:01 PM9/12/22
to pytroll
Dave,
there is no hurry and no timeliness expected. I came along this
can of pandora when I realized that I get ugly pink stains when
using 'nearest' with generate=False for Aqua/Terra composite
'true_color_thin'. I have played with all above tree resamplers for
GEO images where I had no real problems. I was kind of surprised
that this seems to be quite a different story for LEO satellites.
I have my workaround using resampler 'nearest' and automatically
setting generate=True in the case of Aqua/Terra 'true_color_thin'
while everything else is handled with much faster generate=False.
It's not exactly what I expected when reworking my scripts but it
does the job for now. If you find any extra time I'd rather prefer
to see PyCoast 1.6.0 issued. My new distribution of scripts make
use of all that new stuff in main still waiting to be unleashed
for common users that can only update via miniconda packages.
Best regards,
Ernst
there is no hurry and no timeliness expected. I came along this
can of pandora when I realized that I get ugly pink stains when
using 'nearest' with generate=False for Aqua/Terra composite
'true_color_thin'. I have played with all above tree resamplers for
GEO images where I had no real problems. I was kind of surprised
that this seems to be quite a different story for LEO satellites.
I have my workaround using resampler 'nearest' and automatically
setting generate=True in the case of Aqua/Terra 'true_color_thin'
while everything else is handled with much faster generate=False.
It's not exactly what I expected when reworking my scripts but it
does the job for now. If you find any extra time I'd rather prefer
to see PyCoast 1.6.0 issued. My new distribution of scripts make
use of all that new stuff in main still waiting to be unleashed
for common users that can only update via miniconda packages.
Best regards,
Ernst
David Hoese
Oct 17, 2022, 11:23:49 AM10/17/22
to pyt...@googlegroups.com
Ernst,
Pycoast 1.6.0 is now released on PyPI. I'm still waiting for conda-forge to pick up the new version. My guess is it will be available in the next 12 hours to users.
Dave
Pycoast 1.6.0 is now released on PyPI. I'm still waiting for conda-forge to pick up the new version. My guess is it will be available in the next 12 hours to users.
Dave
lobsiger...@gmail.com
Oct 17, 2022, 4:02:27 PM10/17/22
to pytroll
Dave,
thanks a lot for this major PyCoast 1.6.0 release. Fingers crossed there are no bugs left.
I can now prepare a new release of my PyTROLL/SatPy scripts for the "common user".
Best regards,
Ernst
Reply all
Reply to author
Forward
0 new messages