Visible seam between VIIRS granules

[Isaac Wilmur]

Hello everyone,

When attempting to combine data between two VIIRS granules using satpy there appears to be a gap inbetween, see image below

Another image generated by Polar2grid doesn't have this problem

Below is the running scripts and related images, any help would be much appreciated.

Thanks,

Isaac

[Adam Dybbroe]

Isaac,

Could it be that there actually is a missing scan (or part of a scan) in one of the granules and that the resampling in Polar2grid interpolates/fills out the missing part?

Can you check how many scans you have in each granule?

What happens if you increase the search radius (from the default - whatever that is here in this case?) when resampling? I am not very familiar with the EWA resampler, but with the nearest neighbour and bilinear resamplers you can set the search radius via the radius_of_influence attribute.

-Adam

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/3b06e2b6-8e23-4d00-8d7f-29621f38cec3n%40googlegroups.com.

-- 
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Atmospheric Research unit, Research and Development
Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org
nwcsaf.smhi.se
www.smhi.se

[Isaac Wilmur]

Hello,

Having checked through N_Number_Of_Scan, it should be:

48, 48, 47, 48 for the bottom granule

48, 48, 48, 47 for the top granule

With radius_of_influence=10000 (ewa doesn't accept this I believe) and nearest resampler the granules are stitched up but the seam is still very much visible, I'm not sure if it's supposed to be like that

Hope that was helpful,

Isaac

[Isaac Wilmur]

Hi all,

So I managed to "fix" this problem by using the same setup but with de-aggregated data (acquired through CLASS) instead.

That should be it for me but some extra commentaries from contributors would be nice.

Isaac

[David Hoese]

Isaac,

I guess EWA is just magic. Final answer.

But actually, we'd have to look at the data for sure, but one thing you
could try would be to use EWA resampling in Satpy. You can do this by
providing the following arguments the `scn.resample` in addition to your
area definition.

resampler='ewa', weight_distance_max=2.0, weight_delta_max=40.0

This should mimic almost exactly what Polar2Grid 2.3 is doing. If the
line still exists in the output then that means the issue is somewhere
else in Satpy.

Is there any chance you could put the files you're using somewhere we
could access?

Dave

On 5/12/22 05:35, Adam Dybbroe wrote:
> Isaac,
>
> Could it be that there actually is a missing scan (or part of a scan) in
> one of the granules and that the resampling in Polar2grid
> interpolates/fills out the missing part?
>
> Can you check how many scans you have in each granule?
>
> What happens if you increase the search radius (from the default -
> whatever that is here in this case?) when resampling? I am not very
> familiar with the EWA resampler, but with the nearest neighbour and
> bilinear resamplers you can set the search radius via the
> radius_of_influence attribute.
>
> -Adam
>

> *
> *

>
> On 12/05/2022 11:58, Isaac Wilmur wrote:
>> Hello everyone,
>>
>> When attempting to combine data between two VIIRS granules using satpy
>> there appears to be a gap inbetween, see image below
>>
>> Another image generated by Polar2grid doesn't have this problem
>>
>> Below is the running scripts and related images, any help would be
>> much appreciated.
>>
>> Thanks,
>>
>> Isaac
>> --
>> You received this message because you are subscribed to the Google
>> Groups "pytroll" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to pytroll+u...@googlegroups.com.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/pytroll/3b06e2b6-8e23-4d00-8d7f-29621f38cec3n%40googlegroups.com

>> <https://groups.google.com/d/msgid/pytroll/3b06e2b6-8e23-4d00-8d7f-29621f38cec3n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> Adam Dybbroe,
> Satellite Remote Sensing Scientist,
> Atmospheric Research unit, Research and Development
> Swedish Meteorological and Hydrological Institute (SMHI)
> www.pytroll.org
> nwcsaf.smhi.se
> www.smhi.se
>

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com

> <mailto:pytroll+u...@googlegroups.com>.

> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se
> <https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se?utm_medium=email&utm_source=footer>.

[Isaac Wilmur]

Hello,

Per your suggestions I tried to use EWA on both types of datasets and the results are pretty much the same. With aggegrated data producing an image with a gap inbetween granules, de-aggegrated data produced the satisfactory result.

>>>nscn=scn.resample(my_area, resampler='ewa', weight_distance_max=2.0, weight_delta_max=40.0)

Below are the related datasets:

https://drive.google.com/drive/folders/1-zLLjJKpaUryEDrau7sKOZ3W9slBeFG1?usp=sharing

Hope that was helpful,

Isaac

[David Hoese]

Thanks Isaac,

My guess based on your message about the number of scans is that Satpy's
reader is technically being more correct with dealing with the different
number of scans. At least that's my hope because I did update the VIIRS
SDR reader to be smarter about handling "missing" scans. What we've seen
in the past is some odd behavior between number of scans reported in a
file and the amount of actual data. I'm not sure I'll have time to dive
into the files, but glad you got the deaggregated files to work.

Dave

> <https://groups.google.com/d/msgid/pytroll/3b06e2b6-8e23-4d00-8d7f-29621f38cec3n%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/pytroll/3b06e2b6-8e23-4d00-8d7f-29621f38cec3n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> >
> > --
> > Adam Dybbroe,
> > Satellite Remote Sensing Scientist,
> > Atmospheric Research unit, Research and Development
> > Swedish Meteorological and Hydrological Institute (SMHI)

> > www.pytroll.org <http://www.pytroll.org>
> > nwcsaf.smhi.se <http://nwcsaf.smhi.se>
> > www.smhi.se <http://www.smhi.se>

> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pytroll" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pytroll+u...@googlegroups.com
> > <mailto:pytroll+u...@googlegroups.com>.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se
> <https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se>
>
> >

> <https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/e651eb5d-70a5-5b82-2d74-22be612a1e2e%40smhi.se?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/e487a6fe-241e-4177-8037-294a27ebfccbn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/e487a6fe-241e-4177-8037-294a27ebfccbn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Hello,

I have a similar issue with nwcsaf data from EUMETCast. I resample nearest with radius_of_influence=2000.

While this works well for Metop-B data I have ugly gaps between the 1 minute granules from NOAA-19 data.

Looking into the files with ncdump I see that Metop-B granules have 360 scan lines while NOAA-19 granules

have only 351 lines. I must set radius_of_influence=6000 for NOAA-19 to close the gaps. No idea whether

the problem is the SAF processing of the NOAA-19 EUMETCast data or the 'nwcsaf-pps_nc' reader itself.

Regards,

Ernst

[lobsiger...@gmail.com]

Hello,

remembering from the old days that the AVHRR instrument has a rotating mirror that moves at 360 rpm there MUST be 360 scanlines in a 1 minute granule. So after sleeping over this problem it seems now obvious to me that in the NOAA-19 data segments received via EUMETCast 9 lines are simply missing. As the file naming is "W_XX-EUMETSAT-Darmstadt,SING+LEV+SAT,NOAA19+CT_C_EUMS_20220515201800_68385.nc.bz2" I probably have to ask OPS who is responsible for that.

Cheers,

Ernst

[Adam Dybbroe]

Ernst,

It was a while back I looked at this myself, but when I did the number of lines in the NWCSAF products coming via EUMETCast matched the AVHRR granules coming the same way via the EARS-AVHRR service. So, I think the issue is in the level-1 processing with AAPP run at EUMETSAT rather than the NWCSAF/PPS run at EUMETSAT or the Satpy reader. Or rather the acquisition of data is more likely the issue (than the actual level-1 processing). Could it be that the downlink signal is weaker from N19 and therefore we end up having more missing lines there? In general we seem to receive and process NOAA-19 well here in Norrköping, so maybe it is a far fetched hypothesis?

-Adam

To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/455ac67c-cb13-4c23-ac31-7e75367b07e8n%40googlegroups.com.

-- 
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Atmospheric Research unit, Research and Development
Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org

nwcsaf.smhi.se
www.smhi.se

[Panu Lahtinen]

As this isn't anymore about VIIRS data, I've changed the topic.

On 16.5.2022 11.23, Adam Dybbroe wrote:
> Ernst,
>
> It was a while back I looked at this myself, but when I did the number
> of lines in the NWCSAF products coming via EUMETCast matched the AVHRR
> granules coming the same way via the EARS-AVHRR service. So, I think the
> issue is in the level-1 processing with AAPP run at EUMETSAT rather than
> the NWCSAF/PPS run at EUMETSAT or the Satpy reader.

Yes, I think this is the cause for the NWC SAF granules from Eumetcast.
AAPP drops few lines from the beginning and end of the data it is given.
I'm not sure why, but we had this same issue with our old processing
chain where we ran the EARS granules through AAPP one at a time. Now
that we first concatenate the granules to cover our target area, the
deleted scan lines don't matter that much as they typically are outside
the area, and in every case are less evident when they are missing from
the ends of the swath.

And yes, I think this is something that should be asked from Eumetsat
helpdesk.


P

--
Finnish Meteorological Institute
Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
Room: 4B15a, Phone: +358 50 380 3261

[lobsiger...@gmail.com]

Hi Adam and Panu,

as an EUMETCast end user TBH I do not really understand who processes what and where. But what I see is that all these 1 minute NOAA19 granules have exactly 351 lines except for the granules that seem to be AOS and LOS of some EARS ground station probably at the end of EARS coverage. I wrote a small script and extracted these "ny" values for a whole day (my 70th birthday!). It's the same problem for CT, CMA and CTTH files. I also asked OPS at EUMETSAT. I'll report back what they say.

Best regards,

Ernst

[lobsiger...@gmail.com]

Hi Panu,

the trick with concatenation of 1 minute granules seems to be explained in the AAPP forum:

https://nwp-saf.eumetsat.int/site/forums/topic/noaa19/#post-13319

Do you think that's a possible hint I should mail to OPS as well?

Regards,

Ernst

[lobsiger...@gmail.com]

Hi everybody,

I can confirm that EUMETSAT has fixed the NOAA19 NWCSAF problem. Since yesterday 20220519 14:29 the EUMETCast CT/CMA/CTTH 1 minute granules come in with 360 scanlines as expected.

Best regards,

Ernst

Last image before and first image after the EUMETSAT fix (Source: EUMETCast Europe channel E1B-RDS-1, resampled nearest with radius_of_influence=2000)