Fwd: Re: VIIRS issues

[Adam Dybbroe]

FYI.
A short discussion yesterday between Nigel and me on the VIIRS DNB and its correction.

You are welcome to have a look at the two images (corrected and uncorrected) at

	ftp.smhi.se/busers/adybbroe/Nigel

Perhaps you have done something better already?

Best regards
Adam

-------- Original Message --------

Subject:	Re: VIIRS issues
Date:	Tue, 15 Jan 2013 13:33:42 +0100
From:	Adam Dybbroe <Adam.D...@smhi.se>
To:	Atkinson, Nigel <nigel.a...@metoffice.gov.uk>
CC:	Keogh, Simon <simon...@metoffice.gov.uk>, a001673 <martin....@smhi.se>

Nigel,

Happy New Year by the way, if I didn't already wish you (I don't think so)!

No, we didn't work with this since before Christmas. I have been 
occupied with the unpleasant calibration/scale jumps I also reported. So 
far it seems to be something in our operational CSPP setup! But not nice 
at all!

Concerning the DNB features we have also noticed that NOAA/NASA seem to 
know how to fix most of the artefacts. But haven't seen any 
documentation yet.

A pure and smooth scan angle dependent correction wount work. You need 
to take care of the sudden jumps that occur in static zones. We found 
the points analysing a scene, and then we have applied that to others. 
And it seem to work. See the two pictures at

    ftp.smhi.se/busers/adybbroe/Nigel

There is something left, at the left edge of the swath. As I mentioned 
earlier.

We are curious to see if you can do something better.

PS: Can I forward this to the npp list?
Best regards
Adam



On 2013-01-15 13:16, Atkinson, Nigel wrote:
> Hi Adam,
>
> I was wondering if you had made any progress on improving the DNB images
> - e.g. the removal of the "smile effect" mentioned below? From the
> various articles on NPP that have appeared recently (e.g. "black marble"
> pictures) I guess somebody at NOAA or NASA knows how to clean up this
> data, but I've not seen any details. For instance, would a fixed scan
> angle correction do the job or would it have to vary with intensity?
> I've not looked at it in any detail yet, but we are starting to think
> about how we could improve our products.
>
> Regards,
>
> Nigel
>
>> -----Original Message-----
>> From: npp-sa...@googlegroups.com
>> [mailto:npp-sa...@googlegroups.com] On Behalf Of Adam Dybbroe
>> Sent: Wednesday, December 19, 2012 11:23 AM
>> To: npp-sa...@googlegroups.com
>> Subject: VIIRS issues
>>
>> Hi,
>>
>> I don't know about you, but we have stumbled over quite some
>> features or issues with VIIRS over the past autumn where we
>> have started looking more regularly at the data. We are yet
>> unable to say how much is onboard related and how much is
>> related to the SDR processing using CSPP. But it would be
>> nice if you could share your experience as well.
>>
>> Some of the issues we have seen so far:
>>
>> * DNB: Quite strong "smile effect" observed, that is the
>> radiance increases with scan angle, and you get a nice
>> smile-shape, as in the figure here attached. We have
>> experimented a bit applying a correction, and it does improve
>> the RGB imagery somewhat. But there is still some
>> bow-tie-like effects left on the edges, and still some
>> along-track lines left.
>> * Christelle told me that this "smile effect" is actually
>> observed on all M-bands, but I haven't investigated myself.
>> * We do observe occasional jumps in radiance levels across
>> track in the DNB. Usually it is just right in the middle of
>> the swath, but the shift is not sharp (from one line to
>> another) but rather takes place over several lines.
>> * Occasional problems in geolocation. We have seen this on at
>> least one scene a few weeks ago. Unfortunately I don't
>> remember at the moment exactly when. I talked to Herve about
>> this and he said that Pascal had observed similar problems,
>> and had looked into the CSPP logs indicating that there was
>> indeed a problem with geolocation in the actual scene.
>> * Now today we have seen a strange shift in IR calibration, a
>> sharp "jump" in intensity in the RGB images, also affecting
>> PPS, so apparently a real shift in Tb. Not nice! I will try
>> analyze this further, but it would be nice if you could
>> comment if you had seen something similar.
>>
>>
>> I am sure there is more to the list, but this is what I
>> recall just now.
>>
>> Best regards
>> Adam
>>
>>
>>
>> --
>> Adam Dybbroe,
>> Satellite Remote Sensing Scientist,
>> Numerical models and Remote Sensing,
>> Core Services, Swedish Meteorological and Hydrological
>> Institute (SMHI) www.pytroll.org nwcsaf.smhi.se www.smhi.se
>>
>>


-- 
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Numerical models and Remote Sensing,
Core Services, Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org
nwcsaf.smhi.se
www.smhi.se

[Atkinson, Nigel]

At Adam's suggestion I tried creating RGB images with reg and green from VIIRS DNB radiance, and blue from M15 brightness temperature. This works well, see attached example.

 

To remove the scan-dependent bias, I used one overpass to derive a 4064 x 32 point correction, i.e. a different correction for each pixel in a single scan of the mirror. I set the correction equal to the median radiance for each pixel, minus the median radiance at the centre of the swath. Then applied that to other scenes (such as the one in the attached). It seems to work well so far.

 

Nigel

---

From: npp-sa...@googlegroups.com [mailto:npp-sa...@googlegroups.com] On Behalf Of Adam Dybbroe

Sent: Wednesday, January 16, 2013 7:56 AM
To: NPP satellite
Subject: Fwd: Re: VIIRS issues

[Atkinson, Nigel]

All,

 

I've now written up our findings into a draft technical note on VIIRS DNB processing. If anybody is interested, email me directly at my Met Office address and I'll send you a copy.

 

Regards,

 

Nigel

---

From: npp-sa...@googlegroups.com [mailto:npp-sa...@googlegroups.com] On Behalf Of Atkinson, Nigel
Sent: Wednesday, January 16, 2013 11:45 AM
To: npp-sa...@googlegroups.com
Subject: RE: Re: VIIRS issues

[Martin Raspaud]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 24/01/13 12:19, Atkinson, Nigel wrote:
> All,
>
> I've now written up our findings into a draft technical note on
> VIIRS DNB processing. If anybody is interested, email me directly
> at my Met Office address and I'll send you a copy.

Hi Nigel, all,

We've been investing DNB also, and been focussed lately on the
artefacts on the sides of the band when the smile effect is removed. I
attach here the isolated patterns (32 "mean" scanlines) we have on our
test scene. Does anyone a clue on the origin of these ? Could it be
something optical ? Or is some kind of aggregation ?

A strange thing also is that the period of these patterns is 32
scanlines, while it seems it should be 16. Any comments on this ?

Best regards,

Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJRBlQUAAoJEBdvyODiyJI4HysH/R1CfcYmlLJ1hpxNRTASqbDS
SSyKEMWZtzhjFmcqDGAFMtPlzEC6OVoGc/kJejlYqcmNQB3Y1dIWxMJm9nuCCSoZ
dBmPGk/5wj+gv2YIxYlVCIp2fAaM97tl9lPZ7H50HEWMdAzS5UCq+txbPIUJ/bQT
vuzIoqxod+lV4B+ZUHtRmRGaDbZrjzr87pCii3jM/tetpPZvnL9QrkHc4uX/NfJ8
DPgWnD4vJcFSBlVmUVgiopBGu4k/xp8/Hsn9k+PbY5gEP30/Nx4mut3tSb8ifd4o
DJcn2ssrc8uE9vQktOcfC7rPTHOo1mDS2IMKsRdB1kRIMl9DujmN0PL9GVmAk1c=
=izxA
-----END PGP SIGNATURE-----

[Scheirer Ronald]

Hej,

this looks interesting, indeed.

It is hard to guess, what happened here.
Furthermore i wonder why - in an average image - the center is that much darker
than the edges of the swath?
How do you handle the bow-tie removal?

Maybe we should have a look at different scenes (with differen illumination?).

Best
Rolle

--
Ronald Scheirer
SMHI
Folkborgsvägen 1
60176 Norrköping
Sverige

Phone: +46 (0)11 495 8415
Fax: +46 (0)11 495 8250

www.smhi.se

________________________________________
From: npp-sa...@googlegroups.com [npp-sa...@googlegroups.com] on behalf of Martin Raspaud [martin....@smhi.se]
Sent: 28 January 2013 11:34
To: npp-sa...@googlegroups.com
Subject: Re: VIIRS day-night band

Hi Nigel, all,

Best regards,

--
You received this message because you are subscribed to the Google Groups "NPP satellite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to npp-satellit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Adam Dybbroe]

Rolle,

On 2013-01-28 12:28, Scheirer Ronald wrote:
> Hej,
>
> this looks interesting, indeed.
>
> It is hard to guess, what happened here.
> Furthermore i wonder why - in an average image - the center is that much darker
> than the edges of the swath?

Yes, that remains to be explained. Would be nice if someone at NOAA or
Raytheon could explain.

> How do you handle the bow-tie removal?

The DNB uses a CCD, so there should be no bow-ties to remove! :-)

>
> Maybe we should have a look at different scenes (with differen illumination?).

We have. And the "smile effect" is there persistently. At least here at
SMHI.

-Adam

[Martin Raspaud]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 24/01/13 12:19, Atkinson, Nigel wrote:

> All,
>
> I've now written up our findings into a draft technical note on
> VIIRS DNB processing. If anybody is interested, email me directly
> at my Met Office address and I'll send you a copy.

Hi Nigel,

Regarding the smile effect, it looks like the the radiance jumps
correspond to the shift between different aggregation zones, from what
I gather from this article:
https://ams.confex.com/ams/pdfpapers/163765.pdf

Best regards,
Martin

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJRB4v3AAoJEBdvyODiyJI4nskH/0TNBs96Wiznaq4jLDzzF4Nn
JIT6uvZ8+H7ebp2/Sdxd5vGUhSfLD+K+UMMmy1HBT9MW8ea5yaenAoC6sVhRIpIS
joq8+aQAbgnpZoFc2kvqajjTtcBUHztqNPuizPvkzrSghkx8FBBCcXGO2LIYdMqm
dZbWHOpKtOe8khdDyo+agWVWisUi8lzViJNiYxV1mxv5q6yfA2i2bAkOkTxxBAp3
Kc3fMjdSVlhb++aazrQ3tZKyuaNnEThXkYDSyyQXr6GiqypDFqRT03X8SpAeC70c
1talBOGPH9ZVZY+kAL8ogn4psdx12IfW9eAorFo3gH6Q0wGwRvIjsrWz2L8OP1c=
=VdVC
-----END PGP SIGNATURE-----

[Atkinson, Nigel]

Martin,

There's more information in the "JPSS VIIRS Radiometric Calibration ATBD
(Rev B)", available at
http://jointmission.gsfc.nasa.gov/science/documents.html

See pages 132 onwards of the pdf document (printed page 102).

I noticed a couple of things:

1) There is a radiance correction stored on board:
"The data produced by the dark scene represent the
offsets; and these will be stored in the VIIRS for every pixel in the
earth scan for each
CCD stage. VIIRS will subtract the appropriate offset value from each
transmitted pixel
value prior to transmission. Note that the dark scene data stored and
used for offset
correction consists of 4,064 (along-scan) samples x 16 (along-track)
detectors x 4
stages = 260,096 words."

But the issue of how to acquire updated dark scene data was TBD.

So I take that to mean that the offsets could change if NOAA decide to
upload new calibration data.

2) The reason we see a 32-scan cycle rather than a 16-scan cycle is
that:
"The DNB radiometric gains for even-numbered scans will differ from
those for odd-
numbered scans because the reflectances of the two sides of the
Half-Angle Mirror
(HAM) will not be quite the same."

page 51: "The Half Angle Mirror (HAM) is a two-sided mirror that serves
as derotator as it directs
scene energy from the RTA via a fold mirror into the aft optics. It
scans in the same
direction as the RTA but at half its rate."

Other useful articles on the applications of DNB include:
http://journals.ametsoc.org/doi/abs/10.1175/BAMS-87-2-191
http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-11-00192.1

Regards,

Nigel

> -----Original Message-----
> From: npp-sa...@googlegroups.com
> [mailto:npp-sa...@googlegroups.com] On Behalf Of Martin Raspaud
> Sent: Tuesday, January 29, 2013 8:45 AM
> To: npp-sa...@googlegroups.com

[Atkinson, Nigel]

All,

I tried out the newly-released CSPP SDR v1.3 (see Liam Gumley's email of
8th Feb). This has greatly reduced the bias that we had been seeing in
the DNB - see attached images from this morning. These are uncorrected
images. They show:

- a small amount of residual bias structure at edge of scan.
- strong aurora (I presume?).
- step change in the far north. I think we have seen those before, not
sure what the cause is.

As mentioned by Liam, the new features of CSPP v1.3 that seem to be
relevant to DNB calibration are:

- VIIRS SDR now uses predictive solar calibration for M-bands, I-bands,
and Day/Night band,
- VIIRS calibration LUT updates are now available weekly.

[Adam Dybbroe]

Nigel,

On 2013-02-11 12:47, Atkinson, Nigel wrote:
> All,
>

> I tried out the newly-released CSPP SDR v1.3 (see Liam Gumley's email of
> 8th Feb). This has greatly reduced the bias that we had been seeing in
> the DNB - see attached images from this morning. These are uncorrected
> images. They show:
>
> - a small amount of residual bias structure at edge of scan.
> - strong aurora (I presume?).
> - step change in the far north. I think we have seen those before, not
> sure what the cause is.

Nice, thanks.
We will test the new package soon.

-Adam

[Adam Dybbroe]

All,

We can verify your findings Nigel. See attached images from the past night.
The limb brightening or smile effect is corrected for in version 1.3 it
seems. I spoke to Liam about it and he thought it was already fixed in
version 1.2? But apparently both you and we (and also DMI I think) have
verified that running 1.2 did not solve the limb brightening problem.

Also, concerning the step change you mention, yes we observe that quite
often. I am not sure but I have a feeling it is something we see mainly
under New moon conditions, but that is purely a subjective observation.
It is evident also in the example here. Liam said:

"The bright band at the top of your image is most likely due to a stray
light leak on the VIIRS sensor. Efforts are underway to model this
effect and remove it from the imagery, but it will be a while before we
can include this correction in CSPP."

Best regards
Adam

[Atkinson, Nigel]

There is still a small amount of bias structure at the edge of the
images. See the attached extract from my current correction function.
I've just plotted spots 2 to 700 (spot 1 is empty).

But we no longer have a 32-line cycle: it's a 16-line cycle, which makes
the processing much easier.