CSPP VIIRS timeliness

[Adam Dybbroe]

Dear colleagues,

As part of the efforts to get a faster VIIRS SDR processing I have
looked at the average timeliness we have here at SMHI when running CSPP
on full VIIRS swaths (one RDR VIIRS file) in operations. When calculated
over the last couple of days we have on average VIIRS swaths with a
length of around 9.3 SDR granules or 13.3 minutes. Here on our
operational server the timeliness varies between around 15 and 30
minutes, with an average of 21 minutes. Wall clock times that is:
('Minutes - N, min, max, mean: ', 31, 15.47, 30.04, 21.23)

This is with version 1.3 of CSPP (the most recent).

We have 24 CPUs, and CSPP only uses one. So, even if our server is
loaded with lots of other stuff we assume that CSPP gets the full power
from one CPU (not significantly shared with other processes).

The CSPP team informs they have a processing time of around 11 minutes
for a 10 minute swath. Our equivalent is then 15.8 minutes. So there
seem to be some discrepancy there. How they achieve this 30% better
processing speed is yet unknown to me. We have earlier tested running
CSPP on RAM disks and that did not make any significant difference in
speed. We run, as they do, without granule aggregation and without hdf5
compression.

So, any measures of the timeliness you experience would be most welcome!
Please, Meteo-France, UK-Met Office, Eumetsat and FMI, if you can make
some quick estimates of your local CSPP performance, share it!

Martin and I are now, looking into how we can run CSPP on granules as
they come in from RT-STPS if we run the latter in a server mode. This
patched RT-STPS is capable of taking in a stream of CADU data and make
RDR granules in real time.

SCISYS is doing this in test in Bochum, and they kindly provided us with
a case of such RDR granules (as we do not yet do this ourselves). We
processed those with CSPP and got SDR's. However, there is still a
problem with CSPP, when it doesn't know anything about its neighbouring
granules. Our data looked fine, *except* for several corrupted data
lines around the granule boundaries. So, this is unacceptable. I got the
following answer from the CSPP team:

"It sounds like you have identified an issue related to cross-granule
processing in CSPP. For the VIIRS SDR processing, adjacent granules are
not required to process a particular granule, but they will be used if
they are present. We may need to change the logic in streaming mode to
say that to process granule N, you need to have granules N-1 and N+1
available. This means you would have to wait an extra 90 seconds for
granule N+1 before you started processing granule N. Perhaps you can
implement this logic in your processing environment."

I think we can implement this logic. But I thought CSPP would be able to
do this for me. That is a pity.

Liam also said:
"On a related note, we have implemented a multi-core processing option
in the CSPP VIIRS SDR processing Python drivers. You can now specify
that 2 or more CPU cores be used at runtime to process individual
granules. There is no cross granule issue, because each instance of the
VIIRS SDR controller can still see all the unpacked RDR data for the DB
pass. A beta version of the CSPP SDR package with this feature should be
ready in the next few days, so we will send it to you for beta testing.
Our benchmarks have shown that with HDF5 compression and granule
aggregation disabled, the processing time for a 10-minute VIIRS DB pass
can be reduced from 11 minutes to less than 4 minutes."


That is of course a nice step forward, which should benefit us all!
Then, perhaps in a near future, there might not be a need to do the
processing on individual granules with several CSPP instances in parallel?

Best regards
Adam




--
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Numerical models and Remote Sensing,
Core Services, Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org
nwcsaf.smhi.se
www.smhi.se

[Atkinson, Nigel]

Hi Adam,

When I looked recently we were typically taking 25 minutes to process a full pass - that's RT-STPS followed by CSPP SDR processing with ATMS, CrIS and VIIRS on separate threads. That looks consistent with your figures.

Does the run time reported by the CSPP team include RT-STPS? It only takes ~2.5 minutes but if they are not including it and you are, that could help to reduce the discrepancy.

Interesting to hear about the CSPP developments in multi-core processing. I expect we'll hear more about that at the CSPP workshop at Madison in May.

Nigel

> --
> You received this message because you are subscribed to the
> Google Groups "NPP satellite" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to npp-satellit...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

[Adam Dybbroe]

Hi,

On 2013-04-19 11:51, Atkinson, Nigel wrote:
> Hi Adam,
>
> When I looked recently we were typically taking 25 minutes to process a full pass - that's RT-STPS followed by CSPP SDR processing with ATMS, CrIS and VIIRS on separate threads. That looks consistent with your figures.

Okay, thanks.

> Does the run time reported by the CSPP team include RT-STPS? It only takes ~2.5 minutes but if they are not including it and you are, that could help to reduce the discrepancy.

I don't know about the CSPP team, but I didn't include RT-STPS in my
figures. My figures are only from RDR to SDR with CSPP. We do not
process ATMS and CrIS.

> Interesting to hear about the CSPP developments in multi-core processing. I expect we'll hear more about that at the CSPP workshop at Madison in May.

Yes, I hope so.

Cheers
Adam

[Roquet pascale]

Dear Adam,

In April, at Meteo-France CMS/Lannion the observed time processing for the longest VIIRS swaths
(10/11 SDR granules) are :

CSPP VIIRS SRD RT-STPS
mean 00:15:30 00:02:38
min 00:13:23 00:02:22
max 00:17:50 00:02:58

We use CSPP 1.3 with no compression, no aggregation and we perform ancillary data retrieval by cron.

Best Regards,

Pascale

[Adam Dybbroe]

Pascale,

Thanks for the information.
Your numbers are consistent with the numbers provided by Liam as far as
I can see.

Best
Adam