CSPP performance for VIIRS RDR to SDR

[Adam]

Hi,

During the last week I have been trying to get closer to an
understanding of what I should expect of CSPP now and in the future,
concerning latency for VIIRS RDR to SDR processing. At SMHI we see
CSPP processing times varying somewhere of around 30 to 45-50 minutes
(wall clock time), for one locally received NPP VIIRS scene (RDR
file), which has hindered an operational implementation so far. (We
need a latency of no more than 5 minutes.)

I have understood that these are the figures also seen at other DB
centers (DMI and FMI, and EUMETSAT). I have put forward these figures
to NOAA and Raytheon at the conference, but have been met mostly by
surprised faces, and figures that would indicate that the RDR to SDR
processing should be faster. Today I spoke to Mitch Goldberg who
showed me a presentation by Liam Gumley at the last TOVS conference.
You can find the presentation here:

http://cimss.ssec.wisc.edu/itwg/itsc/itsc18/program/files/Gumley_CSPP_ITSC18.pdf

At slide 13 it says the following:
"Nigel Atkinson recommends dual Intel hex-core 3.06 GHz
CPUs and 64 GB RAM (10 min. VIIRS pass processed in 10 min)."

So, maybe Nigel can comment and explain on this?
"dual intel hex-core" doesn't really help much in the case of CSPP as
it is not threaded!?
We have 4xIntel Xeon Hex-core 2.67GHz, so I wouldn't expect a factor
of 4 difference.

The only alternative I see to get the SDR with something that comes
close to a reasonable latency (~3-4 minutes) would be to feed several
CSPP instances with smaller segments/granules. But that I think
remains to be seen how it can be done in a proper way. And if it is
doable, it should definitely be documented and shared with the DR-
community!

I think we really have to join forces to support a DB package that can
meet our requirements for Direct Readout!

PS: Apart, from giving even more headache concerning VIIRS SDR
processing, It was a really good conference! Thanks to all the nice
people I met (again)!
Best regards from Sopot!
Adam

[Atkinson, Nigel]

Hi Adam,

The figure that I quoted at ITSC was from memory, based on a very
limited data sample, and turned out to be a bit optimistic. We
typically see 2 to 3 minutes per 85-sec VIIRS granule. So that's 16-24
minutes for an 8-granule (11.3 minute) pass. Plus a few minutes for
RT-STPS. That's with Intel Xeon X5675 3.06GHz. This is acceptable for
our purposes, but our requirements are probably not as demanding as
yours.

Meteo-France told me in June they were getting similar performance
(Pascal reported 2 minutes per granule on an Intel Xeon X5690 3.46GHz).

Both these are somewhat faster than you seem to be getting, but less
than a factor of 2. I agree that it would be good if the software could
be parallelized in some way.

Nigel