Once-in-a-while error in processing MSG and IODC JPG images

[Fabio Mingione]

Dear Pytroll community,

In our HRIT to JPG MSG and IODC processing scripts (producing images every 15 minutes), we're experiencing an error causing us to lose all MSG and IODC images only for certain time slots (5/6 time slots per day).

As the attached log from one of our scripts/templates suggests, the error seems related to the scene.py satpy module and/or some readers modules. All other logs within the same time slot report the same errors. 

I can't understand why the scripts work perfectly for other time slots (the error happens randomly, once in a while, with no apparent pattern), and a clone virtual machine (producing exactly the same images) misses no output at all, while working with exactly the same scripts and satpy version. 

Do you have any suggestion? Thanks a lot!

Fabio

[Panu Lahtinen]

On 6.6.2022 10.44, Fabio Mingione wrote:
> Dear Pytroll community,
>
> In our HRIT to JPG MSG and IODC processing scripts (producing images
> every 15 minutes), we're experiencing an error causing us to lose all
> MSG and IODC images only for certain time slots (5/6 time slots per day).

As a first guess: you are missing some files. How do you trigger your
script? Or do you use a collector (like
pytroll-collectors/segment_gatherer) for the segments before the script
is run? If you are using Eumetcast to get the data, the files are not
guaranteed to be coming in any particular order, so you if you trigger
for example based on the EPI file you will often miss some segments that
arrive after the EPI file.


P

--
Finnish Meteorological Institute
Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
Room: 4B15a, Phone: +358 50 380 3261

[Fabio Mingione]

Hi Panu,

Thank you very much for your quick reply. Yes, we thought about that too. 

We trigger based on the epilogue and then wait 60 more seconds in case the last desired segment arrives later, which was always sufficient to produce the desired images so far.

 

In any case, we managed to check the folder where segments, PRO and EPI are collected, during a faulty execution (with no output), and everything was present as usual, until the end of the process. So missing files don't seem to be the issue unfortunately. 

[lobsiger...@gmail.com]

Hi Fabio,

I'am processing MSG4,  MSG3 and MSG2 from EUMETCast without segment-gatherer simply by using timeliness.

I start processing 30 minutes after 15minute slots (MSG4, MSG2) and 10minutes after 5minute RSS-slots (MSG3).

As an example: For the MSG4 image denoted 12:00 UTC processing is started 12:30 UTC. This works very well here.

Regards,

Ernst

P.S. This of course asks for a rather reliable EUMETCast reception system with a sound EsNO (SNR) of >13 dB.