Cannot read SNPP and NOAA-21 DNB files

[lobsiger...@gmail.com]

Dear Developers

EUMETCast sends us SNPP, NOAA-20 and NOAA-21 VIIRS data (M-bands and some DNB-bands).
I cannot read DNB-files from good old SNPP anymore and neither from more recent NOAA-21.
There is no problem reading DNB-files from NOAA-20. I use satpy 0.57.0 under GNU/Linux.

Read the whole story with more background here: https://groups.io/g/MSG-1/message/39638

I boiled down my lengthy satpy script to a few lines DNBtest.py to demonstrate the problem:

from satpy import Scene
SNPP   = ['./SVDNBC_npp_d20250622_t0100084_e0101326_b70734_c20250622010534000682_eum_ops.h5']
NOAA20 = ['./SVDNBC_j01_d20250622_t0121283_e0122528_b39338_c20250622012757000085_eum_ops.h5']
NOAA21 = ['./SVDNBC_j02_d20250622_t0214053_e0215300_b13547_c20250622022032000449_eum_ops.h5']
scn = Scene(filenames = NOAA21, reader = 'viirs_compact')



(pytroll) eumetcast@kallisto:~/DNBTEST$ python DNBtest.py
Traceback (most recent call last):
  File "/home/eumetcast/DNBTEST/DNBtest.py", line 5, in <module>
    scn = Scene(filenames = NOAA21, reader = 'viirs_compact')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/scene.py", line 155, in __init__
    self._readers = self._create_reader_instances(filenames=filenames,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/scene.py", line 176, in _create_reader_instances
    return load_readers(filenames=filenames,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/loading.py", line 65, in load_readers
    reader_instance.create_storage_items(
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/yaml_reader.py", line 618, in create_storage_items
    return self.create_filehandlers(files, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/yaml_reader.py", line 630, in create_filehandlers
    filehandlers = self._new_filehandlers_for_filetype(filetype_info,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/yaml_reader.py", line 613, in _new_filehandlers_for_filetype
    return list(filtered_iter)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/yaml_reader.py", line 595, in filter_fh_by_metadata
    for filehandler in filehandlers:
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/core/yaml_reader.py", line 591, in _new_filehandler_instances
    yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/satpy/readers/viirs_compact.py", line 127, in __init__
    self.c_align = da.from_array(self.geography["AlignmentCoefficient"],
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/dask/array/core.py", line 3488, in from_array
    chunks = normalize_chunks(
             ^^^^^^^^^^^^^^^^^
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.11/site-packages/dask/array/core.py", line 3130, in normalize_chunks
    raise ValueError(
ValueError: Chunks do not add up to shape. Got chunks=((2, 28, 2, 3, 2, 3, 3, 3, 5, 4, 5, 4, 4, 4, 4, 4, 3, 5, 3, 4, 3, 23, 23, 3, 4, 3, 5, 3, 4, 4, 4, 4, 4, 5, 4, 5, 3, 3, 3, 2, 3, 2, 46),), shape=(516,)
(pytroll) eumetcast@kallisto:~/DNBTEST$



As described on the MSG-1 list only NOAA20 works fine while SNPP and NOAA21 always fail as above.

If necessary I can provide those files but didn't do it so far because of their rather big sizes.
3997318 Jun 23 18:57 SVDNBC_j01_d20250622_t0121283_e0122528_b39338_c20250622012757000085_eum_ops.h5
4224003 Jun 23 18:57 SVDNBC_j02_d20250622_t0214053_e0215300_b13547_c20250622022032000449_eum_ops.h5
4321006 Jun 23 18:57 SVDNBC_npp_d20250622_t0100084_e0101326_b70734_c20250622010534000682_eum_ops.h5


Cheers,
Ernst

[Panu Lahtinen]

On 23.6.2025 22.47, lobsiger...@gmail.com wrote:
> Dear Developers

Hi Enrst,

> EUMETCast sends us SNPP, NOAA-20 and NOAA-21 VIIRS data (M-bands and
> some DNB-bands).
> I cannot read DNB-files from good old SNPP anymore and neither from
> more recent NOAA-21.
> There is no problem reading DNB-files from NOAA-20. I use satpy 0.57.0
> under GNU/Linux.

I see the same for all the DNB data for Suomi-NPP (52 files) and NOAA-21
(4) I still have available. The data for NOAA-20 (52) works.

I haven't seen this in production because I've set the sunlight coverage
so that the DNB composite is actually completely filtered out during the
northern polar summer.

I'll see if I can debug this a bit more before the summer holiday, but
already from your log the chunk division seems a bit weird:

> ValueError: Chunks do not add up to shape. Got chunks=((2, 28, 2, 3,
> 2, 3, 3, 3, 5, 4, 5, 4, 4, 4, 4, 4, 3, 5, 3, 4, 3, 23, 23, 3, 4, 3, 5,
> 3, 4, 4, 4, 4, 4, 5, 4, 5, 3, 3, 3, 2, 3, 2, 46),), shape=(516,)

For a working NOAA-20 data I get

ipdb> scn["DNB"].data.chunks
((768,), (4064,))


P

--
Arctic Space Centre
Finnish Meteorological Institute
Phone: +358 50 380 3261

[Panu Lahtinen]

Hi,

The DNB data for Suomi-NPP and NOAA-21 seems to be incorrectly
formatted. Running the following script to the files, we can see that
the sum of "NumberOfTiePointZonesScan" and the size of
"AlignmentCoefficient" do not match when these values should always match.

Save the following to "test.py" or something, and run with a single DNB
file as an argument. For a working data the two values match.

###########################

#!/usr/bin/env python

import sys

import h5py

def main():
    fname = sys.argv[1]
    print(fname)
    fid = h5py.File(fnames[0], "r")
print(fid["All_Data"]["VIIRS-DNB-GEO_All"]["AlignmentCoefficient"].size)
print(fid["All_Data"]["VIIRS-DNB-GEO_All"]["NumberOfTiePointZonesScan"][:].sum())
    fid.close()


if __name__ == "__main__":
    main()

###########################


I'll ask around a bit in Pytroll Slack, and if there's no additional
information about these variables, I'll report this to EUMETSAT ops so
they can have a look.


P

[lobsiger...@gmail.com]

Panu,

thanks for your help. As I noted on MSG-1 my daily SNPP images of Switzerland at night using composite

'night_overview' stopped working after 2025 June 3rd while NOAA-20 kept working. So this date probably  marks

 some change in preprocessing SNPP DNB files. No idea whether EUMETSAT or even NOAA is the place to ask.

Ernst

[Panu Lahtinen]

On 25.6.2025 11.05, lobsiger...@gmail.com wrote:
> Panu,
>
> thanks for your help. As I noted on MSG-1 my daily SNPP images of
> Switzerland at night using composite
> 'night_overview' stopped working after 2025 June 3rd while NOAA-20
> kept working. So this date probably  marks
>  some change in preprocessing SNPP DNB files. No idea whether EUMETSAT
> or even NOAA is the place to ask.

Ok, that date matches the date when EUMETSAT activated EARS-NG, see
https://user.eumetsat.int/news-events/alerts-and-maintenance-notifications/uns.announcements.12176

I'll write a message to EUMETSAT OPS later today.

[lobsiger...@gmail.com]

Panu, good catch. My last SNPP 'night_overview' image of Switzerland dates from 2025 June 3rd  01:58 UTC.  Ernst

[lobsiger...@gmail.com]

One more thought: SNPP has been on HVS-1 Service for a long time already while

NOAA-20 was on Basic Service. NOAA-21 was added to HVS-1 recently. NOAA-20 has

now also been moved to HVS-1 and the transition phase STOPS TODAY. So I expect

NOAA-20 will now be treated with the new system and also stops working tomorrow :-\.

https://user.eumetsat.int/news-events/news/update-to-ears-nwc-and-ears-viirs-services

Note: On 10 June, we will streamline the distribution of all EARS-VIIRS data by using the same EUMETCast channels:

    On EUMETCast Europe: the NOAA-20 data, currently distributed on the Basic Service (channel E1B-RDS-2), will be moved to HVS-1, on the same channel (E1H-RDS-1) as SNPP and NOAA-21 data. NOAA-20 data will be available in parallel on the Basic Service and HVS-1 for a two-week period until 25 June, and will then be removed from the Basic Service.

Ernst

[Panu Lahtinen]

On 25.6.2025 14.07, lobsiger...@gmail.com wrote:
>
> One more thought: SNPP has been on HVS-1 Service for a long time
> already while
> NOAA-20 was on Basic Service. NOAA-21 was added to HVS-1 recently.
> NOAA-20 has
> now also been moved to HVS-1 and the transition phase STOPS TODAY. So
> I expect
> NOAA-20 will now be treated with the new system and also stops working
> tomorrow :-\.

I tested the NOAA-20 data from hvs-1/E1H-RDS-1 and it seems to ok, so
there's at least that.

We've been in contact with EUMETSAT, but most of us are starting our
summer holidays after this week, so it might be several weeks that we
can report on the status.

For the same reason, I'll be disabling DNB processing completely for the
summer time.

[lobsiger...@gmail.com]

Panu,

I can still make a NOAA-20 'night_overview' image from HVS-1 today  02:09 UTC

But when I test the latest NOAA-20 DNB file I received on HVS-1 from 08:29 UTC

SVDNBC_npp_d20250625_t0829462_e0831103_b70782_c20250625083857000764_eum_ops.h5

it fails in my simple my test script. So NOAA-20 has now also DNB-ng and does not work anymore.

Have nice holidays!

Ernst

[lobsiger...@gmail.com]

Sorry, I was wrong. I managed to copy and test an SNPP DNB file.

The latest HVS-1 NOAA-20 DNB file from 10:30 I have still works.

SVDNBC_j01_d20250625_t1030418_e1032045_b39386_c20250625103943000489_eum_ops.h5

E

[Panu Lahtinen]

Hi,

EUMETSAT OPS sent a message that they have identified the issue and
they'll be sending some test data later. The "later" was 3 minutes after
the first message, and the files worked. No news yet when they can
implement the changes to operations, but we'll let you know ASAP if we
receive the info before our holidays.


P

> Phone: +358 50 380 3261 <tel:+358%2050%203803261>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com.
> To view this discussion, visit
> https://groups.google.com/d/msgid/pytroll/5e037ce2-d417-4415-b52b-2b9a04a5ecb1n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/5e037ce2-d417-4415-b52b-2b9a04a5ecb1n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Hi,

thanks again for your help. I have now changed my PC to read all  VIIRS data from HVS-1. 

I'll see when my system will again produce all these shiny 'night_overview' images from

all 3 JPS sats. As this is just an amateur reception station, there is no hurry from my side.

Ernst

[lobsiger...@gmail.com]

Hi,

the Format problem seems to be solved. See

https://groups.io/g/MSG-1/message/39675

Ernst