Reading MTG FCI brightness temperature / reflectance data

[Steve Albers]

Greetings: in the 'satpy' MTG example it shows how to read a dataset and an associated composite product. If I want to simply read calibrated brightness temperature or relected data how can this be accessed, for example using 'scn.load'?

[Andrea Meraner]

Hi Steve, 

thank you for your question. You can access the single channels by loading them, e.g. by their name, through the load call:

scn.load(['vis_04', 'ir_105'], upper_right_corner='NE')this will load the single channels automatically at the "highest" available calibration level, i.e. reflectances for VIS-NIR channels and brightness temperatures for IR channels.

You can then access the data in a daskified xarray via 

vis_04_xa = scn['vis_04']

or in a computed numpy array with 

vis_04_np = scn['vis_04'].values 

Hope this helps!

Cheers, 

Andrea

[Steve Albers]

Thanks Andrea - fyi I'm continuing to see some errors when doing this. Attached is my script and results when converting FCI data into NetCDF if that helps with any clues. There may be a compression library issue I'm not getting quite right.

Steve

[Steven Albers]

Hello all – thought I’d ping the group at this point to see if there are any further ideas or if others have converted FCI data to NetCDF.

 

Thanks much,

 

Steve

 

From: Steven Albers <steve....@spire.com>
Date: Tuesday, April 9, 2024 at 4:03 PM
To: pyt...@googlegroups.com <pyt...@googlegroups.com>
Subject: Re: [pytroll] Re: Reading MTG FCI brightness temperature / reflectance data

Thanks much Andrea (and Martin) for your advice! Thus far I’m attempting a simple conversion to NetCDF with these two lines, though it gives some error messages. There might be an issue with the NetCDF aspects of things. I’ll attach the script and its output in case it helps. The idea would be a simple example to read in an MTG channel and write it out as full disk data in NetCDF format.

scn.load(['ir_105'])

 

scn.save_datasets(writer="cf")

 

Take care,

 

Steve

 

From: pyt...@googlegroups.com <pyt...@googlegroups.com> on behalf of Andrea Meraner <andrea....@hotmail.it>
Date: Thursday, April 4, 2024 at 2:12 AM
To: pytroll <pyt...@googlegroups.com>
Subject: [pytroll] Re: Reading MTG FCI brightness temperature / reflectance data

You don't often get email from andrea....@hotmail.it. Learn why this is important

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you have verified the sender’s email address and know the content is safe.

---

 

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/23925827-be2e-42eb-ac5a-4a03b50483ffn%40googlegroups.com.

[Andrea Meraner]

Hi Steve, 

it looks like you're missing the decompression filter. As described in the example, it's enough to install hdf5plugin and add "import hdf5plugin" at the top of your script. https://satpy.readthedocs.io/en/latest/examples/fci_l1c_natural_color.html

Cheers, Andrea

[Steven Albers]

Hello Andrea and all,

 

Thanks much for your comment and interest in this example. My latest attempt imports ‘hdf5plugin’ and I get a message at the bottom saying:

 

“RuntimeError: NetCDF: Filter error: undefined filter encountered”

Is there another filter i should install for MTG data? I’ll attach the script and output log via a zip file.

 

Thanks,

 

Steve

 

--
You received this message because you are subscribed to a topic in the Google Groups "pytroll" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pytroll/IReJh5kjm5g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pytroll+u...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/a9613ee8-1efc-4b78-9aee-d393d355ea31n%40googlegroups.com.

[Steve Albers]

Hi - to help clarify things I believe the MTG data may be read in an decompressed OK. The error seems to arise when writing out the full-disk radiance dataset in NetCDF format. Does anyone have an example where this is being done?

[Andrea Meraner]

Hi Steve, 

sorry for the late reply. The fact that the code fails when writing to the output files is to be expected, since satpy waits to actually load the data into memory until then, so that's when the error occurs. 

But the fact that it's still happening after the hdf5plugin import is worrying... some other users are reporting similar issues, it seems to maybe be a compatibility issue with numpy, netcdf4 and hdf5plugin... could you let us know which packages versions you have installed (with conda list or so)?

Also you could try to limit/downgrade your package versions to netCDF4==1.6.4 and numpy<2, this has solved the issue for some users.

Hope this helps, 

cheers, 

Andrea

[Steven Albers]

Hello Andrea,

 

Thanks much for the suggestions. I reran the script with the latest satpy version 53. Also in the attached zip file we can see I have netCDF4 and numpy versions that should be compatabe in the log file top part. The NetCDF file is included in the zip file as just the header info is written to it so far.

INFO:satpy.writers.cf_writer:Saving datasets to NetCDF4/CF.

NumPy version: 1.24.2

netCDF4 version: 1.6.2

I still get the same error message with the NetCDF writer (and compression) at the bottom of the log file.

 

RuntimeError: NetCDF: Filter error: undefined filter encountered

I wonder if anyone can show a working script that writes MTG/FCI data in NetCDF by any chance?

 

Take care,

 

Steve

To view this discussion, visit https://groups.google.com/d/msgid/pytroll/e447363b-b04a-4674-95c9-3b9962a6648en%40googlegroups.com.

[Andrea Meraner]

Hi Steve, 

the reader (and then the writing, that fails because the files are not decompressed correctly) itself works well for the vast majority of people, but you're now not the first one to report this issue related to hdf5plugin - it is a compatibility issue with lower-level libraries that we are also investigating - however note that the root cause is outside satpy. The fix could come with a next update of h5py.

Did you install your environment with pip? If yes, any chance you can try with conda? If you use pip, this has also worked for some:

pip install satpy netcdf4==1.7.1.post2 h5py==3.11.0 hdf5plugin numpy==1.26.4 h5netcdf

possibly needing 

apt-get update && \ apt-get install --no-install-recommends -y \ python3-pip \ libhdf5-serial-dev \ 

libnetcdf-dev

Another quick attempt you can try first is using

import os.path import hdf5plugin os.environ['HDF5_PLUGIN_PATH']=os.path.dirname(hdf5plugin.__file__)+'/plugins/' import satpy

This has worked for some. Hope it helps!

Cheers, 

Andrea 

[Steven Albers]

Hello Andrea,

 

Interesting details and suggestions. The conda environment setup for this does now work! Attached is a look at the result using ‘ncview’.

 

Thanks very much for your help.

 

Steve

 

To view this discussion, visit https://groups.google.com/d/msgid/pytroll/bf310c65-891f-41fc-9d24-4ade03d7e00en%40googlegroups.com.

[Andrea Meraner]

Hi Steve, that is great to hear! Just to confirm, you were using pip and switching to conda solved it? or what did the trick?

[Steven Albers]

Hello Andrea – Indeed I switched to the conda build as you were suggesting and that worked successfully.

 

Take care,

 

Steve

 

To view this discussion, visit https://groups.google.com/d/msgid/pytroll/76344bd0-10bb-4456-a360-3bce48da822an%40googlegroups.com.