Advice on Satpy custom reader for AVHRR Polar Pathfinder FCDR (filename patterns & time/area)

[li zicong]

Hi Pytroll team,

I’m adding a Satpy reader for the AVHRR Polar Pathfinder FCDR (5-km EASE-Grid, twice-daily 14:00/04:00 LST composites). Prototype works (Scene loads reflectances, BTs, angles, lon/lat), but I’d like guidance on these points:

1. Reader name & file_patterns
   Example filename: Polar-APP_v02r00_Nhem_1400_d20240701_c20240704.nc.

   • Recommended reader name for this composite product? (avhrr_fcdr_polarpathfinder_nc vs app_fcdr_nc?)

   • Best practice for file_patterns: should I parse {region}/{lst}/{date} from filename, or keep patterns loose and read times from globals?

2. Time metadata for composites
   These are day composites centered at local solar time (14:00/04:00).

   • Prefer start_time == end_time == nominal LST or a full day span? Any examples for composite products?

3. Area/CRS
   Data are EASE-Grid (polar) 5 km.

   • Better to expose lon/lat and let Satpy build a SwathDefinition, or provide a fixed AreaDefinition (CRS/extent) for speed/clarity? Any recommended EASE-Grid params/WKT to copy?

4. Standard names
   Using toa_bidirectional_reflectance (units “%”) and toa_brightness_temperature (K). For angles I use sza, raa, scan_angle. Are these names consistent with Satpy conventions?

Happy to share my YAML/FileHandler snippet if useful

Thanks a lot!

Best,
cong

[David Hoese]

Hi Cong,

Thanks for reaching out. Is this reader something you plan on contributing to Satpy directly? We may be able to give further assistance if we see the code. But anyway...

1a. This is a tough one. I'm sure you've seen the documentation about how to name a reader (https://satpy.readthedocs.io/en/stable/dev_guide/custom_reader.html#naming-your-reader). I'm not sure what to recommend for the reader name as I'm not familiar with these products. Normally I would think it'd be something like "avhrr_l2_fcdr". Or would these be considered "level 3" so "l3" instead? If there is no other official format for these files then I don't think the "_nc" is needed (for example, are these also available in HDF5? or some other format?).

1b. For file patterns, I prefer to specify everything as it gives the human readers that look at this later more information on what is in the filename. That doesn't mean that you couldn't or shouldn't read information from global attributes. Whatever is most accurate should probably be used.

2. Good question. I could see an argument for both. Maybe start with start/end time being equal to the LST.

3. AreaDefinition. Usually EASE-Grids have some definition that includes an EPSG code or at least some PROJ.4 parameters.

4. The reflectance and BT names are good enough. There is currently a discussion going on with the CF community about better standard names for the different variants of reflectance products. Do you know if your reflectances include the "/ cos(SZA)"? Other readers usually use the longer CF standard name for "solar_zenith_angle" and the like (see https://github.com/pytroll/satpy/blob/main/satpy/etc/readers/viirs_sdr.yaml for example). I *think* these are the CF "standard_name" attributes that you would see in a normal NetCDF file. If you're asking about what should the product be named (versus the standard name attribute in the metadata) then I'm not sure it matters in the long run, but I prefer the longer "solar_zenith_angle", "satellite_azimuth_angle", and such.

Hope this helps. Let me know if I need to clarify anything. Others in this community can comment if they have differing opinions. Thanks.

Dave

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/bb1bb5dd-3948-4935-a22f-e78968e26d44n%40googlegroups.com.

[Frank]

Hi Izic

For your AVHRR Polar Pathfinder (APP) reader, the following recommendations align with Satpy's architectural best practices and conventions.

1. Reader Name & File Patterns 

Recommended Name: avhrr_app_fcdr_nc.

Reasoning: Satpy follows a <sensor>_<program/software>_<level>_<format> scheme. Using avhrr (sensor) and app (software/algorithm) is clear and consistent with existing readers like avhrr_l1b_gac or clavrx. 

File Patterns: Parse {region}, {lst}, and {start_time} from the filename.

Best Practice: While global attributes are reliable, parsing from the filename allows Satpy’s find_files_and_readers to filter by time before opening any files, which significantly improves performance when browsing large archives.

Loose Pattern: Keep the file pattern specific enough to avoid collisions with other readers but generic enough for minor version changes (e.g., Polar-APP_v.*_{region}_{lst}_d{start_time:%Y%m%d}.nc). 

2. Time Metadata for Composites 

Recommendation: Set start_time and end_time to the nominal local solar time (14:00/04:00) for the specific date.

Reasoning: Satpy uses these times for critical geometry calculations like solar zenith angles. If the data represents a "slice" centered at 14:00, treating it as a single point in time is standard for L3-style products to ensure consistent re-projections and enhancements.

Attribute Structure: Include the "full day span" or processing window in a time_parameters sub-dictionary as observation_start_time and observation_end_time for transparency. 

3. Area/CRS: EASE-Grid (5 km)

Recommendation: Use a fixed AreaDefinition.

Speed/Clarity: Exposing 2D lon/lat arrays forces Satpy to use a SwathDefinition, which is computationally expensive and memory-intensive. Since EASE-Grid is a well-defined projection, an AreaDefinition allows for significantly faster resampling and much smaller file sizes if you save to NetCDF/GeoTIFF. 

Recommended Parameters: The APP 5 km EASE-Grid typically uses the following Proj parameters:

North (Nhem): +proj=laea +lat_0=90 +lon_0=0 +a=6371228 +units=m

South (Shem): +proj=laea +lat_0=-90 +lon_0=0 +a=6371228 +units=m

Note: Ensure your area_extent is defined in meters based on the 5 km grid spacing and center offsets. 

4. Standard Names

Your current names are nearly perfect, but slight adjustments will ensure full compatibility with Satpy's built-in composites (like true_color or natural_color): 

Reflectance: toa_bidirectional_reflectance (units: %) — Correct.

Brightness Temp: toa_brightness_temperature (units: K) — Correct.

Angles:

Use solar_zenith_angle (instead of sza).

Use solar_azimuth_angle and sensor_azimuth_angle to derive relative azimuth if needed.

Use sensor_zenith_angle (instead of scan_angle) to ensure modifiers like get_cos_sza or parallax correction work out of the box.