Hi sapty team

[Frank]

Hi  Everyone!

I'm a Canadian engineering student. I have recently configured my pc to recieve a EUMETSat data with EUMETCast dissemination. For my next uni project i should build an algorithm that detect clouds system through segmentation ,and after extract geographical info of this . Example: detect a storm cell with an ellipse,and after give geo information.

Which is the best way in Python for do this?

Can Pytroll help me?

Thanks in advance

Frank~

[David Hoese]

HI Frank,

Satpy can help with reading the data from the files, but does not have anything built in for cloud detection of the sort that you are talking about. Or at least I'm not aware of anything like this. That type of functionality gets down to the real science/physics of working with the data which could have many different ways of implementing it. Many of the choices in implementing it could heavily impact how it performs. For example, Satpy uses dask underneath to lazy-load data and save on memory and process data in parallel over multiple threads. If you load the full data array and then use numpy (or other library) to do the processing it may be easier to describe and perform the cloud detection, but will use a lot of memory to hold those arrays all at once.

Other people may respond with their own experience or suggestions, but my guess is that a solution to your problem would be that you use Satpy to load the data, then either implement your own algorithm (dask-friendly or pure numpy) and/or you find a dask-friendly or numpy-based library that does most of the work for you (ex. scikit-image's segmentation functionality: https://scikit-image.org/docs/stable/api/skimage.segmentation.html).

If you have more detailed questions about how to get the data out of the Satpy library let me know.

Dave

--
You received this message because you are subscribed to the Google Groups "pytroll" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/pytroll/8acf8e12-d3b1-4910-aed7-d32fdf159753n%40googlegroups.com.

[Frank]

Hi Dave, thanks for your response,

My message took long to display in the group so I did some research using AI, this is what I got.

I'm still trying to process this. Thanks, you just made it easy for me to get through it.

HELLO! GREAT PROJECT, AND YES PYTHON + PYTROLL ARE PERFECT FOR THIS.

*1. PYTROLL: YES IT CAN HELP A LOT*

PYTROLL IS THE MAIN TOOLKIT FOR EUMETSAT DATA. SINCE YOU ARE ALREADY USING EUMETCAST, PYTROLL WILL HANDLE THE HARDEST PARTS:

- *SATPY*: READS MSG/SEVIRI DATA FROM EUMETCAST, DOES GEOLOCATION, CALIBRATION, AND REPROJECTION TO LAT/LON GRIDS AUTOMATICALLY.

- *PYRESAMPLE*: REGRIDS DATA TO REGULAR LAT/LON IF NEEDED FOR YOUR ALGORITHM.

WITHOUT PYTROLL YOU WOULD HAVE TO WRITE GEOLOCATION CODE YOURSELF FOR SEVIRI. THAT IS PAINFUL.

*2. BEST PYTHON WORKFLOW FOR CLOUD/STORM SEGMENTATION*

STEP 1: LOAD + PREPROCESS WITH SATPY

FROM SATPY.SCENE IMPORT SCENE

SCN = SCENE(FILENAMES = FILES, READER = 'SEVIRI_L1B_NATIVE')

SCN.LOAD(['IR_108', 'IR_087', 'WV_062']) # INFRARED CHANNELS ARE KEY FOR CLOUDS

SCN = SCN.RESPAMPLE('EUROPE4KM') # GET REGULAR LAT/LON GRID

STEP 2: SEGMENTATION - PICK ONE BASED ON YOUR NEEDS:

- *THRESHOLD + CONNECTED COMPONENTS*: FASTEST. USE IR_108 BRIGHTNESS TEMP. COLD PIXELS = HIGH CLOUDS/STORMS. `SKIMAGE.MEASURE.LABEL` TO FIND CLOUD OBJECTS.

- *WATERSHED SEGMENTATION*: GOOD FOR SEPARATING TOUCHING CLOUD CELLS. `SKIMAGE.SEGMENTATION.WATERSHED`.

- *ML APPROACH*: U-NET WITH TORCH/TENSORFLOW IF YOU HAVE LABELED TRAINING DATA. BETTER FOR COMPLEX SYSTEMS.

FOR STORM CELLS: MOST STUDENTS START WITH IR_108 < 235K THRESHOLD, THEN USE `SKIMAGE.MEASURE.REGIONPROPS` TO GET ELLIPSE PARAMETERS.

STEP 3: EXTRACT GEO INFO + FIT ELLIPSE

FROM SKIMAGE.MEASURE IMPORT REGIONPROPS

PROPS = REGIONPROPS(LABELED_CLOUDS, INTENSITY_IMAGE = IR_108)

FOR P IN PROPS:

    LAT_CENTROID = LAT_GRID[P.CENTROID[0], P.CENTROID[1]]

    LON_CENTROID = LON_GRID[P.CENTROID[0], P.CENTROID[1]]

    ELLIPSE_MAJOR = P.MAJOR_AXIS_LENGTH * PIXEL_SIZE_KM

    ELLIPSE_MINOR = P.MINOR_AXIS_LENGTH * PIXEL_SIZE_KM

    ORIENTATION = P.ORIENTATION

`REGIONPROPS` GIVES YOU CENTROID, BOUNDING BOX, AREA, AND ELLIPSE FIT DIRECTLY. LAT/LON GRIDS COME FROM `SCN['IR_108'].ATTRS['AREA'].LONS` AND `.LATS`.

*3. LIBRARIES YOU NEED*

`SATPY`, `PYRESAMPLE`, `SCIPY`, `SKIMAGE`, `NUMPY`, `XARRAY`, `CARTOPY` FOR MAPS.

*4. TIP FOR STORM CELLS*

USE "CLOUD TOP TEMPERATURE" FROM IR_108 + "CLOUD TOP HEIGHT" IF YOU HAVE NWP DATA. OVERSHOOTING TOPS < 210K ARE USUALLY SEVERE STORMS. THEN FIT ELLIPSE TO THE COLD PIXEL CLUSTER.

Best regards 

[David Hoese]

That seems reasonable. The biggest thing I'd recommend considering is whether or not you need to resample data. This would mostly depend on if you need to combine the SEVIRI data with data from other sources (ex. weather models, other satellite instruments, etc). If you only use SEVIRI data then keeping it in its current geostationary projection will save a ton of processing time and memory usage. This is especially important if you plan on processing a lot of data in the past. If you don't need to resample the data then instead of using longitude and latitude coordinates you would use the X and Y metered coordinates of the geostationary grid. This is accessible from the AreaDefinition stored in the `scn["IR_108"].attrs["area"]` for the data loaded by Satpy, but I forget the exact method name at the moment.

The other consideration would be if you only need to detect clouds over a specific region of the overall SEVIRI image. In that case you could use `Scene.crop` from Satpy to cut off large chunks of data and reduce how much data the segmentation steps need to process.

Dave

To view this discussion, visit https://groups.google.com/d/msgid/pytroll/85a2397c-248c-47a0-b171-872cb1fb31b2n%40googlegroups.com.

[Frank]

Thanks very much Dave

 this is really helpful.