How to read MSG native format data

[Basit Khan]

Hi there,
I am new to both this user group and satellite data processing.  I got some bulk data from eumetsat in Native (.nat) format. EUMETSAT provides bulk data only in this format. I wonder how can i read these data.

Any suggestion, information, script would be highly appreciated.

Thanks and best regards,

Bas.

[Adam Dybbroe]

Bas,

We prototyped a native reader last autumn, and we came quite far. You
are welcome to test it using the msg_native branches in both mpop and
mipp. There is still some important functionality lacking, such as
slicing. I don't think you can yet load only a subsection of the
MSG-disk. You kind of need to read the full disk, as far as I remember.

Reading and making a quicklook goes as usual, and you can do either of
the two ways shown here:

from mpop.utils import debug_on
debug_on()
from datetime import datetime
from mpop.satellites import GeostationaryFactory

time_slot = datetime(2014, 7, 24, 9, 45)
glbd = GeostationaryFactory.create_scene("meteosat", "10", "seviri",
time_slot)
glbd.load([0.8, 1.6, 3.9, 10.8, 13.2])
img = glbd.image.green_snow()
img.show()


or


from mpop.utils import debug_on
debug_on()

from mpop.satellites import GeostationaryFactory

filename =
"/home/a000680/data/msg_native/MSG3-SEVI-MSG15-0100-NA-20140724095743.719000000Z-NA.nat"

glbd = GeostationaryFactory.create_scene("meteosat", "10", "seviri", None)
glbd.load([0.8, 1.6, 3.9, 10.8, 13.2], filename=filename)
img = glbd.image.green_snow()
img.show()


My meteosat10.cfg file looks like this:

[satellite]
satname = 'meteosat'
number = '10'
instruments = ('seviri',)
projection = 'geos(0.0)'
proj4_params = proj=geos lon_0=0.00 lat_0=0.00 a=6378169.00 b=6356583.80
h=35785831.00

[seviri-level1]
format = 'mipp/native/MSG'
filename = MSG3-SEVI-MSG15-0100-NA-%Y%m%d%H*.nat
dir = /home/a000680/data/msg_native


[seviri-level2]
format = 'mipp_msgnative.NativeReader'

...


I was trying to recall how the reprojection goes, but right now it
doesn't seem to work as it should, so something with the geolocation
needs to be fixed. But tell me once you have come as far as being able
to run the examples above!

-Adam

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Numerical models and Remote Sensing,
Core Services, Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org
nwcsaf.smhi.se
www.smhi.se

[Basit Khan]

Hi Adam,

Thank you very much indeed for a detailed response. I actually have the subsection of the MSG disk (if i understand you correctly) i.e. lats 8N to 37N and Longs 30E to 62E (the middle eastern region). I will give it a go and update you on it. I am not a python guy, actually its the first time i am gonna use it. So, probably the first thing is to install python and related packages mentioned on the webpage.

Thanks and best regards,
Bas.

[Adam Dybbroe]

Hi!

Perhaps you could upload an example file somewhere where I can fetch it.
This way I might be able to test a bit and see where and what remains to
be done. That would potentially save you some time.

-Adam

> > an email to pytroll+u...@googlegroups.com <javascript:>
> > <mailto:pytroll+u...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> Adam Dybbroe,
> Satellite Remote Sensing Scientist,
> Numerical models and Remote Sensing,
> Core Services, Swedish Meteorological and Hydrological Institute (SMHI)

> www.pytroll.org <http://www.pytroll.org>
> nwcsaf.smhi.se <http://nwcsaf.smhi.se>
> www.smhi.se <http://www.smhi.se>

[Martin Felder]

Hi Adam,

I'm having the same issue here - lotsa files in native format...
Thank you for your suggestions, I got the reader to work after finding out that I need to add the channel number to the cfg entries for the channels (it's not there in the templates):

[seviri-3]
frequency = (1.5, 1.64, 1.78)
resolution = 3000.403165817
name = 'IR_016'
size = (3712, 3712)
channel_number = 3

With this, reading and displaying a full disk works fine, but as you mentioned something is missing for proper reprojection. I get something like

...
[DEBUG: 2015-12-07 11:22:16 : mpop.scene] Successfully loaded: set(['IR_120', 'IR_108', 'IR_087'])
[INFO: 2015-12-07 11:22:16 : mpop.projector] Computing projection from meteosatseviri(array([-5570.24853516], dtype=float32), array([-5567.24804688], dtype=float32), array([ 5567.24804688], dtype=float32), array([ 5570.24853516], dtype=float32))(3712, 3712) to euron1...
[INFO: 2015-12-07 11:22:25 : mpop.channel] Projecting channel IR_087 (8.700000μm)...
...

but then the resulting image comes up all black, regardless of what projection I choose. Any idea what is going wrong here?

Best,
Martin

[Adam Dybbroe]

Martin,

On 12/07/2015 11:31 AM, Martin Felder wrote:
> Hi Adam,
>
> I'm having the same issue here - lotsa files in native format...

"Same issue"? Please, I am lost, can you just help me remembering the
discussion?

You are reading native-formatet SEVIRI data?

Which versions/branches of mipp and mpop are you using?

Can you display the unprojected data?

Can you send the couple of lines you use when instantiating the scene
object and loading the data?

Best regards
-Adam


>
> Best,
> Martin

[Martin Felder]

Hi Adam,

thanks for the quick reply.

"Same issue"? Please, I am lost, can you just help me remembering the
discussion?

 

No worries, the "issue" is just that I somehow have to deal with the MSG-SEVIRI "native" format in an automated fashion (because of the volume of data we received).

Which versions/branches of mipp and mpop are you using?

 I just checked out the current origin/msg-native branch for both projects from github this morning. Latest commits seem to be from Sat Dec 13 2014 (mipp) and Mon Dec 8 2014 (mpop).

Can you display the unprojected data?

Yes.

Can you send the couple of lines you use when instantiating the scene
object and loading the data?

time_slot = datetime(2013, 7, 11, 17, 30)
glbd = mpop.satellites.GeostationaryFactory.create_scene("meteosat", "10", "seviri", time_slot)
glbd.load([8.7, 10.8, 12.0])
local_data = glbd.project("euro_north")
img = local_data.image.fog()
img.show()
 
Thanks,
Martin

[Adam Dybbroe]

Hi,

On 12/07/2015 02:24 PM, Martin Felder wrote:

>
> time_slot = datetime(2013, 7, 11, 17, 30)
> glbd = mpop.satellites.GeostationaryFactory.create_scene("meteosat",
> "10", "seviri", time_slot)
> glbd.load([8.7, 10.8, 12.0])
> local_data = glbd.project("euro_north")
> img = local_data.image.fog()
> img.show()

Ok, that's strange. Have you tried with some pre-defined area (in the
area.def file)? Like eurol or euro4 for instance? Assuming the data
covers Northern Europe.

Maybe your native formatet data are not the full MSG disk, but rather a
subsection?

We started last year refactoring MIPP in order to more effectively take
onboard other formats like Himawari and MSG native. But we didn't get
time unfortunately at this past Pytroll developers week to take up the
thread again. This work is found in the restructure branch of mipp, and
was last worked on in June this year.

-Adam

[Martin Felder]

Hmm, no, the data definitely covers the full disk, I can display it without problems.
I've tried different projections and they all come out all black - although the size of the output images varies and seems to fit the the region selected.
It also makes no difference if I do something like

local_data.image.channel_image(10.8).show()

Could you give me a hint where in the code to look?
Otherwise maybe I should just reinstall the whole package on a different machine. Maybe a system library like PROJ4 is too old?

--Martin

[Martin Felder]

Hi Adam,

I think found the bug. When looking at the channel areas, they were read in as

Area ID: meteosatseviri(array([-5570.24853516], dtype=float32), array([-5567.24804688], dtype=float32), array([ 5567.24804688], dtype=float32), array([ 5570.24853516], dtype=float32))(3712, 3712)

which looked a bit strange to me. So in NativeImage.get_header I changed (after some experimentation)

self.area_extent = area_extent

into

self.area_extent = tuple([x[0] * 1000. for x in area_extent])

and viola! - it works.
Cheers, Martin

[Itai Bibi]

Hi Adam,

I know it's been over a year since this thread started but I'm very interested in the code you guys developed.

I've downloaded and set up mpop and mipp on python 2.7 in windows.

If it was up to me, I would have worked on a linux pc, but I'm trying to help a researcher at my institute and the environment is windows.

I tried to run your "example code" on the following file: MSG2-SEVI-MSG15-0100-NA-20160701090416.403000000Z-20160701090434-1193744.nat

This is what I got:

[DEBUG: 2016-12-09 09:19:37 : mpop.satellites] Looking for config file C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\etc\meteosat10.cfg

[ERROR: 2016-12-09 09:19:37 : mpop.satellites] Can't find config file C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\etc\meteosat10.cfg, is PPP_CONFIG_DIR set?

[DEBUG: 2016-12-09 09:19:37 : mpop.satellites] Instrument: seviri

[DEBUG: 2016-12-09 09:19:37 : mpop.satellites] No custom composites provided i config file...

[DEBUG: 2016-12-09 09:19:37 : mpop.satellites] No custom composites provided i config file...

[WARNING: 2016-12-09 09:19:37 : mpop.scene] Can't load data, no config file for meteosat10

Traceback (most recent call last):

  File "natFileReader.py", line 9, in <module>

    img = glbd.image.green_snow()

  File "C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\mpop\instruments\visir.py", line 337, in green_snow

    self.check_channels(0.635, 1.63, 10.8)

  File "C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\mpop\compositer.py", line 61, in check_channels

    self._data_holder.check_channels(*args)

  File "C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\mpop\scene.py", line 561, in check_channels

    if not self[chan].is_loaded():

  File "C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\mpop\scene.py", line 280, in __getitem__

    raise KeyError("No channel corresponding to " + str(key) + ".")

KeyError: 'No channel corresponding to 0.635.' 

the only config file I saw in this folder was geo_image.cfg

Am I missing something with the installation of mpop?

If you want me to upload the .nat file I got from EUMETSAT, there is no problem.

Thanks

[Panu Lahtinen]

On 09.12.2016 11:47, Itai Bibi wrote:
>
> C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\etc\meteosat10.cfg,
> is PPP_CONFIG_DIR set?

> the only config file I saw in this folder was geo_image.cfg
> Am I missing something with the installation of mpop?
> If you want me to upload the .nat file I got from EUMETSAT, there is no
> problem.

Hi,

You'll need to have meteosat10.cfg file in your PPP_CONFIG_DIR. You can
use this file as a template (just rename to "meteosat10.cfg, or use
Meteosat-10 as the platform_name and remove the .template from the
filename):

https://github.com/pytroll/mpop/blob/master/etc/Meteosat-10.cfg.template


Panu

--
Finnish Meteorological Institute
Erik Palménin aukio 1, P.O.Box 503, FIN-00101 Helsinki, Finland
Room: 2B10a, Phone: +358 50 380 3261

[Itai Bibi]

Thank you very much for the quick response.

I run the following line: glbd = GeostationaryFactory.create_scene("meteosat", "10", "seviri", None)

It yielded an error with the time_slot:

  File "C:\python27\lib\site-packages\mpop-1.3.0-py2.7.egg\mpop\satin\mipp_xrit.py", line 151, in load_generic

    glob_epi = satscene.time_slot.strftime(

AttributeError: 'NoneType' object has no attribute 'strftime'

This makes sense, since I've inserted time_slot = None (the forth argument)

But trying to run the same line with only 3 arguments yields an error as well. 

So I've printed the doc which is attached here:

__doc__

This is the satellite instrument class. It is an abstract channel

container, from which all concrete satellite scenes should be derived.

The constructor accepts as optional arguments the *time_slot* of the scene,

the *area* on which the scene is defined (this can be used for slicing of

big datasets, or can be set automatically when loading), and *orbit* which

is a string giving the orbit number. 

These are my questions:

1) Since currently I'm trying to read a file that I already have, what should I do with the time_slot?

2) Is slicing according to area possible?  if so, what should be the area's format?

3) When running the code with some time_slot, and without a specific file (To see what happens) I get the following error:

  File "C:\python27\lib\site-packages\mipp-1.0.0-py2.7.egg\mipp\xrit\sat.py", line 126, in load

    os.path.join(opt['dir'], filename_pro)) % val)

ValueError: Invalid format string

Surely I'll look into it, but any farther help will be great.

Thanks again 

[Itai Bibi]

Regarding my third question, I've traced it back to the meteosat.cfg

Both opt['dir'] and filename_pro get their value from [seviri-level1]

I've changed the dir name to the place in which I'm holding the data, but I think the file format does not match the native format.

It start's with an H which looks more like the HRIT format.

Any suggestions would be very much appreciated.

Itai

[borodin...@gmail.com]

Dear Adam,

Could you help me to interpret the file naming convention of eumetsat files:

prefix        = "MSG3-SEVI-MSG15-0100-NA-"

start_time =  20140724095743 #  %Y%m%d%H%M%S

 ?             = .719000000Z-NA

In particular, what represents the set of characters after <.> in eumetsat files?

Thank you in advance for your help.

Best regards,

Valeria

> an email to pyt...@googlegroups.com
> <mailto:pyt...@googlegroups.com>.

[Sauli Joro]

hi valeria,

please look at the `seviri_l1b_native.yaml`-file in satpy.

20140724095743.719000000Z is the processing time

NA would be the order ID, if the data was ordered from EUMETSAT data Centre

best regards,

sauli

[Valeria Borodin]

Dear Sauli,

Thank you for your help.

Best regards,
Valeria