SATPY_CONFIG_PATH bug in Windows 10

[lobsiger...@gmail.com]

Dear Developers

Windows 10 64Bit PyTROLL/SatPy has a severe problem with SATPY_CONFIG_PATH. I first heard of that bug here:

https://groups.io/g/MSG-1/topic/85002450#31749

As I normally use PyTROLL/SatPy under Debian GNU/Linux I could not reproduce the problem. BACKGROUND INFO:

I wrote an EUMETCast PyTROLL/SatPy Starter Kit 3.0 that works under GNU/Linux amd64 as well as Windows 10 64Bit.
There are two main directories that contain scripts, tools, data and results. These are EMCtools and EMCdata.
Under Windows 10 the default install is C:\EMCtools and D:\EMCdata. GNU/Linux has no drive letters of course.
There is a private config directory C:\EMCtools\pppconfig and the variable SATPY_CONFIG_PATH is set accordingly.
Raw satellite data from EUMETCast may be on the same PC or can be read via network from a special receiver PC.

Each image generation script starts in C:\EMCtools\USRscripts as "python myscript.py timeslot" (or date for LEOs).
The python script then changes directory to D:\EMCdata\tmpdirs\xsat. It copies and maybe decompresses all relevant
data for the timeslot (or a LEO pass) to this scratch dir. Then it globs the needed data segments and stores the
resulting image under directory D:\EMCdata\images\Satellite. The problem is now that private area definitions are
not found anymore. The output below shows that SatPy searches for D:\EMCtools\pppconfig even while the environment
variable SATPY_CONFIG_PATH is correctly pointing to C:\EMCtools\pppconfig. In fact if I make under Windows 10 a
D:\EMCtools\pppconfig and put a copy of my areas.yaml in it, then the script works! Just tested with SatPy v030.

...
   raise AreaNotFound('Area "{0}" not found in file "{1}"'.format(area_name, area_file_name))
pyresample.area_config.AreaNotFound: 'Area "isomrss" not found in file "[\'C:\\\\Users\\\\lobsiger\\\\miniconda3\\\\envs\\\\lobsiger\\\\Lib\\\\site-packages\\\\satpy\\\\etc\\\\areas.yaml\', \'D:\\\\EMCtools\\\\pppconfig\\\\areas.yaml\']"'
...

This problem is rather new. It must have started with the change from PPP_CONFIG_DIR to SATPY_CONFIG_PATH.
The environment variable SATPY_CONFIG_PATH that contains 'C:/EMCtools/pppconfig' seems to be converted to
a list ['C','/EMCtools/pppconfig'] and since my scripts do all their work in a scratch directory on drive D:
'/EMCtools/pppconfig' is interpreted as relative path which is then identical to 'D:/EMCtools/pppconfig'.

IN SHORT: The use of the colon ':' as a delimiter in environment variable SATPY_CONFIG_PATH is a problem
for Windows 10 users that have more than one drive. It breaks my EUMETCast PyTROLL/Satpy Starter Kit 3.0.


Best regards,
Ernst

P.S.
Most Windows users of my kit are beginners that will get stuck as soon as they update SatPy from an older
version. They will not find a workaround like using satpy.config.set(config_path=['C:/EMCtools/pppconfig']).

[David Hoese]

Hi Ernst,

Thanks. This is a major bug and major oversight on my part. Satpy
interprets that environment variable like the *nix PATH variable and
converts it to a list. This is widely used behavior so I don't want to
completely get rid of the `:` delimiter here, so I'll have to find
another way to handle this on Windows.

Things to try in the short term:

1. Replace "C:/" in your path to "/c/" and see if that works. I don't
have the time right now (or a Windows machine) to test how Python treats
this, but will later.
2. Set SATPY_CONFIG_PATH="['C:/EMCtools/pppconfig']". This should make
Satpy skip over the ":" splitting logic and go directly to representing
it as a list (which is what we want).

I'll reply here when I fix this or at a minimum when I document this
better in the documentation. Let me know how it goes.

Dave

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/af0d661d-4f2a-4baf-ac63-06658db0a1aan%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/af0d661d-4f2a-4baf-ac63-06658db0a1aan%40googlegroups.com?utm_medium=email&utm_source=footer>.

[lobsiger...@gmail.com]

Hi Dave,

thanks for the fast reply. I have tried to escape the Windows drive colon in my SATPY_CONFIG_PATH
that points to C:/EMCtools/pppconfig in many ways to no avail. What worked for me is to delete the
environment variable and set in my scripts (impossible for users to edit all my scripts like that!):

...
from satpy import Scene, config                                                # I had to add 'config' in my scripts
...
toodrp = 'C:'                                                                                    # this is used as path for EMCtools
...
config.set(config_path=['toodrp+'/EMCtools/pppconfig'])     # Add this configuration line further down
...


The easiest way was your proposal 2. where I set SATPY_CONFIG_PATH with the Windows env app to
['C:/EMCtools/pppconfig']  Giving this list indeed skipped the satpy code that has the colon problem.

I think the easiest way for you is to just document that GNU/Linux users can set for SATPY_CONFIG_PATH
/here:/there or directly ['/here','/there'] while Windows users *must* use e.g. ['C:/here','D:/there']


Best regards,
Ernst

P.S.
I'll make a note on MSG-1 that Windows 10 users of my EUMETCast PyTROLL/SatPy Starter Kit 3.0
that run into the ':' problem should set SATPY_CONFIG_PATH directly to  ['C:/EMCtools/pppconfig']

[David Hoese]

Ernst,

Gerrit on the Pytroll slack had the suggestion to use Python's
"os.pathsep" which is a colon on *nix systems and semicolon on Windows.
This should make your previous usage work, but will require the new
version of Satpy (that I have yet to fix and release).

Dave

> <https://groups.google.com/d/msgid/pytroll/af0d661d-4f2a-4baf-ac63-06658db0a1aan%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/af0d661d-4f2a-4baf-ac63-06658db0a1aan%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/851bf508-6336-473b-97bc-dc374712692dn%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/851bf508-6336-473b-97bc-dc374712692dn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[David Hoese]

Ernst,

I just released Satpy 0.30.1 to PyPI (should be available on conda-forge
tomorrow) which fixes this issue. Your original usage (even of
PPP_CONFIG_DIR, although deprecated) should work with this version. Let
me know if you are able to try it out and how it goes. Thanks.

Dave

[lobsiger...@gmail.com]

Dave,

I updated with conda to Satpy 0.30.1 and tested my scripts under Windows 10 PRO 64Bit:

Everything works again including two semicolon delimited paths (as you use os.pathsep() now) setting SATPY_CONFIG_PATH=D:/here; D:/there in the Win env app.

Even the old style PPP_CONFIG_DIR still works, reminding the user that he should change to SATPY_CONFIG_PATH now. I'll make a note on the MSG-1 list as well.

Thanks for your help,

Ernst