plt.gca().coastlines() not working on MONSooN

[Seonaid Dey]

Hi,

I am new to using Iris and am working on the MONSooN post processor. I have files containing UK data which I can read in using iris.load. I can also plot the fields using iris.quickplot. However, when I try to then plot coastlines using plt.gca().coastlines() I get a load of errors finishing with 

File "/usr/local/sci/lib/python2.7/zipfile.py", line 761, in _RealGetContents

raise BadZipfile, "File is not a zip file"

BadZipfile: File is not a zip file

Is this problem due to the UK data being in a rotated pole coordinate system or is there something wrong with the iris setup on MONSooN? I have also tried the routine on block plots from http://scitools.org.uk/cartopy/docs/latest/matplotlib/advanced_plotting.html (which takes the data from the correct rotated pole coordinate system) and get exactly the same error.

Any help gratefully received!

Thanks,

Seonaid

[Phil Elson]

Sounds like a cartopy installation problem. Which version of cartopy are you running? With the Met Office's installation we have a directory of pre-downloaded shapefiles which we configure cartopy to point to. Whilst this isn't necessary for cartopy, it does avoid individual users filling up their homespace with shapefiles which are universally used - so this is definately something that the MONSooN sys-admins should be looking at. (More details are available at http://scitools.org.uk/cartopy/docs/latest/cartopy.html#cartopy.config)

In the meantime, the cartopy download mechanism should be working for individuals, but clearly isn't for your installation. I was expecting that the file downloaded from Natural Earth was no longer a zip file, but I've just tested the downloader on my own machine and everything was working without a hitch, so I wonder if you are behind a firewall/proxy which is intercepting the download? Did you get the warning:

DownloadWarning: Downloading: http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip
  warnings.warn('Downloading: {}'.format(url), DownloadWarning)

? Are you able to try downloading that file, and inspecting whether it is actually a zip file when it arrives to you? Something like:

$> wget http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip
$> file ne_110m_coastline.zip
ne_110m_coastline.zip: Zip archive data, at least v2.0 to extract

Thanks,

Phil

[ScottHosking]

I'm also having this issue on JASMIN (jasmin-sci1), i.e. I get the same error message message when I use the line "plt.gca().coastlines()".
 
However, I am able to retrieve the ne_110m_coastline.zip file myself using wget.

The installed version of cartopy is '0.7.0'

[ScottHosking]

sorry, I should have been more clear.  The error I get is:

/usr/lib/python2.7/site-packages/cartopy/io/__init__.py:255: DownloadWarning: Downloading: http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip
  warnings.warn('Downloading: {}'.format(url), DownloadWarning)

[Andrew Dawson]

Hi Scott

 

Are you just seeing this:

 

/usr/lib/python2.7/site-packages/cartopy/io/__init__.py:255: DownloadWarning: Downloading: http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip
  warnings.warn('Downloading: {}'.format(url), DownloadWarning)

 

or do you get an error as well (or does your script not work or something else bad happen)?

 

The above is just a warning letting you know that the data for the coastline is being downloaded, if it is successful in downloading your plot should work as normal (except for a delay of downloading data).

 

If not then get back to us and we'll see what we can do.

[ScottHosking]

Hi Andrew,

Thanks for the reply. With the line "plt.gca().coastlines()" I do see the warning you mentioned but I also get the errors I've included below. If I include "plt.gca().coastlines()" then plt.savefig does not produce a plot, whereas if I remove it it works fine (albeit without coastlines).

In [3]: execfile('test1.py')
---------------------------------------------------------------------------
URLError                                  Traceback (most recent call last)
<ipython-input-3-b05d02b86493> in <module>()
----> 1 execfile('test1.py')

/home/users/shosking/Iris/test1.py in <module>()
     14 plt.gca().coastlines()
     15
---> 16 plt.savefig('test1.png')

/usr/lib/python2.7/site-packages/matplotlib/pyplot.pyc in savefig(*args, **kwargs)
    470 def savefig(*args, **kwargs):
    471     fig = gcf()
--> 472     return fig.savefig(*args, **kwargs)
    473
    474 @docstring.copy_dedent(Figure.ginput)

/usr/lib/python2.7/site-packages/matplotlib/figure.pyc in savefig(self, *args, **kwargs)
   1361             kwargs.setdefault('edgecolor', rcParams['savefig.edgecolor'])
   1362
-> 1363         self.canvas.print_figure(*args, **kwargs)
   1364
   1365         if transparent:

/usr/lib/python2.7/site-packages/matplotlib/backend_bases.pyc in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs)
   2091                 orientation=orientation,
   2092                 bbox_inches_restore=_bbox_inches_restore,
-> 2093                 **kwargs)
   2094         finally:
   2095             if bbox_inches and restore_bbox:

/usr/lib/python2.7/site-packages/matplotlib/backends/backend_agg.pyc in print_png(self, filename_or_obj, *args, **kwargs)
    489
    490     def print_png(self, filename_or_obj, *args, **kwargs):
--> 491         FigureCanvasAgg.draw(self)
    492         renderer = self.get_renderer()
    493         original_dpi = renderer.dpi

/usr/lib/python2.7/site-packages/matplotlib/backends/backend_agg.pyc in draw(self)
    437
    438         try:
--> 439             self.figure.draw(self.renderer)
    440         finally:
    441             RendererAgg.lock.release()

/usr/lib/python2.7/site-packages/matplotlib/artist.pyc in draw_wrapper(artist, renderer, *args, **kwargs)
     52     def draw_wrapper(artist, renderer, *args, **kwargs):
     53         before(artist, renderer)
---> 54         draw(artist, renderer, *args, **kwargs)
     55         after(artist, renderer)
     56

/usr/lib/python2.7/site-packages/matplotlib/figure.pyc in draw(self, renderer)
    997         dsu.sort(key=itemgetter(0))
    998         for zorder, a, func, args in dsu:
--> 999             func(*args)
   1000
   1001         renderer.close_group('figure')

/usr/lib/python2.7/site-packages/matplotlib/artist.pyc in draw_wrapper(artist, renderer, *args, **kwargs)
     52     def draw_wrapper(artist, renderer, *args, **kwargs):
     53         before(artist, renderer)
---> 54         draw(artist, renderer, *args, **kwargs)
     55         after(artist, renderer)
     56

/usr/lib/python2.7/site-packages/cartopy/mpl/geoaxes.pyc in draw(self, renderer, inframe)
    288
    289         return matplotlib.axes.Axes.draw(self, renderer=renderer,
--> 290                                          inframe=inframe)
    291
    292     def __str__(self):

/usr/lib/python2.7/site-packages/matplotlib/artist.pyc in draw_wrapper(artist, renderer, *args, **kwargs)
     52     def draw_wrapper(artist, renderer, *args, **kwargs):
     53         before(artist, renderer)
---> 54         draw(artist, renderer, *args, **kwargs)
     55         after(artist, renderer)
     56

/usr/lib/python2.7/site-packages/matplotlib/axes.pyc in draw(self, renderer, inframe)
   2084
   2085         for zorder, a in dsu:
-> 2086             a.draw(renderer)
   2087
   2088         renderer.close_group('axes')

/usr/lib/python2.7/site-packages/matplotlib/artist.pyc in draw_wrapper(artist, renderer, *args, **kwargs)
     52     def draw_wrapper(artist, renderer, *args, **kwargs):
     53         before(artist, renderer)
---> 54         draw(artist, renderer, *args, **kwargs)
     55         after(artist, renderer)
     56

/usr/lib/python2.7/site-packages/cartopy/mpl/feature_artist.pyc in draw(self, renderer, *args, **kwargs)
    103         except ValueError:
    104             warnings.warn('Unable to determine extent. Defaulting to global.')
--> 105         geoms = self._feature.intersecting_geometries(extent)
    106
    107         # Project (if necessary) and convert geometries to matplotlib paths.

/usr/lib/python2.7/site-packages/cartopy/feature.pyc in intersecting_geometries(self, extent)
    111             extent_geom = shapely.geometry.box(extent[0], extent[2],
    112                                                extent[1], extent[3])
--> 113             return (geom for geom in self.geometries() if
    114                     extent_geom.intersects(geom))
    115         else:

/usr/lib/python2.7/site-packages/cartopy/feature.pyc in geometries(self)
    178             path = shapereader.natural_earth(resolution=self.scale,
    179                                              category=self.category,
--> 180                                              name=self.name)
    181             geometries = tuple(shapereader.Reader(path).geometries())
    182             _NATURAL_EARTH_GEOM_CACHE[key] = geometries

/usr/lib/python2.7/site-packages/cartopy/io/shapereader.pyc in natural_earth(resolution, category, name)
    242     format_dict = {'config': config, 'category': category,
    243                    'name': name, 'resolution': resolution}
--> 244     return ne_downloader.path(format_dict)
    245
    246

/usr/lib/python2.7/site-packages/cartopy/io/__init__.pyc in path(self, format_dict)
    215         else:
    216             # we need to download the file
--> 217             result_path = self.acquire_resource(target_path, format_dict)
    218
    219         return result_path

/usr/lib/python2.7/site-packages/cartopy/io/shapereader.pyc in acquire_resource(self, target_path, format_dict)
    298         url = self.url(format_dict)
    299
--> 300         shapefile_online = self._urlopen(url)
    301
    302         zfh = ZipFile(StringIO.StringIO(shapefile_online.read()), 'r')

/usr/lib/python2.7/site-packages/cartopy/io/__init__.pyc in _urlopen(self, url)
    254         """
    255         warnings.warn('Downloading: {}'.format(url), DownloadWarning)
--> 256         return urllib2.urlopen(url)
    257
    258     @staticmethod

/usr/lib/python2.7/urllib2.pyc in urlopen(url, data, timeout)
    124     if _opener is None:
    125         _opener = build_opener()
--> 126     return _opener.open(url, data, timeout)
    127
    128 def install_opener(opener):

/usr/lib/python2.7/urllib2.pyc in open(self, fullurl, data, timeout)
    398             req = meth(req)
    399
--> 400         response = self._open(req, data)
    401
    402         # post-process response

/usr/lib/python2.7/urllib2.pyc in _open(self, req, data)
    416         protocol = req.get_type()
    417         result = self._call_chain(self.handle_open, protocol, protocol +
--> 418                                   '_open', req)
    419         if result:
    420             return result

/usr/lib/python2.7/urllib2.pyc in _call_chain(self, chain, kind, meth_name, *args)
    376             func = getattr(handler, meth_name)
    377
--> 378             result = func(*args)
    379             if result is not None:
    380                 return result

/usr/lib/python2.7/urllib2.pyc in http_open(self, req)
   1205
   1206     def http_open(self, req):
-> 1207         return self.do_open(httplib.HTTPConnection, req)
   1208
   1209     http_request = AbstractHTTPHandler.do_request_

/usr/lib/python2.7/urllib2.pyc in do_open(self, http_class, req)
   1175         except socket.error, err: # XXX what error?
   1176             h.close()
-> 1177             raise URLError(err)
   1178         else:
   1179             try:

URLError: <urlopen error [Errno 111] Connection refused>

[Andrew Dawson]

This looks like Cartopy is attempting to download the required shapefiles for the coastline but is failing because it can't establish a connection. It seems likely that you problem might be due to a restriction in the JASMIN system, blocking the download of the data.. I'm not sure though.

 

You said you were able to download the shapefiles with wget, was that on JASMIN too? In the same session you are trying to use Cartopy in? I'm not familiar with JASMIN, but if you are able to download these files manually and put them where Cartopy expects to find them then you should be able to get it working. However, it sounds like it would be worth contacting JASMIN support and talking them through your issue to see if they can help.

[ScottHosking]

Morning Andrew,

This looks like Cartopy is attempting to download the required shapefiles for the coastline but is failing because it can't establish a connection. It seems likely that you problem might be due to a restriction in the JASMIN system, blocking the download of the data.. I'm not sure though.

 

You said you were able to download the shapefiles with wget, was that on JASMIN too? In the same session you are trying to use Cartopy in?

Yes, I was using ipython then logged out back to bash and ran "wget http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip ." This connected and downloaded as expected.
 

I'm not familiar with JASMIN, but if you are able to download these files manually and put them where Cartopy expects to find them then you should be able to get it working. However, it sounds like it would be worth contacting JASMIN support and talking them through your issue to see if they can help.

I'll contact the JASMIN support team today.  Also, as the installed version of cartopy is still v0.7 I'll suggest an upgrade too.  Is there a directory where the shape files should be stored?


 

[Andrew Dawson]

The locations where the data are stored is configurable. The default is most likely to put it in subdirectories of  ~/.local/share in each user's home directory. It seems it is possible to have a site-wide directory that everyone can use, but this would probably need to be set up by the JASMIN admins at Cartopy installation time. I believe this can be anywhere on the system, and could be helpful to avoid each user downloading the same data. I think the admins would need to pre-download data though... I'm no expert on this, perhaps Phil can provide some input?

[ScottHosking]

For a system like JASMIN, it doesn't really make sense that each user downloads and keeps their own copy of the shape files.  I'll wait to see if Phil has some advice before I create a new ticket.

On a related note, I've commented on a ticket to update Iris+Cartopy on JASMIN, http://proj.badc.rl.ac.uk/cedaservices/ticket/226

[ScottHosking]

Iris and Cartopy have now been updated on JASMIN to v1.6.1 and v0.10.0, respectively.  However, the problem with retrieving the coastline shapefiles still exists.

I'll create a ticket to try and get this fixed.

[ScottHosking]

In case anyone is interested, a ticket has been created to fix this issue on JASMIN.

see http://proj.badc.rl.ac.uk/cedaservices/ticket/322

[marqh]

Hello Scott

cartopy can be configured with a siteconfig.py in it's root directory

this enables data paths to be set up, such as:

import os

# The shared location for storing downloaded data.
def update_config(config):
    config["pre_existing_data_dir"] = "/data/local/shared_data"

    # for the writable data directory (i.e. the one where new data goes),
    # follow the XDG guidelines
    # (http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html)
    default_data = os.path.join(os.path.expanduser('~'), '.local', 'share')
    config["data_dir"] = os.path.join(os.environ.get("XDG_DATA_HOME",
                                                     default_data),
                                      'cartopy')

We put a useful set of example data in such a location on servers, to save repeated downloading.

Exactly what is put there is up to you and your users,

a range of coastline data and locations can be useful, such as

/data/local/shared_data]$ find . -type d
.
./shapefiles
./shapefiles/gshhs
./shapefiles/gshhs/l
./shapefiles/gshhs/c
./shapefiles/gshhs/i
./shapefiles/gshhs/h
./shapefiles/gshhs/f
./shapefiles/natural_earth
./shapefiles/natural_earth/physical
./shapefiles/natural_earth/cultural

this can be added to as need arises.

hope this helps

mark

[ScottHosking]

Thank you Mark, and sorry for the slow reply.

Is there any way I can use a path in my home directory to store coastline data until this issue is fixed by the admins?

[ScottHosking]

I've found a temporary solution:

Download http://www.nacis.org/naturalearth/110m/physical/ne_110m_coastline.zip and extract it.  Then copy the 3 files (list below) to ~/.local/share/cartopy/shapefiles/natural_earth/physical

• 110m_coastline.dbf
• 110m_coastline.shp
• 110m_coastline.shx

Note: you may need to rename these files first, removing "ne_" from the beginning of the files.

Then use plt.gca().coastlines(resolution='110m') from within python

[ScottHosking]

I've found another solution to retrieve the coastline data on JASMIN (see item 1 in the FAQ: http://proj.badc.rl.ac.uk/cedaservices/wiki/JASMIN/FAQ).

You need to setup a web proxy as follows:

$ export http_proxy=wwwcache.rl.ac.uk:8080
$ export https_proxy=wwwcache.rl.ac.uk:8080