Pyshp as... Fiona (with the __geo_interface__ )

[M. Laloux]

    One of the great advantages of Fiona (Sean Gillies) is its ability to quickly examine the contents of a shapefile as dictionaries or write a shapefile in the same way, thanks to generators/iterators and the  __geo_interface__ :

           import fiona
           f = fiona.open('point.shp')
           f.next()
           {'geometry': {'type': 'Point', 'coordinates': (161821.09375, 79076.0703125)}, 'id': '0', 'properties': {u'DIP_DIR': 120, u'STRATI_TYP': 1, u'DIP': 30}}
           f.next()['geometry']['coordinates']
           (161485.09375, 79272.34375)
           f.next()['properties']['DIP']
           55



    but now, with the __geo_interface__ for shapes of Christian Lederman, it is possible to do the same thing with Pyshp:

            def records(filename):
                reader = shapefile.Reader(filename)
                fields = reader.fields[1:]
               field_names = [field[0] for field in fields]
               for sr in reader.shapeRecords():
                    geom = sr.shape.__geo_interface__
                    atr = dict(zip(field_names, sr.record))
                    yield dict(geometry=geom,properties=atr)

            a = records('point.shp')
            a.next()
           {'geometry': {'type': 'Point', 'coordinates': (161821.09375, 79076.0703125)}, 'properties': {'DIP_DIR': 120, 'STRATI_TYP': 1, 'DIP': 30}}
           a.next()['geometry']['coordinates']
          (161485.09375, 79272.34375)
           a.next()['properties']['DIP']
           55

           b = record('line.shp')
           b.next()
           {'geometry': {'type': 'LineString', 'coordinates': ((146891.65625, 88174.265625), (146951.25, 88174.265625))}, 'properties': {'VAR_FORM': 'NEFA', ''CLASSE': 'VAR'', 'LENGTH': 59.59}}
        
          c= records('polygon.shp')
          c.next()
          {'geometry': {'type': 'Polygon', 'coordinates': (((166500.64206965509, 114695.34460173383), (166480.3066551598, 114693.60541532337), (166468.3914291781, 114692.18187126353), (166464.39182593071, 114692.26803581366),(166500.64206965509, 114695.34460173383)),)}, 'properties': {'CLASSE': 'VAR', 'FORM': 'X','VAR_FORM': 'NEFA', 'SYMBOL': 65}}

and you can work directly with shapely (like with pygeoif):

      from shapely.geometry import shape   
      a = records('point.shp')
      print shape( a.next()['geometry'])
      POINT (160689.5000000000000000 78371.9218750000000000)
      b = records('line.shp')
      print shape( b.next()['geometry'])
      LINESTRING (146891.6562500000000000 88174.2656250000000000, 146951.2500000000000000 88174.2656250000000000)
      c = records('polygon.shp')
      print shape( c.next()['geometry'])
      POLYGON ((166500.6420696550922003 114695.3446017338283127, 166480.3066551598021761 114693.6054153233708348, 166468.3914291780965868 114692.1818712635285920, 166464.3918259307101835 114692.2680358136567520, 166500.6420696550922003 114695.3446017338283127))

see https://github.com/mlaloux/PyShp-as-Fiona--with-geo_interface-/blob/master/README.md

I'm working now on writing shapefiles

[Christian Ledermann]

there is also

https://github.com/cleder/geo_file_conv/blob/master/geo_file_conv/shp_to_kml.py
https://github.com/cleder/geo_file_conv/

Ok this code is just a demo and WFM but it gives you an idea

the __geo_interface__ makes it easy to convert from [gis format here] to
[gis format here] in a few lines of pure python code, so it can run on pretty
much anything that can process 0 and 1

8-)

--
Best Regards,

Christian Ledermann

Nairobi - Kenya
Mobile : +254 702978914

<*)))>{

If you save the living environment, the biodiversity that we have left,
you will also automatically save the physical environment, too. But If
you only save the physical environment, you will ultimately lose both.

1) Don’t drive species to extinction

2) Don’t destroy a habitat that species rely on.

3) Don’t change the climate in ways that will result in the above.

}<(((*>

[Christian Ledermann]

And by the way: this is the first iteration of the __geo_interface__ for pyshp
so test the hell out of it and report bugs!
I am sure there are some, I just have not found them yet ;)

[M. Laloux]

no bug for the moment but it is possible to go further in the imitation of Fiona:

with Fiona:

    import fiona
    c = fiona.open('strati.shp')
    c.schema
    {'geometry': 'Point', 'properties': {u'DIRECTION': 'int', u'PENDAGE': 'int', u'TYPE': 'str:10'}}

With Pyshp

    import shapefile
    def schema(reader):
       properties = dict((d[0],d[1:]) for d in reader.fields[1:])
       return {'properties' : properties,
               'geometry' : reader.shapes()[1].__geo_interface__['type']}

    shapefile.Reader.schema = property(lambda self: schema(self))
    
    c = shapefile.Reader('strati.shp')
    c.schema 
    {'geometry': 'Point', 'properties': {'DIRECTION': ['N', 3, 0], 'PENDAGE': ['N', 2, 0], 'TYPE': ['C', 10, 0]}}

[Karim Bahgat]

Laloux, you mentioned you were working on adding support for writing to shapefiles. Not sure if it is what you meant but I wrote a fork for myself as well that allows to write individual geojson dict shapes with the shapefile writer so it can be saved. Thus it is possible to easily save the results from say a Shapely union. This might be interesting for you too Ledermann. You can find the fork at:

https://github.com/karimbahgat/pyshp-fork-speedup-and-geojson-write

What's the possibility to try to merge all these features into one extra feature full shapefile fork instead of many different ones?

I would be up for trying to collaborate over at github if you guys are interested. Ideal would would be to merge it into the main pyshp repo but can't find a way to contact Lawhead and maybe he just wants to keep the main version a bit more stable.

[Joel Lawhead]

Karin,

I love all the work you're doing. I'd like to migrate the main repository to Github from Google Code's subversion setup.  I have a Github account as user GeospatialPython.  I don't see any reason why we can't incorporate your fork once that migration is done. 

Some other todo items I have are:

- Make tests (in README.txt) self-contained so they are not dependent on external shapefiles.  This request was made by some Linux distros and other package distributors. 

- incorporate the public domain dbfpy code base for much better Dbf support. 

- Joel 

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

[Christian Ledermann]

On Tue, Jan 21, 2014 at 5:44 AM, Joel Lawhead <geospati...@gmail.com> wrote:

Karin,

I love all the work you're doing.

+1

 

I'd like to migrate the main repository to Github from Google Code's subversion setup.  

+1 a migration to github will be helpfull :)

 

I have a Github account as user GeospatialPython.  I don't see any reason why we can't incorporate your fork once that migration is done. 

Some other todo items I have are:

- Make tests (in README.txt) self-contained so they are not dependent on external shapefiles.  This request was made by some Linux distros and other package distributors. 

- incorporate the public domain dbfpy code base for much better Dbf support. 

which library are you talking about? link?

 

- Joel 

On Monday, January 20, 2014, Karim Bahgat <karim.bah...@gmail.com> wrote:

Laloux, you mentioned you were working on adding support for writing to shapefiles. Not sure if it is what you meant but I wrote a fork for myself as well that allows to write individual geojson dict shapes with the shapefile writer so it can be saved. Thus it is possible to easily save the results from say a Shapely union. This might be interesting for you too Ledermann. You can find the fork at:

https://github.com/karimbahgat/pyshp-fork-speedup-and-geojson-write

What's the possibility to try to merge all these features into one extra feature full shapefile fork instead of many different ones?

I would be up for trying to collaborate over at github if you guys are interested. Ideal would would be to merge it into the main pyshp repo but can't find a way to contact Lawhead and maybe he just wants to keep the main version a bit more stable.

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

[Joel Lawhead]

This is the Dbf library I mentioned:

http://dbfpy.sourceforge.net

It could be backwards compatible with the current API but I think it handles the header information much better.  And the license is compatible. 

- Joel

[Christian Ledermann]

I was thinking about https://pypi.python.org/pypi/dbf with which I made good experiences.

On Wed, Jan 22, 2014 at 2:37 PM, Joel Lawhead <geospati...@gmail.com> wrote:

This is the Dbf library I mentioned:

http://dbfpy.sourceforge.net

It could be backwards compatible with the current API but I think it handles the header information much better.  And the license is compatible. 

dbf is BSD licensed, would that be compatible?

[Joel Lawhead]

You're right. That looks more advanced. 

[Karim Bahgat]

Great, and glad you like it! I'll just wait untill you (Joel) have migrated PyShp to GitHub, then fork your most recent version properly, redo my changes, and then try to send you my changes as "pushes" (if that's what they're called). Then we can possibly all talk about what to keep/change/rename. 

About the dbf plans, what specifically did (Joel) you have in mind in regards to the benefits of switching to a different dbf basecode? Is it to get fewer bugs? I've personally never had any problems with PyShp's dbf reader. 

And a curiosity question for Ledermann: Ever since your __geo_interface__ code was incorporated into the main PyShp, are there any remaining differences between the main code and your GitHub fork that you wish to merge later on, or is it currently just a copy?

Personally I think it's a shame that everyone's sort of "forced" to go with the flow and move to GitHub, Google Code is just so much more neater and aesthetically pleasing to the eye (I would have used it instead if they hadn't discontinued the code hosting/download section). 

Excited for the PyShp module to grow further, it's the only one I use since OGR and ShapeUtil are such a pain to install. For the future I am thinking of maybe also filling in the "select" function in the PyShp Editor class (though I wonder if it would be good to also have in the Reader), should be relatively easy to implement a loop and some if statements.

PS: Joel, you might or might not be interested to know that I used and credited your PyShp module in this free downloadable geocoding software that I recently wrote, which you can check out at:
http://geocodeanything.wordpress.com/
 
Karim

- Joel

- Joel 

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best Regards,

Christian Ledermann

Nairobi -

--
Best Regards,

Christian Ledermann

Nairobi - Kenya
Mobile : +254 702978914

<*)))>{

If you save the living environment, the biodiversity that we have left,
you will also automatically save the physical environment, too. But If
you only save the physical environment, you will ultimately lose both.

1) Don’t drive species to extinction

2) Don’t destroy a habitat that species rely on.

3) Don’t change the climate in ways that will result in the above.

}<(((*>

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Joel Lawhead]

Karim,

That sounds like a plan.  I've started the migration to Github. I've loved GoogleCode but I guess it's just a fact of life that the leading version control system changes every 8-10 years.   

There have been many problems with the Dbf class. For basic operations it works fine. For issues such as Unicode there have been

problems and little things like date fields have popped up. Most of the issues are small but have show-stopping impacts for many users.  The class is very minimal and could be quickly improved by adopting a working code base. 

I think the geo interface changes are fully incorporated. 

I hope to continue to expand pure python Geospatial solutions.  Most people are happy using Python APIs to C libraries and often that approach is the best way to do things.  

But I find a distinct joy in implementing pure python geospatial algorithms. And there is a growing number of people who share that passion.  Well-written Python code makes concepts easier to understand. And as you mentioned a simple python library is just easier to install and use. 

I will definitely check out your geocoder and see if I can do a post about it.

- Joel

E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Christian Ledermann]

On Wed, Jan 29, 2014 at 5:04 AM, Karim Bahgat <karim.bah...@gmail.com> wrote:

Great, and glad you like it! I'll just wait untill you (Joel) have migrated PyShp to GitHub, then fork your most recent version properly, redo my changes, and then try to send you my changes as "pushes" (if that's what they're called). Then we can possibly all talk about what to keep/change/rename. 

About the dbf plans, what specifically did (Joel) you have in mind in regards to the benefits of switching to a different dbf basecode? Is it to get fewer bugs? I've personally never had any problems with PyShp's dbf reader. 

I did not run into problems yet, but IIRC pyshps dbf reader does not support encodings, so if you get a shape file in Cyrillic there may be problems. Apart from that I favor 'outsourcing' as many things as possible, so you have as many eyes as possible on it for bug reports and (although I doubt there will be any for such an ancient file type) feature enhancements

 

And a curiosity question for Ledermann: Ever since your __geo_interface__ code was incorporated into the main PyShp, are there any remaining differences between the main code and your GitHub fork that you wish to merge later on, or is it currently just a copy?

No, all my changes have been incorporated into trunk

 

Personally I think it's a shame that everyone's sort of "forced" to go with the flow and move to GitHub, Google Code is just so much more neater and aesthetically pleasing to the eye (I would have used it instead if they hadn't discontinued the code hosting/download section). 

I like github because it allows easy integration of TravisCI, coveralls.io, etc. If it wasn't for that a google code mercurial repository would also be fine, just something that allows you easily to fork and send pull requests.

 

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best Regards,

Christian Ledermann

[Joel Lawhead]

Karim,

PyClipper is VERY interesting.  It does seem to have a pure python implementation of the winding numbers algorithm for doing buffers (aka polygon offsets).  That algorithm is the holy grail.  All other shapely algorithms are fairly easy to implement.  The buffer algorithm is extremely difficult.  When I mentioned it in a comment before, I had found a pure python implementation in the Py2D project but it was for gaming and it was failing on all but the simplest shapefiles. 

I will set up a GitHub project to get things going but because this project is based on Clipper it should work for what we need.

FYI, I've moved the pyshp code to GitHub and have been working on getting the documentation ported over to markdown.  https://github.com/GeospatialPython/pyshp

Then my next step will be to incorporate the latest round of contributions including yours.

By the way, your posts here were flagged as potential spam so I made sure that won't happen again.

- Joel

On Mon, Mar 24, 2014 at 10:29 AM, Karim Bahgat <karim.bah...@gmail.com> wrote:

Joel, I forget where, but I think you mentioned somewhere that you wanted to create a pure Python geographic vector analysis library (sort of like a pure Python Shapely). I recently found a pure Python library called PyClipper and thought it might be an interesting place for you to start. It has many geometric methods like point in poly, intersect, union, etc. Only backdraw is it's a bit difficult to understand the documentation, and very few examples. 

Here is the link: https://github.com/clothbot/clipper

As you can see there are many hooks for different languages, but in the "Python" folder you will find the "clipper.py" file which is pure Python and works on its own. I've also attached a basic example script that you can run to see some of the operations and code for how to do them (run the script, push Enter/Return to perform an intersect, and left-click your mouse to do a point in poly test). 

Not sure about the licensing, but hopefully you can simply reuse some of the more advanced parts of the code while making the front-end functions more shapefile and user-friendly. Oh and btw, if and when you put up the geographic clipping project on GitHub as open to contributors I'll probably hop on that train because I love geographic algorithm stuff (doesnt mean I'm good at it, but I do love it :P ). 

Karim

---------

- Joel

- Joel 

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt

E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Karim Bahgat]

That's great, super happy PyClipper helped you out. Very excited to see it once this pure Python geometry lib gets going on Github.

And I noticed the PyShp migration which is great. Let me know whether you want me to refork the newest version of your repo and then make a pull request, or if you're going to handle it yourself to make sure it goes smooth. 

[Karim Bahgat]

Hey Joel,

I just remember we were talking about PyClipper and you mentioning you wanted to make a pure-Python geometry manipulation library ala Shapely. Well, in a fit of unexpected inspiration I wanted to briefly test out if I could create a simple interface to PyClipper, and it ended up with something useful, creating a pure-Python shapely-like package, which I'm simply calling "Shapy". For now it supports the basic shapely geometry attributes and methods, but is still lacking a few. A few bugs and optimizations are probably also due. It's up on Github if you want to check it out, and maybe if you like where it's going you could use it as a starting point for the plans you had. It will take me a while to add all features and make it robust, so contributions would be awesome. Here's a post briefly describing how far I've gotten: 

http://thepythongischallenge.wordpress.com/2014/06/04/shapy-a-new-pure-python-shapely-in-the-works/

and for the Github link:

https://github.com/karimbahgat/Shapy

Best,
Karim

- Joel

- Joel 

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt

E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Joel Lawhead]

Karim,

That is huge!  It's perfect!  You focused on the most important parts: shapely compliance and buffering polygons.  Buffering polygons was by far the hardest part.  Adding points and lines will not be difficult. 

I am very, very excited about this. 

- Joel

--

You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Joel Lawhead, PMP
E-mail: jlaw...@geospatialpython.com
Web: GeospatialPython.com
Twitter: @SpatialPython

--
You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Karim Bahgat]

Thanks, glad you liked it :) I also pushed point and multipoint distance functions, and circular buffer now. Anyway, the only thing I have noticed is that polygon union as well as buffer (and maybe sometimes intersect) will just fail on simple shapes for some unknown reason in the pyclipper algorithm, so that's a potential limitation here unless we can fix it somehow. I opened an issue with the pyclipper repo at sourceforge, so we'll see how that turns out. But I just don't really know where or why it happens, and the pyclipper process is a very complex web of functions and classes to understand. But if you are familiar with the buffer algorithm I'm almost positive the error occurs in the part of the code that says 'raise Exception("Intersect error")'.

So slightly bad news, but hopefully it can be fixed :)

Karim

To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpython+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Karim Bahgat]

Quick followup: I just got a response from Angus (http://sourceforge.net/p/polyclipping/bugs/101/), the author of Clipper and he said that the pure-Python version was an old one, one he write while first learning Python, and one he no longer supports. Which makes sense why there would be some bugs in there, and the fact that all recent bug fixes have not been incorporated. All in all, I think this means that PyClipper might be a bit "rusty", but I think with some effort it can probably be cleaned up. If I ever get the time or some type of funding I would definetively read up on the Vatti clipping algorithm and try fixing it, but unless some other willing soul comes around we'll just have to use the module for what it's worth in the meantime. 

Karim

[Joel Lawhead]

Thanks for the update.  

--

You received this message because you are subscribed to the Google Groups "Geospatial Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geospatialpyth...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.