Is it possible to output a GeoQuerySet to a shapefile?

[Ben Welsh]

I've found myself building temporary models specifically for storing a GeoQuerySet -- and then dumping them out to shp using the pgsql2shp command line utility. (http://www.postgis.fr/tarball/postgis-cvs/loader/README.pgsql2shp)

While it's possible I'm overlooking some obvious documentation, or, at the other extreme, making some ridiculous pony request -- I'm curious whether there might be a more direct method for dumping things out.

The reason this comes up is that I work with a number of coworkers who are longtime ArcView users and don't write much python. I want to involve them more in my work, and it's handy to be able to throw them a format they can quickly get a grip on. The other alternative I've stumbled into is writing out the GeoQuerySet to CSV with an X and Y field when I'm dealing with point data. 

I just know there's something smarter I could be doing. So I figured I'd bother the smart people. Thanks for bearing with me.

[Josh Livni]

I believe Dane has done some work on this:

http://bitbucket.org/springmeyer/django-shapes/wiki/Home

 -Josh

[Justin Bronn]

Ben Welsh wrote:
> I've found myself building temporary models specifically for storing a
> GeoQuerySet -- and then dumping them out to shp using the pgsql2shp
> command line utility.
> (http://www.postgis.fr/tarball/postgis-cvs/loader/README.pgsql2shp)
>

You should seriously take a look at the ogr2ogr command-line program
that comes with GDAL. It can go directly from PostGIS to shapefile (or
CSV, KML, GPX, etc.).

> While it's possible I'm overlooking some obvious documentation, or, at
> the other extreme, making some ridiculous pony request -- I'm curious
> whether there might be a more direct method for dumping things out.

Yes there is a more direct method, and Dane has made django-shapes for
this purpose (as Josh pointed out). However the end game would be to
extend the existing GDAL (OGR) API within GeoDjango to support
DataSource creation. For a while now I've been wrestling in my mind how
the API should look and think it should be done via a `create` class method:

ds = DataSource.create('shapefile', '/path/to/shp', ...)

The first argument is the driver type (or Driver instance) and the
second argument is the path (or equivalent) to the data. The hard
part is figuring out what goes in the `...`. Either a dict (which has
no ordering) or a tuple of field names and field types is necessary.

Thus, I'm open to suggestions on how this should look -- just keep in
mind I want to be able to support creation of OGR data sources
generally, and just not shapefiles.

-Justin

[Ben Welsh]

Thanks for the links, guys. Dane's extension looks like it'll be a help. And I'll take a look at ogr2ogr too.

[Dane Springmeyer]

Hey Ben,

Yes, I'd echo Justin that large parts of django-shapes will hopefully be obsolete once the OGR api is extended inside GeoDjango.

But, the original motivation of the application was for the purpose of spitting out a zipped shapefile (since shapefiles are made of up multiple parts) to the browser via an arbitrary GeoQuerySet, and I don't presume that the ogr driver has logic for that. The idea being that your coworkers could easily download the updated shapefile by hitting the same url.

So, if this is useful its the ShpResponder Class that you'll be after:

http://bitbucket.org/springmeyer/django-shapes/src/tip/shapes/views/export.py

and see the docs here: http://bitbucket.org/springmeyer/django-shapes/src/tip/docs/views.txt

but consider it pretty rough at this point as it does not properly handle types nor related tables.

Dane

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

These Ideas are very nice! I was really excited to hear about the
functionality already present in django shapes (though I have yet to
test it)

Regarding the OGR API Justin was talking about, it would be very much
welcome, because although OGR already has the functionality to create
Shapefiles, the OGR API is a little awkward. I have nevetheless used
it successfully to mix shapefiles with other databases, add
extralayers, etc.

One of the things I would love to see in Geodjango is the ability to
upload shapefiles and have permanent models created from them, so that
it is possible to put together a web-based map repository on which
maps can be browsed and easily remixed through-the-web, to generate
new shapes, kmls, etc...

Is this feature on the near term roadmap for Geodjango?

cheers,

Flávio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6IXYACgkQC7yysgGhWouRHgCfXJmX/CdxaDmFXEePrZ+dsE0s
8uQAniShFBUECKE0dpfFseqTTWUYQ0CM
=dHNU
-----END PGP SIGNATURE-----

[Guillaume Sueur]

Hi Flavio,

I've just done this kind of thing, thanks to the help of Justin Bronn. I
works fine for shapefile, MapInfo tab files being a little bit more
tricky (multiple geometry types in layers...)
But it does the job : upload a zipfile, unzip, get the shapefile, build
a model from its fields (layermapping), create the table and populates
it. The model is then accessible from the admin site. BUT I stil have
pretty buggy behaviours on production server where MaxRequestPerChild is
high for performance reasons. The model disappears and reappers randomly :-(

I can share that approach if you wish.

Regards

Guillaume

Flávio Codeço Coelho a écrit :

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Guillaume,

Yes, I am interested in seeing what you have done. Do you plan to make
the code available as open-source?
I would certainly be glad to test it and try to offer improvements as
I need something like this for my own purposes.

regards,

Flávio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6QmUACgkQC7yysgGhWoscUwCgmw6ImQ/ZjA/wENuviU2nkwwg
3h0AnRkQii41Lw0AgJGRyV5stLVDAGXQ
=HND8
-----END PGP SIGNATURE-----

[Guillaume Sueur]

Flávio Codeço Coelho a écrit :

> Hi Guillaume,
>
> Yes, I am interested in seeing what you have done. Do you plan to make
> the code available as open-source?

Sure, but as more than half of the code comes from Justin, we should ask
him first.

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok,

I imagine he will see this message and respond soon, I hope.

Flávio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6Zp0ACgkQC7yysgGhWotUnACbBOGZ5jl+pQmI0ESvh6L/tWzk
x0MAoKFaZBzkLMrfP3YwkDj5VOaAby0u
=+Ysd
-----END PGP SIGNATURE-----

[Ariel Mauricio Nunez Gomez]

I also have developed that functionality(On the fly model creation), I would be very interested in comparing them and trying to make sure it runs as buggyless as possible.

Best,

Ariel

[Guillaume Sueur]

sure !
do you use mod_python as well ?

Ariel Mauricio Nunez Gomez a écrit :

[Ariel Mauricio Nunez Gomez]

mod_wsgi

[Guillaume Sueur]

how do you handle the performances/reload issue ?
Just after having register the dynamic model it appears on the admin
page, but it can disappear after few click, like if the admin was
refreshed, without the dynamic inclusion and without re-read of
models.py. I don't really understand that behaviour. If
MaxRequestPerChild is set to 1, I have no problem as the models.py is
reloaded each time, but it's not a good idea for a production server.

Guillaume

Ariel Mauricio Nunez Gomez a écrit :

> mod_wsgi
>
> On Fri, Mar 13, 2009 at 10:46 AM, Guillaume Sueur
> <guillau...@neogeo-online.net

> <mailto:guillau...@neogeo-online.net>> wrote:
>
>
> sure !
> do you use mod_python as well ?
>
> Ariel Mauricio Nunez Gomez a écrit :
> > I also have developed that functionality(On the fly model creation), I
> > would be very interested in comparing them and trying to make sure it
> > runs as buggyless as possible.
> >
> > Best,
> >
> > Ariel
> >
> > On Fri, Mar 13, 2009 at 8:58 AM, Flávio Codeço Coelho
> > <fcco...@gmail.com <mailto:fcco...@gmail.com>

> <mailto:jbr...@gmail.com <mailto:jbr...@gmail.com>>

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Cool, and is your code available somewhere?

Flávio

iEYEARECAAYFAkm6krQACgkQC7yysgGhWov8oACfVHtwrpfniKS1qFMqn3DyHTSb
ImIAnREUNcx1TdYnXWYd1adl+snqDkfi
=9aRK
-----END PGP SIGNATURE-----

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Maybe recofiguring your Django app to run as fast-cgi may help with
the MaxRequestPerChild problem.

Flávio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6kz8ACgkQC7yysgGhWouCkACfXbhJt/GY16H3suCZiIOVGEN+
G8AAoI3XDvOMwhf9sroZZgdGXGwl+6J0
=VY8H
-----END PGP SIGNATURE-----

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think this may have to do with the fact that modpython keeps a long
running process which only gets restarted if the webserver gets
restarted. So your problem may have to do with refreshing of the models.

Flávio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6lDsACgkQC7yysgGhWos3OQCeItWBq6L1EBOQaNuf21cYs9VU
e2oAn0tN9T2LQmEnhahEbsIxx5o5J1VC
=Euw9
-----END PGP SIGNATURE-----

[Justin Bronn]

> how do you handle the performances/reload issue ?
> Just after having register the dynamic model it appears on the admin
> page, but it can disappear after few click, like if the admin was
> refreshed, without the dynamic inclusion and without re-read of
> models.py. I don't really understand that behaviour. If
> MaxRequestPerChild is set to 1, I have no problem as the models.py is
> reloaded each time, but it's not a good idea for a production server.
>

Basically, you need a way to persist the dynamically created models
somehow. There's two approaches I can think of: (i) actually write out
models to models.py or (ii) persist the models information in the
database via another model, and set module attributes dynamically.

> Sure, but as more than half of the code comes from Justin, we
> should ask him first.

Here's the app, in it's incomplete state:

http://geodjango.org/hg/dyngeo/

I'll try and work on it some more, and integrate changes of others, but
it'll have to take a backseat to the priority of getting 1.1 out.

-Justin

[Guillaume Sueur]

indeed. I would like to keep a high MaxRequestPerChild for performances,
but force the reload of my new models.py. I played around with utime()
to change the modification time of some important files but without
noticeable success.
I know I can just solve the problem by using fcgi but you know, I want
it all !

Guillaume SUEUR
Gérant
Neogeo Technologies SARL
46 RUE MATABIAU
31000 TOULOUSE
Tél : 05 31 98 17 07
Port. : 06 74 91 95 20
Web : http://www.neogeo-online.net

[Ariel Mauricio Nunez Gomez]

Basically, you need a way to persist the dynamically created models
somehow.  There's two approaches I can think of: (i) actually write out
models to models.py or (ii) persist the models information in the
database via another model, and set module attributes dynamically.

 I did (ii), I have a DataSource and Layer model in my app that have the OGR Datasource info and the layer specific information. In the Layer object I have a get() method that retrieves the dynamic model or registers it if it's not registered.

Ariel.

[Guillaume Sueur]

I did (i) !

Ariel Mauricio Nunez Gomez a écrit :
>

> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google
> Groups "geodjango" group.
> To post to this group, send email to geod...@googlegroups.com
> To unsubscribe from this group, send email to
> geodjango+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/geodjango?hl=en

[Ariel Mauricio Nunez Gomez]

I started with code generation too, but it was way too hackerous for my taste :)
Maybe it's just that I am polarized by: http://code.djangoproject.com/browser/django/trunk/AUTHORS#L461

Ariel

[Guillaume Sueur]

Cool. I noticed that the syncdb was shaking the admin part a little too
much. It was changing from french to english. So I only use a small
install script.

Here is the whole thing as attachement. Feel free to criticize as is it
done by someone good at geographic things, but pretty crap at django
insights...

Regards,

Guillaume

Justin Bronn a écrit :

--

[Flávio Codeço Coelho]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The code looks clean and understandable, I'll give it a try, thanks!

Flávio

Justin Bronn wrote:
>
>
> Here's the app, in it's incomplete state:
>
> http://geodjango.org/hg/dyngeo/
>
> I'll try and work on it some more, and integrate changes of others,
> but it'll have to take a backseat to the priority of getting 1.1
> out.
>
> -Justin
>
> >
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkm6oFsACgkQC7yysgGhWos9ZACbBACXQx/n9JfBctmyobUCIwhi
cb8AnRX4vINghxEZRkbfrvJp/kpaWCgG
=2Ny0
-----END PGP SIGNATURE-----