GNIP 17 - Data Conversion on Upload/Import
David Winslow
Jeffrey Johnson
> See the proposal on the wiki at
> https://github.com/GeoNode/geonode/wiki/GNIP-17----Data-Conversion-on-Upload-Import
>
> What alternative approaches do you see? Maybe a pro/con analysis of the
> ogr2ogr approach would be useful.
It makes sense to me that GeoTools/GeoServer could eventually have an
OGR data store similar to the existing GDAL one. I'm mostly in favor
of using OGR for now, but the big downside is that it requires that
native library to be compiled, and this can be problematic across
platforms. Also, the initial implementation that Dialog did just
shells out to ogr2ogr and I dont think this is the best approach
either. It may make sense to use the ogr python bindings here. I
*dont* think it makes sense to try and write new GeoTools stores for
all of these various types of data since there are many.
One specific note about KML: Recent OGR builds can use googles own
libkml to do the kml parsing and this is hugely advantageous since kml
is a spec that has been (and will continue to be) implemented in ways
that it was never intended to do various wacky things. Using googles
own library (which is used by google earth) takes some of the pain out
of parsing these bizzare kml documents.
Im very open to other approaches here, but dont really have any good
ideas on how to do this. Anyone who does, please speak up.
> I think the uploader work that OpenGeo has been doing for the MapStory
> project may be relevant here as well.
I believe this is already noted in the GNIP, and I'd be very grateful
if someone on that team could add some information here. I think it
makes sense to harmonize the approaches here and have a single
consistent uploader that does these various things.
Ian Schneider
MapStory
> > project may be relevant here as well.
>
> I believe this is already noted in the GNIP, and I'd be very grateful
> if someone on that team could add some information here. I think it
> makes sense to harmonize the approaches here and have a single
> consistent uploader that does these various things.
shapefile, and a few other formats that have not been thoroughly
tested. There are a few transformations that can be applied and
asynchronous processing is supported. To drive this process from
Geonode, a python REST client is used.
Specifically in mapstory, our primary workflow is (and therefore the
driver of features):
1) accept shapefile (zipped or otherwise)
2) configure transformation of number or text into timestamp
3) ingest shapefile into postgres
4) polling of asynchronous processing to allow progress bar
5) create index on time column in database
6) configure geoserver appropriately if time fields are present
Looking forward, the importer has been designed to accept various
formats in a pluggable fashion. Likewise with transformations.
The short term road map for additions is (driven by mapstory
requirements):
1) add ingestion of CSV/Excel (initially with a fixed schema)
2) creation of view tables to allow joining non-spatial datasets with
well-known spatial datasets using common fields (FIPS, state name, etc)
Jeffrey Johnson
> > > I think the uploader work that OpenGeo has been doing for the
> MapStory
>> > project may be relevant here as well.
>>
>> I believe this is already noted in the GNIP, and I'd be very grateful
>> if someone on that team could add some information here. I think it
>> makes sense to harmonize the approaches here and have a single
>> consistent uploader that does these various things.
>
> High level summary: the uploader currently handles ingesting tiff,
> shapefile, and a few other formats that have not been thoroughly
> tested. There are a few transformations that can be applied and
> asynchronous processing is supported. To drive this process from
> Geonode, a python REST client is used.
>
> Specifically in mapstory, our primary workflow is (and therefore the
> driver of features):
> 1) accept shapefile (zipped or otherwise)
> 2) configure transformation of number or text into timestamp
It makes sense that at this point in the workflow, we can also do
format conversion. I.e. from MapInfo to shapefile, or XYZ format
straight into Postgres or similar.