babel: kml -> exhibit json

[David Huynh]

Hi all,

I started to add support for KML -> Exhibit JSON conversion in Babel.
This is very very basic for now. To try it, go to
http://service.simile-widgets.org/babel/
Select "KML" in the first column of radio buttons, "Exhibit JSON" in the
second, "data is on some web sites" in the third, and paste in this test URL

http://simile-widgets.googlecode.com/svn/babel/trunk/converters/kml-converter/tests/test1.kml
Here's what you're supposed to see

http://service.simile-widgets.org/babel/translator?reader=kml&writer=exhibit-json&mimetype=text/plain&url=http%3A%2F%2Fsimile-widgets.googlecode.com%2Fsvn%2Fbabel%2Ftrunk%2Fconverters%2Fkml-converter%2Ftests%2Ftest1.kml

To enable this in Exhibit (before it gets added to Exhibit officially),
add this code after including exhibit-api.js

<script>

Exhibit.BabelBasedImporter.mimetypeToReader["application/vnd.google-earth.kml+xml"]
= "kml";
Exhibit.importers["application/vnd.google-earth.kml+xml"] =
Exhibit.BabelBasedImporter;
</script>

Then, a KML data link would look like this

<link rel="exhibit/data" type="application/vnd.google-earth.kml+xml"

href="http://simile-widgets.googlecode.com/svn/babel/trunk/converters/kml-converter/tests/test1.kml"
/>

I haven't tested this Exhibit integration myself. I know of one small
catch, which is that the KML converter generates pairs of lng,lat,
whereas the Exhibit map view expects lat,lng. You can make the Exhibit's
original map view takes lng,lat with this hack

Exhibit.MapView._accessorSpecs[1].alternatives[0].bindings[0].bindingNames
= [ "lng", "lat" ];

If you're using the polygon map view used by
http://people.csail.mit.edu/dfhuynh/projects/election08/election08.html
then you'd just need this in the view div
ex:latlngOrder="lnglat"

Some more thoughts are needed if different data sources use different
lat/lng orders.

Please let me know how this works out for you.

David

[Jon Crump]

David, John, et al.

I think adding kml support to Babel is a great idea! Nice!

I think expanded GIS data views will be very helpful in exhibits, and I'm
working on polygons for my timeline/googleMaps mashup as well.

You may already be familiar with this project. These folks have a number
of useful things at: http://code.google.com/p/primary-maps-2008/

Without installing any sort of big GIS tool, I found this python script
particularly handy:
http://code.google.com/p/primary-maps-2008/source/browse/trunk/shpUtils.py
which suggested to me that it might be possible to induce Babel to handle
ESRI shapefiles as well.

Unfortunately, shapefiles for medieval manorial boundaries are not exactly
thick on the ground. ;-)

Jon

[John Callahan]

I was not aware of the Google Primary Maps 2008 code project. And they
do have several scripts to read. However, they were probably written
for very specific needs. I bet that project was disbanded immediately
after the U.S. election and had their programmers move to the next one.
Google received a lot of attention for their (very cool) real-time
tracking map-based mashups.

http://www.gdal.org/ogr/index.html

For geodata conversion, I would recommend GDAL/OGR. It's an open source
project that is essentially the backbone of data i/o for all FOSS geo
packages. It's small, fast, efficient, and has an active community
supporting it. I have access to some high-priced GIS software and I use
GDAL/OGR whenever I can! OGR contains the vector-based (points, lines,
polygons) routines and drivers. (GDAL for rasters.) OGR supports
read/write capabilities for many formats, including:

ESRI shapefile
GeoRSS (Simple and GML)
KML
GeoJSON
..and many others...


Personally, I think it'd be great if Exhibit could support the basic
geometries (points, lines, polygons) in KML and GeoRSS Simple.

- John

[John Callahan]

OK. I first downloaded David's application at
http://people.csail.mit.edu/dfhuynh/projects/election08/election08.html
to my own server and everything worked fine. I then created two polygon
KML files on my own (one through Google Earth and the other by
converting from an ESRI shapefile.) I ran both of these through Babel
and downloaded the output as json files. I used these two new json
files as data sources in the election08 application. Everything looked
great. Polygons showed up where they should. So, the translation seems
to work.

I needed to make only a few changes to election08.html to make the data
sources from Babel work.

1) As David mentioned, add ex:latlngOrder="lnglat" to the polygon map view
2) changed ex:polygon=".border" to ex:polygon=".polygon"
3) edited line 263 in polygon-map-view.js to use ";" as the separator
between coordinate pairs. (it was set to "|")


No such luck adding the data sources directly from Babel. Using the
same election08.html, I took out the links to the previous json data
files and added the following immediately after including exhibit-api.js.

<script>
Exhibit.BabelBasedImporter.mimetypeToReader["application/vnd.google-earth.kml+xml"]
= "kml";
Exhibit.importers["application/vnd.google-earth.kml+xml"] =
Exhibit.BabelBasedImporter;
</script>

<link rel="exhibit/data" type="application/vnd.google-earth.kml+xml"
href="http://simile-widgets.googlecode.com/svn/babel/trunk/converters/kml-converter/tests/test1.kml"
/>

This did not work for the test1.kml files or for the two kml files I
created. The error I got was a javascript dialgue box stating "Failed
to load javascript file:" wiht the following as the file:

http://simile.mit.edu/babel/translator?reader=kml&writer=exhibit-jsonp&url=http%3A%2F%2Fsimile-widgets.googlecode.com%2Fsvn%2Fbabel%2Ftrunk%2Fconverters%2Fkml-converter%2Ftests%2Ftest1.kml&callback=Exhibit.JSONPImporter._callbacks.cb1

When you enter this URL directly in a browser, you get

Babel error: No reader of name kml


- John

[David Huynh]

Jon,

Thanks for the link! I didn't know about it before. But looking at it, I
couldn't find any useful U.S. county boundary file...

David

[David Huynh]

My exhibit is still using the simile.mit.edu babel service. Could you
try to switch the exhibit-api.js URL to

http://api.simile-widgets.org/exhibit/2.2.0/exhibit-api.js

That should use the new babel service, which supports KML.

David

[John Callahan]

Yes, that was it. Linking to the updated exhibit-api.js file worked. I
started with two KML files, one created by hand in Google Earth and
another converted from an ESRI shapefile. I successfully added these to
an Exhibit. They displayed fine, the facets worked (tested using the
"vote for" field in the election08 app), and onClick worked as expected.

All this works great for polygons (I was using polygon-map-view.js in
the election08 app). Now, how do we get points, lines, and polygons on
the same map? I'm currently linking to the following. Should I link
to trunk for both?

http://api.simile-widgets.org/exhibit/2.2.0/extensions/map/map-extension.js
http://api.simile-widgets.org/exhibit/2.2.0/exhibit-api.js

- John




David Huynh wrote:

[Jon Crump]

On Wed, 18 Mar 2009, David Huynh wrote:

>
> Jon,
>
> Thanks for the link! I didn't know about it before. But looking at it, I
> couldn't find any useful U.S. county boundary file...

Yes, we have no bananas. I just found the python code there useful for a
quick and dirty way of extracting lat/lon vertices from ESRI shapefiles,
and thought others might find it useful as well. I take your point about
the size of a US counties dataset and I haven't found the kind of coarse
grained data you seek.

Thank you John for this http://www.gdal.org/ogr/index.html
I'll be looking into this. If I'm going to be doing much in the way of
creating multiple map overlays, I'll probably need something more than a
python script and google map's API (Though I've made good progress with
just these)

Thanks all,
Jon

[David Huynh]

John Callahan wrote:
> Yes, that was it. Linking to the updated exhibit-api.js file worked. I
> started with two KML files, one created by hand in Google Earth and
> another converted from an ESRI shapefile. I successfully added these to
> an Exhibit. They displayed fine, the facets worked (tested using the
> "vote for" field in the election08 app), and onClick worked as expected.
>
> All this works great for polygons (I was using polygon-map-view.js in
> the election08 app). Now, how do we get points, lines, and polygons on
> the same map? I'm currently linking to the following. Should I link
> to trunk for both?
>
> http://api.simile-widgets.org/exhibit/2.2.0/extensions/map/map-extension.js
> http://api.simile-widgets.org/exhibit/2.2.0/exhibit-api.js
>

Yes, you should link to trunk for both, with the understanding that it's
development code and can be broken abruptly with the next code checkin.

I'd need to consolidate all of the map functionalities back into the map
extension... I'll try to get to it one of these days.

David