Geo-json zip code geometry (CA)

[Johan Sundström]

Inspired by (and borrowing the framework from) http://mbostock.github.com/d3/talk/20111018/#25 I cooked up a little mashup of USGS earthquake response data by zip code for a little quake in Piedmont this weekend:

http://bl.ocks.org/1343714

I initially grabbed the California zipcode geometries from http://www2.census.gov/geo/tiger/TIGER2010/ZCTA5/2010/tl_2010_06_zcta510.zip and churned the shapefile through http://mapshaper.org/ in an attempt to simplify the shapes for the approximate resolution I was going for so the geometry wouldn't weigh in too heavy and then converting to geo-json via ogr2ogr -f "GeoJSON" out.json tl_2010_06_zcta510-polygons-ms1.shp, but judging by the unfiltered output (this one takes a while to load; its "all-zips-in-California" geometry is 15 MB large):

http://bl.ocks.org/1343564

...I'm still missing some geometry, as it doesn't fill the area of the state. (Or am I wrong to assume zip codes tesselate to fill a state's entire surface area?)

It's as visible in the first faster-loading link, where the empty space two zip codes North of the epicenter sticks out like a sore thumb.

To produce the top json (which is a more metadata-tastic version of the lower one, filtered by visible zip codes) I retrofitted zip-codes into the data manually (well, shell- and emacs-aidedly) by first filtering out the geometry from the rest:

ogrinfo -al tl_2010_06_zcta510.dbf | egrep -v '^  (|MULTI)POLYGON'

...then scraping together a zip code to name map and combining it with the ZCTA5CE10 fields (zip codes) from above, and stuffing both properties back into the json, whose shapes are listed in the same order as above output.

Are there better ready-made sources for geo json zipcode geometries available already? What kind of tools do people use to get both geometry and shape metadata into their json?

-- 

 / Johan Sundström, http://ecmanaut.blogspot.com/

[Tom Carden]

On Wednesday, November 9, 2011 11:45:16 AM UTC-8, Johan Sundström wrote:

Are there better ready-made sources for geo json zipcode geometries available already? What kind of tools do people use to get both geometry and shape metadata into their json?

I'm about to embark on a similar exercise. Simplifying one-by-one in MapShaper seems to be the way everyone does it. If I do this I will find a way to share the results, however informally... unless someone else has already done it?

Tom.

[Scott Murray]

Hey, Tom,

I researched this a few months ago, and it seems MapShaper has the monopoly on these algorithms.  :-)

I briefly explored building a new tool that would take shapefiles as input, then output simplified/cleaned GeoJSON while maintaining metadata (which MapShaper jettisons).

Would love to collaborate on such a tool, as there is dire need for this in the community.

Scott

[marc fawzi]

Hi Scott,

There is a dire need and growing.

I've been reusing GeoJSON from the Unemployment map example (D3 Examples) but noticed that the state border doesn't always agree with the county borders, so a tool that would allow us to graphically correct the path would be the ultimate imo.

Also, I have no idea what to do if I'm asked to do a map for Iceland or the UK. My sole source of GeoJSON has been the files bundled with the aforementioned D3 example. 

It would be nice for a tool to emerge that would make it easy to find the shape files, clean them up and allow manual correction of errors between county and state borders. 

Thanks! :)

Marc

[Mike Bostock]

> Would love to collaborate on such a tool, as there is dire need for this in
> the community.

I've been meaning to add topology preservation to the D3 simplify plugin:

https://github.com/d3/d3-plugins/tree/master/simplify
http://bost.ocks.org/mike/simplify/

I have ideas on how, but I haven't had time to do it yet,
unfortunately. The first thing to add would be consolidating
coincident points on shared boundaries, so that removing a point
removes it from both adjacent features.

Mike

[marc fawzi]

Wow re: Simplify

Related:

This was mentioned on the list recently but it's the best how-to-document I've come across so far, so here it is again:

http://vallandingham.me/shapefile_to_geojson.html

[Hadley Wickham]

>> Are there better ready-made sources for geo json zipcode geometries
>> available already? What kind of tools do people use to get both geometry and
>> shape metadata into their json?
>
> I'm about to embark on a similar exercise. Simplifying one-by-one in
> MapShaper seems to be the way everyone does it. If I do this I will find a
> way to share the results, however informally... unless someone else has
> already done it?

Beg Mark Harrower for the map shaper source code?

I've read the related paper a couple of times, but have never quite
managed to implement it as well as map shaper.

Hadley

--
RStudio / Rice University
http://had.co.nz/

[Hadley Wickham]

> I've been meaning to add topology preservation to the D3 simplify plugin:
>
> https://github.com/d3/d3-plugins/tree/master/simplify
> http://bost.ocks.org/mike/simplify/
>
> I have ideas on how, but I haven't had time to do it yet,
> unfortunately. The first thing to add would be consolidating
> coincident points on shared boundaries, so that removing a point
> removes it from both adjacent features.

There are a few hints in:

http://www.cartogis.org/docs/proceedings/2006/bloch_harrower.pdf

and

M. Harrower and M. Bloch. Mapshaper.org: A map generalization web
service. Exploring Geovisualization, 2006.

If you figure out a good algorithm, I'd almost certainly port it to R.

[Walter Svekla]

Have you guys seen File Bakery? There may be the potential to extend this:

http://filebakery.com/

Drag and Drop a zipped ESRI Shapefile .zip OR a MS Access .accdb/.mdb file anywhere on the screen and get a fresh .csv.

[Tom Carden]

On Thursday, October 18, 2012 1:17:49 PM UTC-7, Hadley Wickham wrote:

There are a few hints in:

http://www.cartogis.org/docs/proceedings/2006/bloch_harrower.pdf

and

M. Harrower and M. Bloch. Mapshaper.org: A map generalization web
service. Exploring Geovisualization, 2006.

If you figure out a good algorithm, I'd almost certainly port it to R.

The second paper is available here http://maps.grammata.com/cga2006_paper.pdf

The key seems to be transforming the polygon rings structure from geojson/shapefile into a topological structure (both Mike and I have talked to Matthew Bloch about this before), and to simplify the whole dataset in one go (in my case it won't be sufficient to simplify one US state at a time, because adjacent states share line segments at the border).

I may take a crack at this myself, or possibly adapt Mike Migurski's python library (called Bloch, naturally): https://github.com/migurski/Bloch - I'd dearly like to have a scriptable workflow for this!

Tom.

[Tom Carden]

Here are notes on approaching the problem using PostGIS:

http://trac.osgeo.org/postgis/wiki/UsersWikiSimplifyPreserveTopology

http://trac.osgeo.org/postgis/wiki/UsersWikiSimplifyWithTopologyExt

http://gis.stackexchange.com/questions/178/simplifying-adjacent-polygons

I've looked into the GEOS and Boost::Geometry libraries to accomplish this before, nothing seems to support it out of the box.

Tom.

[Jason Davies]

Here is my first attempt:

http://www.jasondavies.com/maps/simplify/

Relevant commit to d3-plugins:

https://github.com/d3/d3-plugins/commit/806c760d9164ff22a05ace29d7c62881e61031a1

I'm sure some things can be improved e.g. it inserts effective areas of
infinity at the points where shared borders "change hands", but I'm
pretty happy with the results.

Internally, it's a two-pass algorithm. The first pass builds a hash
table mapping each point to a different numerical ID (for speed) for
every new combination of lines that share that point.

The second pass simply splits lines into chains every time this ID
changes, and performs Visvalingam on each chain separately. Then, an
effective area of infinity is set for the first and last points so they
don't get dropped. Note that we need to do this for both the first and
last points, so that there are two "infinite" area points at the
changeover positions, since chains may be traversed in different orders.

The rest of the code is really just scaffolding required to navigate the
GeoJSON format.

The other potential improvement is to use a more dedicated data
structure for the hash table instead of JavaScript's built-in object.
This could avoid excessive creation of string keys. On the other hand,
this simplify.project(…) step only needs to occur once, so it may not be
worth bundling additional code just for this step.

--
Jason Davies, http://www.jasondavies.com/

[Johan Sundström]

On Wed, Oct 24, 2012 at 9:06 AM, Jason Davies <ja...@jasondavies.com> wrote:
> On Thu, Oct 18, 2012 at 12:00:24PM -0700, Mike Bostock wrote:
>> > Would love to collaborate on such a tool, as there is dire need for this in
>> > the community.
>>
>> I've been meaning to add topology preservation to the D3 simplify plugin:
>>
>> https://github.com/d3/d3-plugins/tree/master/simplify
>> http://bost.ocks.org/mike/simplify/
>>
>> I have ideas on how, but I haven't had time to do it yet,
>> unfortunately. The first thing to add would be consolidating
>> coincident points on shared boundaries, so that removing a point
>> removes it from both adjacent features.
>
> Here is my first attempt:
>
> http://www.jasondavies.com/maps/simplify/
>
> Relevant commit to d3-plugins:
>
> https://github.com/d3/d3-plugins/commit/806c760d9164ff22a05ace29d7c62881e61031a1

Ooo; great work!