Neo4j Spatial and OSM Nodes
187 views
Skip to first unread message
Geolama
Mar 5, 2012, 4:23:42 PM3/5/12
to Neo4j
I am a new to Neo4j/Neo4j Spatial but extremely familiar with
geospatial feature data in other than OSM models. From what I have
been able to glean from the documentation, the OSMLayer does not ever
considered that an OSM Node can be a feature that is not necessarily a
part of a Way or that can only be found as part of a relation. It
seems that OSM Nodes are not directly visible but are always
subservient to OSM Ways. Where is my thinking make a wrong turn here?
geospatial feature data in other than OSM models. From what I have
been able to glean from the documentation, the OSMLayer does not ever
considered that an OSM Node can be a feature that is not necessarily a
part of a Way or that can only be found as part of a relation. It
seems that OSM Nodes are not directly visible but are always
subservient to OSM Ways. Where is my thinking make a wrong turn here?
Peter Neubauer
Mar 5, 2012, 4:36:00 PM3/5/12
to ne...@googlegroups.com
Well,
that depends on how you import the data. You can make OSMNodes
explicit entities by importing them as such, see
OSMImporter.importFile(db,path, true) which will create an explicit
Node layer (IIRC).
that depends on how you import the data. You can make OSMNodes
explicit entities by importing them as such, see
OSMImporter.importFile(db,path, true) which will create an explicit
Node layer (IIRC).
Test and let me know - contribution in this area is highly appreciated.
Cheers,
/peter neubauer
G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer
Neo4j 1.6 released - dzone.com/6S4K
The Neo4j Heroku Challenge - http://neo4j-challenge.herokuapp.com/
Geolama
Mar 5, 2012, 5:27:37 PM3/5/12
to Neo4j
I tried your suggestion but I still do not see the OSM Node layer when
using Neoclipse 1.5. Perhaps this older version of Neoclipse is
insufficient for viewing the Neo4j Spatial OSM Graph.
On Mar 5, 1:36 pm, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:
using Neoclipse 1.5. Perhaps this older version of Neoclipse is
insufficient for viewing the Neo4j Spatial OSM Graph.
On Mar 5, 1:36 pm, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:
> Well,
> that depends on how you import the data. You can make OSMNodes
> explicit entities by importing them as such, see
> OSMImporter.importFile(db,path, true) which will create an explicit
> Node layer (IIRC).
>
> Test and let me know - contribution in this area is highly appreciated.
>
> Cheers,
>
> /peter neubauer
>
> G: neubauer.peter
> S: peter.neubauer
> P: +46 704 106975
> L: http://www.linkedin.com/in/neubauer
> T: @peterneubauer
>
> Neo4j 1.6 released - dzone.com/6S4K
> The Neo4j Heroku Challenge -http://neo4j-challenge.herokuapp.com/
> that depends on how you import the data. You can make OSMNodes
> explicit entities by importing them as such, see
> OSMImporter.importFile(db,path, true) which will create an explicit
> Node layer (IIRC).
>
> Test and let me know - contribution in this area is highly appreciated.
>
> Cheers,
>
> /peter neubauer
>
> G: neubauer.peter
> S: peter.neubauer
> P: +46 704 106975
> L: http://www.linkedin.com/in/neubauer
> T: @peterneubauer
>
> Neo4j 1.6 released - dzone.com/6S4K
Geolama
Mar 5, 2012, 5:46:48 PM3/5/12
to Neo4j
I think there's a problem in the OSMImporter.reIndex method. Look for
the two '>>>>>' for the problem:
public void reIndex(GraphDatabaseService database, int
commitInterval, boolean includePoints, boolean includeRelations) {
if (commitInterval < 1)
throw new IllegalArgumentException("commitInterval must be
>= 1");
System.out.println("Re-indexing with GraphDatabaseService: " +
database + " (class: " + database.getClass() + ")");
setLogContext("Index");
SpatialDatabaseService spatialDatabase = new
SpatialDatabaseService(database);
OSMLayer layer =
(OSMLayer)spatialDatabase.getOrCreateLayer(layerName,
OSMGeometryEncoder.class, OSMLayer.class);
// TODO: The next line creates the relationship between the dataset
and
// layer, but this seems more like a side-effect and should be done
// explicitly
OSMDataset dataset = layer.getDataset(osm_dataset);
layer.clear(); // clear the index without destroying
underlying data
long startTime = System.currentTimeMillis();
Traverser traverser =
database.getNodeById(osm_dataset).traverse(Order.DEPTH_FIRST,
StopEvaluator.END_OF_GRAPH,
ReturnableEvaluator.ALL_BUT_START_NODE,
OSMRelation.WAYS, Direction.OUTGOING, OSMRelation.NEXT,
Direction.OUTGOING);
Transaction tx = database.beginTx();
>>>>> boolean useWays = false;
int count = 0;
try {
layer.setExtraPropertyNames(stats.getTagStats("all").getTags());
>>>>> if (useWays) {
beginProgressMonitor(dataset.getWayCount());
for (Node way : traverser) {
updateProgressMonitor(count);
incrLogContext();
stats.addGeomStats(layer.addWay(way, true));
if (includePoints) {
Node first = way.getSingleRelationship(OSMRelation.FIRST_NODE,
Direction.OUTGOING).getEndNode();
for (Node proxy : first.traverse(Order.DEPTH_FIRST,
StopEvaluator.END_OF_GRAPH, ReturnableEvaluator.ALL,
OSMRelation.NEXT, Direction.OUTGOING)) {
Node node = proxy.getSingleRelationship(OSMRelation.NODE,
Direction.OUTGOING).getEndNode();
stats.addGeomStats(layer.addWay(node, true));
}
}
if (++count % commitInterval == 0) {
tx.success();
tx.finish();
tx = database.beginTx();
}
} // TODO ask charset to user?
} else {
beginProgressMonitor(dataset.getChangesetCount());
for (Node changeset : dataset.getAllChangesetNodes()) {
updateProgressMonitor(count);
incrLogContext();
for (Relationship rel :
changeset.getRelationships(OSMRelation.CHANGESET, Direction.INCOMING))
{
stats.addGeomStats(layer.addWay(rel.getStartNode(), true));
}
if (++count % commitInterval == 0) {
tx.success();
tx.finish();
tx = database.beginTx();
}
} // TODO ask charset to user?
}
tx.success();
} finally {
endProgressMonitor();
tx.finish();
}
long stopTime = System.currentTimeMillis();
log("info | Re-indexing elapsed time in seconds: " + (1.0 *
(stopTime - startTime) / 1000.0));
stats.dumpGeomStats();
the two '>>>>>' for the problem:
public void reIndex(GraphDatabaseService database, int
commitInterval, boolean includePoints, boolean includeRelations) {
if (commitInterval < 1)
throw new IllegalArgumentException("commitInterval must be
>= 1");
System.out.println("Re-indexing with GraphDatabaseService: " +
database + " (class: " + database.getClass() + ")");
setLogContext("Index");
SpatialDatabaseService spatialDatabase = new
SpatialDatabaseService(database);
OSMLayer layer =
(OSMLayer)spatialDatabase.getOrCreateLayer(layerName,
OSMGeometryEncoder.class, OSMLayer.class);
// TODO: The next line creates the relationship between the dataset
and
// layer, but this seems more like a side-effect and should be done
// explicitly
OSMDataset dataset = layer.getDataset(osm_dataset);
layer.clear(); // clear the index without destroying
underlying data
long startTime = System.currentTimeMillis();
Traverser traverser =
database.getNodeById(osm_dataset).traverse(Order.DEPTH_FIRST,
StopEvaluator.END_OF_GRAPH,
ReturnableEvaluator.ALL_BUT_START_NODE,
OSMRelation.WAYS, Direction.OUTGOING, OSMRelation.NEXT,
Direction.OUTGOING);
Transaction tx = database.beginTx();
>>>>> boolean useWays = false;
int count = 0;
try {
layer.setExtraPropertyNames(stats.getTagStats("all").getTags());
>>>>> if (useWays) {
beginProgressMonitor(dataset.getWayCount());
for (Node way : traverser) {
updateProgressMonitor(count);
incrLogContext();
stats.addGeomStats(layer.addWay(way, true));
if (includePoints) {
Node first = way.getSingleRelationship(OSMRelation.FIRST_NODE,
Direction.OUTGOING).getEndNode();
for (Node proxy : first.traverse(Order.DEPTH_FIRST,
StopEvaluator.END_OF_GRAPH, ReturnableEvaluator.ALL,
OSMRelation.NEXT, Direction.OUTGOING)) {
Node node = proxy.getSingleRelationship(OSMRelation.NODE,
Direction.OUTGOING).getEndNode();
stats.addGeomStats(layer.addWay(node, true));
}
}
if (++count % commitInterval == 0) {
tx.success();
tx.finish();
tx = database.beginTx();
}
} // TODO ask charset to user?
} else {
beginProgressMonitor(dataset.getChangesetCount());
for (Node changeset : dataset.getAllChangesetNodes()) {
updateProgressMonitor(count);
incrLogContext();
for (Relationship rel :
changeset.getRelationships(OSMRelation.CHANGESET, Direction.INCOMING))
{
stats.addGeomStats(layer.addWay(rel.getStartNode(), true));
}
if (++count % commitInterval == 0) {
tx.success();
tx.finish();
tx = database.beginTx();
}
} // TODO ask charset to user?
}
tx.success();
} finally {
endProgressMonitor();
tx.finish();
}
long stopTime = System.currentTimeMillis();
log("info | Re-indexing elapsed time in seconds: " + (1.0 *
(stopTime - startTime) / 1000.0));
stats.dumpGeomStats();
Geolama
Mar 5, 2012, 11:28:19 PM3/5/12
to Neo4j
I've come to the conclusion that for OSM, Neo4j Spatial needs to
borrow a concept from the Vector Product Format (VPF) (VMAP0, for
example): Entity Node. As VPF had topology, it needed to distinguish
between topological nodes (just a vertex with no tags) and a Feature
represented as a Node like for instance a Light Stanchion. These
Entity Nodes need to have a direct Relation to the osm_dataset much
like WAYS.
borrow a concept from the Vector Product Format (VPF) (VMAP0, for
example): Entity Node. As VPF had topology, it needed to distinguish
between topological nodes (just a vertex with no tags) and a Feature
represented as a Node like for instance a Light Stanchion. These
Entity Nodes need to have a direct Relation to the osm_dataset much
like WAYS.
Peter Neubauer
Mar 6, 2012, 3:27:41 AM3/6/12
to ne...@googlegroups.com
Geolama,
just checked, it works. Look at the db under
http://dl.dropbox.com/u/5925006/one-street-with-points.zip to find a
store that imported not only ways but even points indexed in the
RTree, see https://github.com/neo4j/spatial/blob/master/one-street.osm
for the base data.
just checked, it works. Look at the db under
http://dl.dropbox.com/u/5925006/one-street-with-points.zip to find a
store that imported not only ways but even points indexed in the
RTree, see https://github.com/neo4j/spatial/blob/master/one-street.osm
for the base data.
This is with 1.6.1, please use
https://github.com/neo4j/neoclipse/downloads to open it and let me
know if it works out for you.
Cheers,
/peter neubauer
G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer
Neo4j 1.6 released - dzone.com/6S4K
The Neo4j Heroku Challenge - http://neo4j-challenge.herokuapp.com/
Geolama
Mar 6, 2012, 11:41:59 AM3/6/12
to Neo4j
Peter,
It's not that I don't see OSM Nodes as part of OSM Ways, it's that
there is no easy way to find non-Way composed OSM Nodes. I have an
edited OSM data file that contains only OSM Nodes that can stand alone
outside of an OSM Way. I will send it to via private email for you to
run create a graph db and view in Neoclipse.
Geolama
On Mar 6, 12:27 am, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:
> Geolama,
> just checked, it works. Look at the db underhttp://dl.dropbox.com/u/5925006/one-street-with-points.zipto find a
> for the base data.
>
> This is with 1.6.1, please usehttps://github.com/neo4j/neoclipse/downloadsto open it and let me
It's not that I don't see OSM Nodes as part of OSM Ways, it's that
there is no easy way to find non-Way composed OSM Nodes. I have an
edited OSM data file that contains only OSM Nodes that can stand alone
outside of an OSM Way. I will send it to via private email for you to
run create a graph db and view in Neoclipse.
Geolama
On Mar 6, 12:27 am, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:
> Geolama,
> just checked, it works. Look at the db underhttp://dl.dropbox.com/u/5925006/one-street-with-points.zipto find a
> store that imported not only ways but even points indexed in the
> RTree, seehttps://github.com/neo4j/spatial/blob/master/one-street.osm
> for the base data.
>
> This is with 1.6.1, please usehttps://github.com/neo4j/neoclipse/downloadsto open it and let me
> know if it works out for you.
>
> Cheers,
>
> /peter neubauer
>
> G: neubauer.peter
> S: peter.neubauer
> P: +46 704 106975
> L: http://www.linkedin.com/in/neubauer
> T: @peterneubauer
>
> Neo4j 1.6 released - dzone.com/6S4K
>
> Cheers,
>
> /peter neubauer
>
> G: neubauer.peter
> S: peter.neubauer
> P: +46 704 106975
> L: http://www.linkedin.com/in/neubauer
> T: @peterneubauer
>
> Neo4j 1.6 released - dzone.com/6S4K
Craig Taverner
Mar 6, 2012, 5:00:34 PM3/6/12
to ne...@googlegroups.com
I've not followed everything here, but I can comment on how it should be working.
The OSM model is a connected graph that is not a perfect match for the class layer/feature/geometry arrangement of GIS systems. As such we need some explicit information for mapping the graph to geometries or features. What the OSMImporter does is have a bunch of hard-coded rules for deciding if something in the graph can be seen as a geometry. By default all WAYs are mapped to LineString geometries, and in fact the node representing the WAY will always have a relationship to a geometry node of type LineString, and the geometry nodes are added to the RTree index.
But RELATIONs and NODEs (using capitalization for OSM concepts to reduce ambiguity) may or may not represent geometries. We try to detect cases that do, and create those geometries. So, for example, a RELATION composed of WAYs that form a closed ring will be connected to a geometry node of type Polygon, and added to the RTree. Any NODE that contains properties will also be recognised as a Point geometry and a relation made to a geometry node of type Point (and added to the RTree). Free floating NODEs (not part of WAYs) are interpreted as Point geometries.
To summarize the rules for whether an OSM NODE is added to the RTree as a Point geometry:
- All NODEs if the 'includePoints' flag is set
- Nodes with properties other than location (points of interest)
- NODEs not part of WAYs
Geolama
Mar 7, 2012, 12:53:58 AM3/7/12
to Neo4j
My problem is that the OSM model is close to what I need but not close
enough. I should not ask OSM to bend to my will; instead I will look
into creating a spatial model that is based on OSM concepts but that
uses a much more robust application schemas from NGA. NGA's NSG
Application Schema (NAS) defines the Entities, Relations, Views (of
Entities) and View Groups (of Views) for a broad spectrum of
geospatial features (both concrete and abstract). I will be converting
VPF features into the appropriate NAS entities and load them into a
Neo4j Spatial data store.
enough. I should not ask OSM to bend to my will; instead I will look
into creating a spatial model that is based on OSM concepts but that
uses a much more robust application schemas from NGA. NGA's NSG
Application Schema (NAS) defines the Entities, Relations, Views (of
Entities) and View Groups (of Views) for a broad spectrum of
geospatial features (both concrete and abstract). I will be converting
VPF features into the appropriate NAS entities and load them into a
Neo4j Spatial data store.
Peter Neubauer
Mar 7, 2012, 2:59:36 AM3/7/12
to ne...@googlegroups.com
Yes,
that would be great! The OSM domain model is just one of the possible
domains you can map, this is one of the design goals of Neo4j Spatial.
So, if you build your domain structure, and the GeometryEncoder that
you then specify in your layer definition. That way, you can bridge
anything into GIS.
that would be great! The OSM domain model is just one of the possible
domains you can map, this is one of the design goals of Neo4j Spatial.
So, if you build your domain structure, and the GeometryEncoder that
you then specify in your layer definition. That way, you can bridge
anything into GIS.
https://github.com/neo4j/spatial/blob/master/src/test/java/org/neo4j/gis/spatial/LayersTest.java#L58
for an Example with a custom Encoder (SimplePointEncoder).
Let us know if you need help, and feedback is greatly appreciated.
Cheers,
/peter neubauer
G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer
Neo4j 1.6 released - dzone.com/6S4K
The Neo4j Heroku Challenge - http://neo4j-challenge.herokuapp.com/
Craig Taverner
Mar 7, 2012, 6:10:58 PM3/7/12
to ne...@googlegroups.com
I've not followed everything here, but I can comment on how it should be working.
The OSM model is a connected graph that is not a perfect match for the class layer/feature/geometry arrangement of GIS systems. As such we need some explicit information for mapping the graph to geometries or features. What the OSMImporter does is have a bunch of hard-coded rules for deciding if something in the graph can be seen as a geometry. By default all WAYs are mapped to LineString geometries, and in fact the node representing the WAY will always have a relationship to a geometry node of type LineString, and the geometry nodes are added to the RTree index.
But RELATIONs and NODEs (using capitalization for OSM concepts to reduce ambiguity) may or may not represent geometries. We try to detect cases that do, and create those geometries. So, for example, a RELATION composed of WAYs that form a closed ring will be connected to a geometry node of type Polygon, and added to the RTree. Any NODE that contains properties will also be recognised as a Point geometry and a relation made to a geometry node of type Point (and added to the RTree). Free floating NODEs (not part of WAYs) are interpreted as Point geometries.
To summarize the rules for whether an OSM NODE is added to the RTree as a Point geometry:
- All NODEs if the 'includePoints' flag is set
- Nodes with properties other than location (points of interest)
- NODEs not part of WAYs
On Tue, Mar 6, 2012 at 5:41 PM, Geolama <the.g...@gmail.com> wrote:
Reply all
Reply to author
Forward
0 new messages