Getting OOM error while running SDN

[Pattabi]

Getting this error while loading bean which has a 'wkt/spatial' index
using  - Spring data-neo4j  3.0.0.BUILD-20140207...., neo4j 2.0.0

Note: I do not know how to do batch insert a spatial (as it is my understanding that I need to create a layer with that WKT, add these nodes to that layer)
I looked at the code 'SpatialIndexImplementation' nodeIndex. If I have to do batch insert, adding to this index would be sufficient?

The stack trace is as shown below:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.store.BufferedIndexOutput.<init>(BufferedIndexOutput.java:26)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:438)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
    at org.apache.lucene.index.FormatPostingsPositionsWriter.<init>(FormatPostingsPositionsWriter.java:43)
    at org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:57)
    at org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
    at org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:51)
    at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
    at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
    at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
    at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
    at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
    at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
    at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
    at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
    at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:392)
    at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:634)
    at org.neo4j.index.impl.lucene.LuceneDataSource.refreshSearcher(LuceneDataSource.java:505)
    at org.neo4j.index.impl.lucene.LuceneDataSource.refreshSearcherIfNeeded(LuceneDataSource.java:635)
    at org.neo4j.index.impl.lucene.LuceneDataSource.getIndexSearcher(LuceneDataSource.java:577)
    at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:293)
    at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:251)
    at org.neo4j.gis.spatial.indexprovider.LayerNodeIndex.findExistingNode(LayerNodeIndex.java:152)
    at org.neo4j.gis.spatial.indexprovider.LayerNodeIndex.add(LayerNodeIndex.java:132)
    at org.neo4j.gis.spatial.indexprovider.LayerNodeIndex.add(LayerNodeIndex.java:1)
    at org.springframework.data.neo4j.fieldaccess.IndexingPropertyFieldAccessorListenerFactory$IndexingPropertyFieldAccessorListener.valueChanged(IndexingPropertyFieldAccessorListenerFactory.java:86)
    at org.springframework.data.neo4j.fieldaccess.DefaultEntityState.notifyListeners(DefaultEntityState.java:137)
    at org.springframework.data.neo4j.fieldaccess.DefaultEntityState.setValue(DefaultEntityState.java:114)
 

[Michael Hunger]

How many nodes are you adding at once and how big is your Java heap?

You don't have to create a layer, the index automatically creates the layer.

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Pattabiraman]

Michael,

This happens even with Single node. I have a total of 41k addresses and the code fails while inserting 5600th node. Tried in batches of 1000s to single digit. Could this be a data problem? Same data and code works fine in 1.9.5 even with commit interval of 1000. I am attaching the SDN class and the method that loads data.

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/TnwBO3xKrv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.

[Pattabi]

MH,
If I have to replicate the SDN's behavior for adding a Node to spatial index, do I need to do all the following?
Create a Geom node, add to index by calling add(node,key, value) and create relationship types - RTREE_REFERENCE?

[Michael Hunger]

Just with the core api.

db.index().forNodes("spatial").add(node, "dummy","dummy");

depending on the config 

node has to have geo-properties (lat, lon)

or

node has to have geo-properties (point in well-known-text (wkt))

SDN uses this index-config: 

MapUtil.stringMap(

                      IndexManager.PROVIDER, "spatial", "geometry_type" , "point","wkt","wkt")

and "wkt" as property-name for the well-known-text-point.

http://docs.spring.io/spring-data/data-neo4j/docs/2.3.x/reference/htmlsingle/#reference_spatial

Michael

[Pattabi]

Thanks Michael. I will try this.

I tried the following and getting an error while adding the SpatialDatabaseRecord to the Spatial Index Impl.
The error is
Caused by: javax.transaction.RollbackException: Tx status is: STATUS_MARKED_ROLLBACK
    at org.neo4j.kernel.impl.transaction.TransactionImpl.enlistResource(TransactionImpl.java:237)
    at org.neo4j.kernel.impl.transaction.xaframework.XaConnectionHelpImpl.enlistResource(XaConnectionHelpImpl.java:103)
    at org.neo4j.kernel.impl.index.IndexConnectionBroker.acquireResourceConnection(IndexConnectionBroker.java:56)
    ... 6 more



        IndexImplementation indexImpl = new SpatialIndexImplementation (graphService);        
        Index<Node> nodeIndex = indexImpl.nodeIndex("Geocode", SpatialIndexProvider.SIMPLE_WKT_CONFIG);
        Iterable<Node> allNodes = graphService.getAllNodes();
        for(Node node: allNodes){
            Transaction tx = graphService.beginTx();
            try{
            if(node.hasProperty("addressLine1")){
                Node added = graphService.createNode();
                added.setProperty("wkt", node.getProperty("wkt"));
                System.out.println("Adding node " + added.getId());
                nodeIndex.add(added, "addressLine1", node.getProperty("addressLine1"));
            }
            }catch(Exception e){
                e.printStackTrace(System.out);
                tx.failure();
            }
            tx.success();

[Michael Hunger]

You forgot tx.finish();

What is the content of your wkt field?

Transaction tx = graphService.beginTx();
do-stuff
tx.succes();
tx.finish();

[Pattabi]

Thanks :) just noticed that it was a placebo. will fix.
the WKT is in this format - POINT( %f %.2f )
wkt field looks fine -

[Pattabi]

Michael,

Was able to add the nodes to layer.
When I execute 'withinDistance' using the spatial index 'Geocode' I get index not found exception
I notice that 'Geocode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__' lucene index is created and is not empty. Did I miss anything?

[Michael Hunger]

Can you share the code you used?

Thanks so much.

Michael

[Pattabi]

Please find below:


    @SuppressWarnings("deprecation")
    public void addSpatial(){
        Transaction tx = graphService.beginTx();

        IndexImplementation indexImpl = new SpatialIndexImplementation (graphService);       
        Index<Node> nodeIndex = indexImpl.nodeIndex("Geocode", SpatialIndexProvider.SIMPLE_WKT_CONFIG);

        Label label = DynamicLabel.label("Address");
        GlobalGraphOperations global = GlobalGraphOperations.at(graphService);
        Iterable<Node> allNodes = global.getAllNodesWithLabel(label);
       
        for(Node node: allNodes){

           
            try{
            if(node.hasProperty("addressLine1")){
                Node added = graphService.createNode();
                added.setProperty("wkt", node.getProperty("wkt"));
                System.out.println("Adding node " + added.getId());
                nodeIndex.add(added, "addressLine1", node.getProperty("addressLine1"));
            }
            }catch(Exception e){
                e.printStackTrace(System.out);
                tx.failure();
            }
            tx.success();
                   
        }

        tx.finish();

[Michael Hunger]

I meant for the withindistiance search.

Michael

[Pattabiraman]

The cypher is:
start address=node:Geocode('withinDistance:[28.003793,-81.933069, 50.0]')
return address limit 10;

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/TnwBO3xKrv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.