Neo4j plugin works slowly

[Svetlana Usacheva]

Hello! I have a problem with my plugin. Firstly, i tried work with my database of 150K elements from my app using RestGraphDatabase for connection. Implementation  takes about 3 s. Then i built jar and put it in folder "plugins". So now i can use my procedures from plugin, but they work much slower (about 17 s). Can i speed up my plugin and how i can do it?

 

Procedures looks like that:

    import org.neo4j.procedure.Context;

    import org.neo4j.procedure.Description;

    import org.neo4j.procedure.Name;

    import org.neo4j.procedure.Procedure;

    public class MyTraverser{

    @Context

    public GraphDatabaseService db;

    @Procedure(value = "path")

    @Description("createPath")

    public void createPath(@Name("startPoint") String startPoint, @Name("endPoint") String endPoint) {

        try (Transaction tx = db.beginTx()) {

            TraversalDescription td = db.traversalDescription().depthFirst()

                    .uniqueness(Uniqueness.NODE_PATH)

                    .relationships(RelationshipTypes.rel, Direction.BOTH)

                    .evaluator(Evaluators.includingDepths(0, 10));

            Traverser tr = td

                    .evaluator(Evaluators.includeWhereEndNodeIs(endPoint)

                    .traverse(startPoint);

            tx.success();

        }

    }

}

[Michael Hunger]

Can you share your full code somewhere?

RestGraphDB should not be used.

Which version are you runnign on?

Your plugin doesn't do anyting you have to actually use the traverser to extract the nodes or paths.



Do your really need 10 levels of bidirectional (BOTH!) relationships? That is potentially billions of paths.
You can add a counter in the relationship-expander to see how many rels are hit.

Can't you just use shortestpath?

> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Miosota]

Yes, i didn't use RestGraphDB in my plugin. I used it in my test project for connection from Intellij idea project to neo4j database. Database consists of about 50K nodes and 100K relationships. 

 

I'm running neo4j 3.3.1. For plugin creation use neo4j-kernel 3.3.1 .

For test i created a procedure below. It finds paths from one point to another including at least one point from a set of 3 points. Traverser spend too much time to find this paths. 17-20s.  

And i really need bidirectional. And the graph later will be much more bigger (millions elements). 

May be i need to change smth in neo4j properties? I dont know.

My RAM 8G. 

dbms.memory.heap.initial_size=1000m

dbms.memory.heap.max_size=2000m

dbms.memory.pagecache.size=5000m

@Procedure(value = "mytraverser.pathIncludingAtLeastOneNode")
@Description("createIncludingAtLeastOneNode")
public void createIncludingAtLeastOneNode(@Name("startPoint") String startPoint, @Name("endPoint") String endPoint) {

    try (Transaction tx = db.beginTx()) {
        TraversalDescription td = db.traversalDescription().depthFirst()
                .uniqueness(Uniqueness.NODE_PATH)

                .uniqueness(Uniqueness.RELATIONSHIP_PATH)

                .relationships(RelationshipTypes.rel, Direction.BOTH)
                .evaluator(Evaluators.includingDepths(0, 10));

        Traverser tr = td

                .evaluator(Evaluators.includeWhereEndNodeIs(db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904013846978").getId())))
                .evaluator(new PathEvaluator.Adapter() {
                    @Override
                    public Evaluation evaluate(Path path, BranchState branchState) {
                        if (path.length()==0) return Evaluation.INCLUDE_AND_CONTINUE;
                        else {
                            ArrayList<Node> listNode = new ArrayList<>();
                            for (Node n:path.nodes()){
                                listNode.add(n);
                            }
                            if (listNode.contains(db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904013846972").getId())) ||
                                    listNode.contains(db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904013846980").getId())) ||
                                    listNode.contains(db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904013846974").getId()))) {
                                return Evaluation.INCLUDE_AND_CONTINUE;
                            } else {
                                return Evaluation.EXCLUDE_AND_CONTINUE;
                            }
                        }
                    }
                })
                .sort(new PathSorting("weight"))
                .traverse(db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904413851091").getId()));

        createPath(tr, "IncludingAtLeastOneNode");
        tx.success();
    }
}

среда, 31 января 2018 г., 14:59:09 UTC+3 пользователь Michael Hunger написал:

[Michael Hunger]

Do you have an index or constraint on :nodes(id) ?

If not create one

Don't do this

db.getNodeById(db.findNode(Labels.nodes, "id", "9149892904013846978").getId()))

but

db.findNode(Labels.nodes, "id", "9149892904013846978")

Find the nodes once upfront and assign them to variables

and not during every step of the travesal.

Don't use a list for lookup but a Set

You can use Uniqueness.NODE_PATH to assure that a node only appears once in a path

Only ONE Uniqueness can be used.

As you continually check for nodes on each  branch in the evaluator, you only have to test the endNode

return includeNodesSet.contains(path.endNode()) ? return INCLUDE_AND_CONTINUE : EXCLUDE_AND_CONTINUE;

What does createPath do?

your procedure doesn't return anything?

You don't use the start-point.

Try to leave off the sorting and see if it helps.

Run the procedure multiple times to see what the warmed up time is.

Add a counter to your evaluator and output that counter so you see how many Paths it touches.

If possible you should limit it to certain rel-types or labels.

[Svetlana Usacheva]

Thanks a lot! Try follow your recommendations. 

createPaths just show me found paths. It takes about second.

And yeah, i tried to leave sorting and without it plugin worked faster. 

Here my sort and now i use only sorting by "weight":

public class PathSorting implements Comparator<Path>{

    private String sortType;

    public PathSorting(String sortType){
        this.sortType=sortType;
    }

    @Override
    public int compare (Path path1, Path path2){

        switch (sortType){
            case "length" :
                if (path1.length()!=path2.length()) {
                    return path1.length()<path2.length() ? -1 : 1 ;
                }else return 0;

            case "weight" :
                long p1Weight_0 = 0, p2Weight_0 = 0;

                for (Relationship r : path1.relationships()){
                    p1Weight_0 += (long)r.getProperty("weight");
                }

                for (Relationship r : path2.relationships()){
                    p2Weight_0 += (long)r.getProperty("weight");
                }

                if (p1Weight_0!=p2Weight_0) {
                    return p1Weight_0<p2Weight_0 ? -1 : 1 ;
                }
                else return 0;
            default:return 0;
        }
    }

}

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

[Michael Hunger]

As during a sort those weight sums will be built up time and again it's better to cache them per path e.g. in an identity hashmap.

You can also use branch state for sorting afaik.

Michael

>> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

[Svetlana Usacheva]

Good idea!! Thanks again. I forgot about branch state.

>> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Neo4j" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.