performance when deleting large numbers of nodes

[John Fry]

Hello All,

I have a graph of about 200M relationships and often I need to delete a larges amount of them.

For the proxy code below I am seeing huge memory usage and memory thrashing when deleting about 15M relationships.

When it hits tx.close() I see all CPU cores start working at close to 100% util and thrash for > 30mins.

I need this to work in <5mins ideally.

(note when I execute large amounts of changes to properties or create large amounts of new properties I don't have such issues)

Any advice? Why is this happening?

Regards, John.

         int txc = 0;

         // serially delete the links

         try ( Transaction tx = db.beginTx() ) {

    for (int i=0; i<deletedLinks.size(); i++) {

    Relationship rel = db.getRelationshipById(deletedLinks.get(i));

    rel.delete();

        txc++;

    if (txc>50000) {

    txc=0;

    tx.success();

    }

    }

            tx.success();

    tx.close();

        }

        catch (Exception e) {

            System.out.println("Exception link deletion: " + e.getMessage());

        }

[Clark Richey]

You need to periodically commit.  Holding that many transactions in memory isn't efficient.  

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Fry]

Thanks Clark - is there any good/recommended way to nest the commits?

Thx JF

[Clark Richey]

Don't them. Just create a counter and every x deletes commit the transaction and open a new one. 

Sent from my iPhone

[John Fry]

Clark - this works. It is still slow. I guess multithreading may help some.... 

        Transaction tx = db.beginTx();

    //try ( Transaction tx = db.beginTx() ) {

    for (int i=0; i<deletedLinks.size(); i++) {

    Relationship rel = db.getRelationshipById(deletedLinks.get(i));

    rel.delete();

     txc++;

    if (txc>50000) {

    txc=0;

    tx.success();

    tx.close();

    tx = db.beginTx();

    }

    }

            tx.success();

    tx.close();

        //}

        //catch (Exception e) {

        //    System.out.println("Exception link deletion: " + e.getMessage());

        //}

[Clark Richey]

Yes. That's a lot to delete doing it in parallel will definitely help

Sent from my iPhone

[Michael Hunger]

Shouldn't be slow. Faster disk. Concurrent batches would help. 

Von meinem iPhone gesendet

[Chris Vest]

If you perform deletes in parallel, it can be worth investing some time in making the code smart enough to choose disjoint data sets in transactions that run in parallel; e.g. no node should be start or end node in more than one parallel transaction at a time. That way they won’t contend on locks, or worse, run into deadlocks and roll back.

--

Chris Vest

System Engineer, Neo Technology
[ skype: mr.chrisvest, twitter: chvest ]