Out of memory error on delete edge with 600k records

[Kapil Ranade]

Version: orientdb-community-2.1-rc3

I have an edge class with about 600k records. I just want to remove all the records and create new edges.

I cannot use "drop" or "truncate" as it says it can result in broken records. Use unsafe.

If I try to delete these 600k records OrientDB gives OutOfMemory error and creates a heap dump. 

Any solution to this problem? 600k are NOT a lot of edges.

[Kapil Ranade]

Do I have any way to delete lot of edge records without corrupting other data? 

[Andrey Lomakin]

Hi,
Could you split removal of 600k records by several transactions with 100k records ?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kapil Ranade]

Thanks,

Tried following query. Took 268.921997 sec(s) to delete 100k records !! 

delete edge worksAt where @rid< #15:100000

I get following "TIP" in server log while deleting !

 [TIP] Query 'SELECT FROM worksAt WHERE @rid < #15:100000' fetched more than 50000 records: to speed up things.....

I will delete rest of the records now in batch of 100k.

[MV-dev1]

Hello again Kapil.

I wrote a bug related to this last night -- see https://github.com/orientechnologies/orientdb/issues/4327

GC not correcting on subsquent calls after 'OutOfMemoryError: GC overhead limit exceeded' #4327

• I did mention that I was unable to DELETE as well

My problem got worse because once I started getting the GC limit exceeded errors than I couldn't even close the database.  Not sure if you're seeing this?

Also, noted there is how I added a LIMIT 10000 to my CREATE queries so that I didn't experience the problem as quickly.  You can probably use the same syntax for your DELETE with a SELECT.

Finally, I read some threads which suggested increasing RAM and swap space limits, specifically...

• set MAXDISKCACHE=8192  (I tried 8192)
• set MAXHEAP=-Xmx512m  (I tried 1024 but still had the problem)

You can make these changes in your server.[bat/sh] and/or console.[bat/sh].

600k records isn't really that big in real world scenarios, right?

My feedback to the Orient dev team...

This product is one of the best things I've seen since OOP was added to C++ in 1989!  I LOVE THE IDEAS!   But, sadly, it's not ready yet for a real production deployment as I seem to be hitting landmine bugs at every turn.

I'm personally committed to try and help make this succeed -- I'll soon be digging into the code and proposing my own bug fixes instead of just complaining without being useful.

[Kapil Ranade]

Thanks MV-dev1 for your inputs regarding the issue !

@OrientDB, Yes I feel exactly the same !!

[ant.p...@gmail.com]

I have the same problem here.

Version: orientdb-community-2.1-rc3

I would like to delete 2,900,000 edges but I have the same memory errors. Why does deleting require that much memory ?

[MV-dev1]

Can you try with 2.1-rc4?

There was a follow-up in my bug report which says this has been improved.

https://github.com/orientechnologies/orientdb/issues/4327

[W. Craig Trader]

Short form -- you're building a transaction of changes, and you're changing a lot of records.  To delete one link, you end up modifying the source and target nodes (plus the edge itself if you're using heavyweight edges).  If you build a transaction with 2.9M changes, it *is* going to take a lot of memory.

- Craig -

--