Get memory limit error when replicate a big database

[Rustam Yuzmukhametov]

In my project I need replicate big database. It contains about 2 * 10^5 documents.

I use ForestDB as database storage (7949e8d59b9b30f7711be47a121dfbda9c82f2a8).  

At first I pull the database. After successfull pull I start a push replication.

On this step I get a crash - memory limit. 

I profile the app and discover that push replication save all revision in RAM.

This logic places in CBL_ForestDBStorage.mm file in changesSinceSequence method. The method create an array of all changes.

Also I don't have this error when I use SQLite database storage.

It is possible to change push logic without necessary to save all revisions in RAM?

[Jens Alfke]

On Jun 8, 2015, at 6:46 AM, Rustam Yuzmukhametov <ryuzmuk...@gmail.com> wrote:

This logic places in CBL_ForestDBStorage.mm file in changesSinceSequence method. The method create an array of all changes.

It returns an array of CBL_Revision. When the replicator calls that method it asks for only the docID and revID, not the body, so each object will be very small (maybe 150 bytes.) CBL_SQLiteStorage has exactly the same method (it’s from a protocol they both implement) so it also returns the same type of array. So I don’t think this itself is the issue.

There may be a memory leak in the ForestDB code, or a missing @autoreleasepool inside a loop. Have you tried running in Instruments with the Leaks instrument? (And could you file a bug report, please? Thanks!)

—Jens

[Jens Alfke]

On Jun 8, 2015, at 8:44 AM, I wrote:

It returns an array of CBL_Revision. When the replicator calls that method it asks for only the docID and revID, not the body, so each object will be very small (maybe 150 bytes.)

I see the problem now. The way that -changesSinceSequence is being called by the replicator causes it to load the document bodies. It temporarily needs each doc’s body in order to find all the conflicting revisions, but then it keeps the body in the CBL_Revision object it creates … so all of the bodies get loaded into memory at once.

This should be really easy to fix, although unfortunately it’s too late for 1.1.

—Jens

[Jens Alfke]

This is fixed on the master branch now (commit 9146aa9). Thanks for reporting it!

—Jens

[Rustam Yuzmukhametov]

Thank you, Jens,

I check it now and it works correct for me.

вторник, 9 июня 2015 г., 5:35:59 UTC+5 пользователь Jens Alfke написал: