Performance question regarding replication

[Alexander Harm]

In my setup I replicate several small databases to in-memory cache-database. What disturbs me is the really poor performance. Some numbers:

doc_count = 1063

Loading dump file into in-memory PouchDB 1: 2.276 sec

Replicating from in-memory PouchDB 1 to in-memory PouchDB 2: 13.16 sec

I expected in-memory to be quite performant and the amount of time it takes for 1.000 docs kind of scares me since I'm planning to use more like a 100.000 docs in the future.

Is this normal behaviour?

And on a sidenote, are there any plans to use LokiJS as in-memory adapter?

Regards, Alexander

[Nolan Lawson]

What device and browser are you testing this on? Also the speed of PouchDB will depend a lot on the size of your documents.

In-memory PouchDB isn't as fast as it could be. We're aware of it, but there's no fix planned in the future. There are no plans for a LokiJS adapter either. Sorry about that!

- Nolan

[Alexander Harm]

I use it on

device:

Apple MacBook Pro Core2Duo, 6GB RAM, SSD

browser:

lastest Chrome

documents:

smallish, maybe 10 key-value pairs (just short strings like name etc)

Alexander

 

--
You received this message because you are subscribed to a topic in the Google Groups "PouchDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pouchdb/BdqeJoTsauE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/18a77bb3-1655-436f-9da9-85260634740f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nolan Lawson]

I'll also point out that if you plan on having 100k separate documents, then PouchDB will probably not suit your needs. PouchDB is optimized for syncing and revision management, e.g. a user with ~20 documents that are shared among colleagues and may go through many revisions. Unless you expect your users to individually modify 100k documents and then track changes across all of those 100k documents, then PouchDB's revision-management overhead will indeed cause a performance hit.

LokiJS may be a better solution in your case, if you're targeting desktop browsers. If you're targeting mobile devices, then you will probably run out of memory very quickly, so you may want to consider LocalForage, DexieJS, or YDN-DB instead, which do not have revision semantics.

- Nolan

[Alexander Harm]

90% of the docs will probably be read-only with very few changes. Can Pouch on node handle such amounts better?

Alexander

--
You received this message because you are subscribed to a topic in the Google Groups "PouchDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pouchdb/BdqeJoTsauE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/d06ea5af-1e55-449b-9cb1-bc7ca2ebee2b%40googlegroups.com.

[Jan Lehnardt]

On 16 Mar 2016, at 18:50, Alexander Harm <harm.al...@gmail.com> wrote:

90% of the docs will probably be read-only with very few changes. Can Pouch on node handle such amounts better?

If you are on a server/desktop environment, I’d recommend good ol’ CouchDB.

Best

Jan

--

You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.

To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/A25C3F84-350A-4547-9F20-B41A83F4E9D0%40googlemail.com.

[Nolan Lawson]

If 90% of the documents never change, then an even better solution in my opinion is pouchdb-dump and pouchdb-load: https://github.com/nolanlawson/pouchdb-load

Else you can wait for CouchDB 2.0 which will have faster replication. Or you can start using it in alpha right now; it's as simple as:

docker run -d -p 3001:5984 klaemo/couchdb:2.0-dev --with-haproxy --with-admin-party-please -n 1

[Alexander Harm]

Hello Nolan,

I currently use the following setup:

- a node process that listens to changes and appends them to a ndj-file

- clients use pouchdb-load to get this file from server

- clients start replicating

The idea was to load several “project”-dbs and then build from these small dbs one "cache”-db. My problem is not the server-client replication but the replication between these “project"-dbs and the “cache”-db. I just expected a replication between two in-memory-pouches to be blazingly fast. The roughly 13 seconds it took for a 1000 docs left me startled.

 Alexander

--
You received this message because you are subscribed to a topic in the Google Groups "PouchDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pouchdb/BdqeJoTsauE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/1dccd4ed-b2e2-4044-9eff-eecfca69b984%40googlegroups.com.

[Nolan Lawson]

Hi Alexander,

That's a great test scenario. I agree that replication between two in-memory PouchDBs should be fast. If it's not, then it's a great test case for profiling, etc., and I encourage anybody with the free time to do so. :)

Cheers,
Nolan