Is CouchDB/PouchDB a good solution for chat messaging?

[Aneesa Awaludin]

Hello,

We started using CouchDB/PouchDB for our in-app chat about two months ago, and of course CouchDB is growing at a fast pace. We have already received 11k+ messages so far. One of the issues I notice was the slow initial replication from CouchDB to PouchDB in users' devices. We are currently on weekly sprints, so we are deploying to App Store and Play Store quite often. Every time a user updates our app, his/her device will need to replicate his/her docs (we added a filter) from CouchDB into PouchDB.

Do you think CouchDB/PouchDB is a good solution for this? The docs will be revised thrice at max (sent --> received, unread --> read), and should stay as it is after that.

Note: We are in the process of updating our app to Ionic 2 - will it help with the initial replication issue?

Thank you.

[Dale Harvey]

So by default PouchDB isnt the ideal database for a chat messaging system, however those limitations are fairly artificial and can be fixed with some work and are issues I would like to see addressed within pouchdb.

1. PouchDB has overhead storing revisions, this can be fixed with

    new PouchDB('name', {auto_compaction: true});

The only downside to this is that you wont be able to fetch old revisions that are not in conflict, there is almost never a reason to do that however so making this the default behaviour may happen in the future. You may also want to take a look at revs_limit (https://pouchdb.com/api.html#create_database) although reducing that has a little more complicated impact (it may increase the number of conflicts)

2. The replicator will transfer old revisions and is slow for large datasets

You mentioned having a filter on your replication which is a good start, I would test your filter to ensure pouchdb is not transferring anything you dont want it to, if you are still seeing replication be slow then please file a but on it and we can see what can be done about it, I definitely want to address replication speed and have some plans in mind for it. You may also want to take a look at https://github.com/nolanlawson/pouchdb-replication-stream (+ dump / load)

Cheers

Dale

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/04dac674-dd6c-4912-99fa-3064c7205475%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nolan Lawson]

A few other solutions you can look into:

- pouchdb-dump-cli and pouchdb-load - dump a database and load it into PouchDB: https://github.com/nolanlawson/pouchdb-dump-cli

- SQLite Plugin 2 + node-websql - you can use the node-websql adapter to prebuild a PouchDB database into a .db file, and then use SQLite Plugin 2 to load the prepopulated database at startup. this is even faster than the pouchdb-load method, because the database is already built up: https://github.com/nolanlawson/node-websql

Slow replication, in general, is an issue that the Couch/Pouch community is well aware of. It should be fixed in CouchDB 2.0 and 1.7 with the introduction of the new _bulk_get endpoint. This endpoint is already available in PouchDB Server if you want to try it out (the PouchDB replicator will use it automatically).

Dale is right that auto_compaction can decrease the overhead of storing revisions (since only the winning revision is stored), but the above methods will probably do the most to improve your current performance problems.

Also FWIW CouchDB/PouchDB is probably not a database I would recommend for building a chat application, as you will have trouble scaling it unless you use filtered replication to trim old messages (which is probably most efficient to do serverside, by the way, using separate CouchDB databases, i.e. do filtered replication from big database A to smaller database B, then replicate B to the device, and maybe change B every day so it only contains the most recent messages).

As a temporary workaround, you can simply use PouchDB to query the _changes feed from the remote CouchDB and then store that locally (using any old database, not necessarily PouchDB), then blow it away every day to remove old messages. In the future we will implement the purge() function which will make trimming old documents much easier, but for now it doesn't exist.

In short: you have lots of options here. :)

- Nolan