Is it possible to have multiple CBL listeners and replicators in the same app?

[Brendan Duddridge]

Hi,

I'm building my app so I can have multiple NSDocuments, each with their own database instance, open at the same time. This works great so far, but I'm just starting to add the sync capability now.

I'd like each database within the NSDocument to sync independently of each other with their counterparts on the other devices in the mesh (peer to peer).

Is it possible to have multiple CBL Listeners running at the same time and multiple replicators running at the same time?

Do I need to use a different port number for each listener? Or do I just use a different serviceType for each database I want to sync? I'm thinking a unique service type for each NSDocument's database. Or perhaps use a txtRecord to manage that, similar to the GrocerySync sample.

Thanks!

Brendan

[Jens Alfke]

On Sep 25, 2015, at 2:39 AM, Brendan Duddridge <bren...@gmail.com> wrote:

Is it possible to have multiple CBL Listeners running at the same time and multiple replicators running at the same time?

You only need one listener, because it serves all databases (of that CBLManager.) The listener is an HTTP server, and the databases are located at paths /db1, /db2, etc.

Do I need to use a different port number for each listener? Or do I just use a different serviceType for each database I want to sync?

If you want each database to have its own Bonjour service, you’d need to publish that yourself instead of letting the listener do it. It’s very little code, though; just create an NSNetService and publish it. Each one would have the same port number but would specify a different path.

Alternatively, you can use the one service that the listener provides, and use the TXT record to list the databases. The drawback is that the TXT record has limited size because it has to fit in one IP packet; it should stay below about 1500 bytes.

—Jens

[Brendan Duddridge]

Hi Jens,

Thanks for this info.

One thing to note is that each of my NSDocument instances has its own CBLManager I instantiate and keep in an ivar on my NSDocument subclass. But maybe I need to re-think how I do that. I could always have one CBLManager at the NSApp level and just create a separate database for each NSDocument instance.

Thanks!

Brendan

[Jens Alfke]

On Sep 25, 2015, at 10:00 AM, Brendan Duddridge <bren...@gmail.com> wrote:

One thing to note is that each of my NSDocument instances has its own CBLManager I instantiate and keep in an ivar on my NSDocument subclass.

Oh — yeah, in that case you need a listener for each Manager because they can’t share. Each listener has to run on a different TCP port.

But maybe I need to re-think how I do that. I could always have one CBLManager at the NSApp level and just create a separate database for each NSDocument instance.

CBLManager keeps all its databases in a single folder, so your document model does require a separate CBLManager for each document.

(Also, I know I’ve said this before, but keeping Couchbase Lite databases in user-visible documents isn’t supported. If the user moves a document in the filesystem while it’s open, CBL will definitely get confused, because various components, including ForestDB, remember filesystem paths and assume that those don’t change.)

—Jens

[Brendan Duddridge]

Hi Jens,

Yes, I know you had said that. I'm still storing my documents in the user's Documents app sandbox area though.

What's strange is I did some testing with renaming the document file while it was open and everything still worked fine as far as database reads and write. I haven't started syncing things yet though. I figured renaming the document while it was opened would have generated some sort of database access error, but it didn't. Strange.

Thanks,

Brendan

[Jens Alfke]

On Sep 25, 2015, at 12:16 PM, Brendan Duddridge <bren...@gmail.com> wrote:

What's strange is I did some testing with renaming the document file while it was open and everything still worked fine as far as database reads and write. I haven't started syncing things yet though. I figured renaming the document while it was opened would have generated some sort of database access error, but it didn't. Strange.

Both ForestDB and SQLite keep the database file open and just issue read/write calls on the file descriptor, for the most part. That works fine even if the file gets moved.

The things that I believe will fail, for a ForestDB-based database, are

- Compacting a database (this creates a new db file in the directory and eventually replaces the old file with the new one)

- Querying or indexing a view whose backing db file wasn’t already open at the time

- Reading or creating attachments

I’d like to not have these problems, but they might be hard to fix. In particular, ForestDB has file paths baked into its implementation at a pretty deep level.

—Jens

[Brendan Duddridge]

Ah that's very interesting. And ya, I didn't test those kinds of things. Just reading and updating records (not attachments or re-indexing). I will try that though just to see what happens.

Yes, it would be great if you could move the database without any issues like that. I will add that as a feature request to Github.

Well I think what I can do then is throw up a notification if the app detects the document has been renamed or moved and to tell the user to put it back and close the document first before doing that sort of thing. Plus ample warning up front though. I think the benefits of a document based database are just too great though to not try and give this a go. So far I'm really enjoying being able to open multiple documents, each with their own unique set of data. Plus it makes it easy for me to allow a user to encrypt some databases but not others (for less private data).

Thanks!

Brendan

[Brendan Duddridge]

Hi Jens,

Further to the multiple CBL listeners, do you think there might be a problem with using port 0 to generate an ephemeral unique port number instead of hard coding port numbers?

I guess I can imagine a problem if someone closes a document on one device which is still opened on another, then re-opens that document, sync wouldn't be able to automatically continue without further network discovery to determine the updated port number to connect to.

If that's the case then I'd need to either ask the user to enter in a port number per document (a terrible idea) or generate a random unused port number and store that inside the document's prefs to be used again later. Something that all devices can connect to later on. Could also be a problem if at some later time another service used that same port when the document was closed. If the document was opened then it wouldn't be able to use its stored port.

Thanks,

Brendan

[Jens Alfke]

On Sep 25, 2015, at 2:48 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I guess I can imagine a problem if someone closes a document on one device which is still opened on another, then re-opens that document, sync wouldn't be able to automatically continue without further network discovery to determine the updated port number to connect to.

That shouldn’t be a problem if you’re using an approach like the one in my P2P library that’s in the GrocerySync branch. The peers find each other by using the name and UUID of the advertised service, not by remembering an IP address or port number. The service gets resolved to an address/port when it’s time to start the replication.

—Jens

[Brendan Duddridge]

Oh excellent!

Ok, sounds good.

Thanks for all your help Jens, you rock!!!

Brendan