Multiple databases and syncing

[Brendan Duddridge]

Hi,

I currently have all my data in a single database file. But I'm contemplating splitting it up into two database files. One to store all my metadata and another to store all my regular data. Will there be any complications with this setup in terms of synchronization or just general management of both databases? I'm thinking that I can just have a couple of CBLManagers, each managing a separate database file. When I want to write to one or the other, I'll just create my CBLModel objects to save to the specific database file the documents were created in. I think it should work, but I'm not sure what will happen when it comes to syncing. The regular, non-metadata database will reference objects in the other database file, but I don't think that should be a problem since there's no foreign key relationships in Couchbase anyway as it's all managed in the application.

Each database will also have a separate set of attachments. This should be fine too since each database file will have a separate database name anyway.

Just looking for some advice if this is a good or a bad idea and what complications I may encounter as I develop this. Things I should watch out for or make sure I take care of, etc.

Thanks,

Brendan

[Jens Alfke]

On Jul 7, 2015, at 1:22 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I currently have all my data in a single database file. But I'm contemplating splitting it up into two database files. One to store all my metadata and another to store all my regular data. Will there be any complications with this setup in terms of synchronization or just general management of both databases?

You can do it, but it adds complexity and I’m wondering what benefits you think you’d get.

The only reason I could see to do this is if you want to keep one database for local data and never sync it (although you could also do that with one database using a replication filter.) Or if you’re doing something very fancy with document IDs and somehow absolutely need overlapping namespaces.

I'm thinking that I can just have a couple of CBLManagers, each managing a separate database file.

You only need one CBLManager to manage any number of databases.

When I want to write to one or the other, I'll just create my CBLModel objects to save to the specific database file the documents were created in. I think it should work, but I'm not sure what will happen when it comes to syncing.

Each db has to be synced independently. You can create replications for both of them; it shouldn’t be a problem. They’ll sync simultaneously. (Of course it won’t be any faster. Probably slightly slower due to bookkeeping overhead.)

The regular, non-metadata database will reference objects in the other database file, but I don't think that should be a problem since there's no foreign key relationships in Couchbase anyway as it's all managed in the application.

In any CBLModel subclass that has a property that’s a reference to a document in the other database, you’ll need to override -databaseForModelProperty:, so that it knows which database to look up the docID in.

—Jens

[Brendan Duddridge]

In any CBLModel subclass that has a property that’s a reference to a document in the other database, you’ll need to override -databaseForModelProperty:, so that it knows which database to look up the docID in.

Ah excellent. I'm glad I asked about this. That'll be important to know.

One of the reasons I wanted to do this is just to have separation of regular data and metadata. The metadata information could be saved out separately to a file in order for someone to share a complete template of their database without having to include all of the regular data. I know I could probably do this in code by reading all the metadata separately from the regular data and dumping it to a separate file, but it would be super easy to simply copy the metadata database independently of the regular data database and bang, you have an empty database with everything setup already for you ready to enter your regular data.

Thanks!

Brendan

[Jens Alfke]

On Jul 7, 2015, at 3:39 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I know I could probably do this in code by reading all the metadata separately from the regular data and dumping it to a separate file, but it would be super easy to simply copy the metadata database independently of the regular data database and bang, you have an empty database with everything setup already for you ready to enter your regular data.

I think you’d be better off writing it to a different file. That way you control the file format and exactly what the file contains. Couchbase Lite database files aren’t really meant as an interchange format; they’re more of an implementation detail of how the data gets stored locally.

—Jens

[Brendan Duddridge]

I think you’d be better off writing it to a different file. That way you control the file format and exactly what the file contains. Couchbase Lite database files aren’t really meant as an interchange format; they’re more of an implementation detail of how the data gets stored locally.

Well I'm actually embedding the database into an NSDocument/UIDocument NSFileWrapper (kind of like a UIManagedDocument) so users can share the actual document files if they like (and open multiple documents at the same time if they like). So that's why I was thinking that the dual database approach would work well in that situation. Both would be embedded within the file wrapper and then I can just save out a copy without the regular database and just include the metadata database.

Thanks,

Brendan

[Jens Alfke]

On Jul 7, 2015, at 10:29 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Well I'm actually embedding the database into an NSDocument/UIDocument NSFileWrapper (kind of like a UIManagedDocument) so users can share the actual document files if they like

That makes me nervous for a number of reasons…

* A user can move a document at any time, but there is no support in Couchbase Lite for a database being moved while it’s open. I don’t know if SQLite can handle this without hand-holding; I know ForestDB can’t. And the CBLManager and the attachment store both keep absolute paths and assume they don’t change.

* Every database has a unique ID that’s used as part of storing checkpoints for replication. If you copy a document containing a database, you now have two copies of the same database, with the same UUID (which is no longer unique). That’s going to mess up checkpointing because both instances of the database will try to store the same checkpoint on the server. This isn’t fatal, but it will greatly slow down replication, because when it happens the replicator has to ignore the checkpoint and start over from scratch.

* Databases may internally store sensitive information like login session cookies, as part of the replicator’s saved state.

I could probably think of more problems if I thought about this longer :)

None of these are insurmountable, but they’d require significant engineering work inside Couchbase Lite, and no one before has ever expressed a need for databases-as-documents so it hasn’t come up as a potential feature.

—Jens

[Brendan Duddridge]

That makes me nervous for a number of reasons…

And now me too :-) I can certainly alter my approach. It's not too late for my project. 

* A user can move a document at any time, but there is no support in Couchbase Lite for a database being moved while it’s open. I don’t know if SQLite can handle this without hand-holding; I know ForestDB can’t. And the CBLManager and the attachment store both keep absolute paths and assume they don’t change.

The NSDocument's setFileURL method will be called if the document is moved by another process, so I was going to override this and close the database and re-set the CBLManager's URL and re-open it with the new URL.

* Every database has a unique ID that’s used as part of storing checkpoints for replication. If you copy a document containing a database, you now have two copies of the same database, with the same UUID (which is no longer unique). That’s going to mess up checkpointing because both instances of the database will try to store the same checkpoint on the server. This isn’t fatal, but it will greatly slow down replication, because when it happens the replicator has to ignore the checkpoint and start over from scratch.

This could be problematic even without a document based database. A user could easily copy the same database file to multiple devices and then start replication.

Does the replicator automatically detect this situation and automatically start replication from scratch? That would be ok if it did I think. Although I don't know if the replicator would set a new unique UUID for one of the databases so maybe this will be constantly problematic.

What's the best way to handle this situation?

* Databases may internally store sensitive information like login session cookies, as part of the replicator’s saved state.

This also could be a problem. Especially if the user gives a copy of their database to someone else. This can happen to anyone I guess and not just for my NSDocument idea.

Thanks,

Brendan

[Jens Alfke]

On Jul 7, 2015, at 11:43 PM, Brendan Duddridge <bren...@gmail.com> wrote:

The NSDocument's setFileURL method will be called if the document is moved by another process, so I was going to override this and close the database and re-set the CBLManager's URL and re-open it with the new URL.

That’s very prone to race conditions, since the method is called after the document is moved. 

This could be problematic even without a document based database. A user could easily copy the same database file to multiple devices and then start replication.

It’s a lot less likely if the databases are invisibly housed in Application Support.

Does the replicator automatically detect this situation and automatically start replication from scratch? That would be ok if it did I think. Although I don't know if the replicator would set a new unique UUID for one of the databases so maybe this will be constantly problematic.

The replicator will reset, but it’ll happen whenever the ‘other’ copy of the database was the last one to replicate with that server. The database never changes its UUID.

—Jens

[Brendan Duddridge]

In my app I allow users to make a backup copy of their database from one device and restore it to another if they wish. My app just makes a zip file of the sqlite file and any file attachments. I'd like to do the same with Couchbase Lite. The user can make a backup file on iOS and restore to Mac or vice versa. I'd like to be able to keep that functionality.  There's a replaceUUIDs method on CBLDatabase that seems like it would be perfect for this scenario.  And I could always add a call to that method that the user can initiate if they were to do something like replace the database file in the Finder instead of from doing a restore within the app. If they have troubles with replication I can tell them to "reset sync" to clear that up.

Although what would happen on the server side if the client databases suddenly had different UUIDs? Does the server keep track of the UUIDs for each device? I would almost think not since a device might go offline permanently at some point. Then the server would be keeping track of a potentially non-existent database.

Thanks!

Brendan

[Jens Alfke]

On Jul 8, 2015, at 10:56 PM, Brendan Duddridge <bren...@gmail.com> wrote:

In my app I allow users to make a backup copy of their database from one device and restore it to another if they wish. My app just makes a zip file of the sqlite file and any file attachments. I'd like to do the same with Couchbase Lite. The user can make a backup file on iOS and restore to Mac or vice versa. I'd like to be able to keep that functionality.

Hm. Why not just use replication? As an example, I don’t need a feature to back up my Dropbox (or my Kindle library) from one computer and restore it to another; that’s just what those services do automatically.

There's a replaceUUIDs method on CBLDatabase that seems like it would be perfect for this scenario.

Yes, except it’s not public. But of course we can’t stop you from calling it ;) I agree that it’s a good workaround for the copying-a-database problem until/unless we work out something better.

It’s best to call -compact on the database before you copy it, to keep the file size down.

Although what would happen on the server side if the client databases suddenly had different UUIDs? Does the server keep track of the UUIDs for each device?

Sort of, although passively. Every client stores a local document on the server that stores the checkpoint data. The ID of the document is derived from the local database UUID and the replication settings. The server doesn’t care what this is; as far as it’s concerned, a client is just using the /db/_local/ endpoint to store an opaque bit of data.

You’re right that this does potentially leak storage on the server if a client goes away, but it’s only a few hundred bytes per client.

—Jens