Questions about 2.0

[Brendan Duddridge]

Hi Jens,

Great work on 2.0! I'm super stoked about being able to work with it once it has CBLModel and replication support.

I know there's some debate about whether or not map/reduce will be available in 2.0. I do love the sound of the new query engine and I'm thrilled to convert from map/reduce over to it. Hopefully that'll be possible. I have some very complex map/reduce functions in my app.

But aside from map/reduce, are there any other things that are in 1.x that won't be coming to 2.0?

A couple things I hope are going to be there eventually are:

1. Peer-to-peer replication

2. This function from CBLDocument:

BOOL success = [newDoc putExistingRevisionWithProperties:jsonObject

attachments:attachmentProps

    revisionHistory:revIds

    fromURL:nil

                      error:&error];

I use putExistingRevisionWithProperties extensively for unarchiving and now for providing CloudKit syncing for my customers. So far it works really well for me.

Thanks!

Brendan

[Jens Alfke]

On Feb 16, 2017, at 12:03 PM, Brendan Duddridge <bren...@gmail.com> wrote:

But aside from map/reduce, are there any other things that are in 1.x that won't be coming to 2.0?

You won’t have full access to a document’s revision tree, or the ability to ignore conflicts and have a document with multiple branches of its history (which in all the cases I’ve seen happens only accidentally, from someone forgetting to check for conflicts.) CBL isn’t a version control system, and providing that level of history turns out to be quite expensive, especially for the replicator.

We’re currently not planning to have geo-query in 2.0. It's only ever been available on iOS/Mac, and I don’t know how many people have used it. It wouldn’t be that hard to reimplement in LiteCore and expose in the public API, but it’s lower priority than other features including full-text search. (If it’s important to you, let us know!)

P2P will definitely be supported. In fact one of our largest paying customers uses it.

Inserting existing revisions should be supported too, although it isn’t in the current 2.0 API spec.

—Jens

[Brendan Duddridge]

Hi Jens,

I'm glad that inserting existing revisions will be supported and P2P as well.

We’re currently not planning to have geo-query in 2.0. It's only ever been available on iOS/Mac, and I don’t know how many people have used it. It wouldn’t be that hard to reimplement in LiteCore and expose in the public API, but it’s lower priority than other features including full-text search. (If it’s important to you, let us know!)

However, geo queries were a big feature of my Tap Forms 5 release and it will be difficult for me to migrate to CBL 2 without them. Or I'll have to remove that whole feature, or at least cripple it. My app allowed you to filter a list of records based upon the area of the map that was being displayed at that moment. If you zoomed in to the map with marked locations, fewer records would be fetched. So it made a great way of organizing your data by location.

Here's a quick 3 minute video that shows how it's used in Tap Forms 5:

https://vimeo.com/173863608

The presenter in the screencast didn't really show the "Filter records by area" function other than to say you can do it. But the Geo query stuff in CBL is the driver behind that feature.

So hopefully there's a way to get it in there. It's a very useful feature.

Thanks,

Brendan

[Jens Alfke]

On Feb 16, 2017, at 9:22 PM, Brendan Duddridge <bren...@gmail.com> wrote:

However, geo queries were a big feature of my Tap Forms 5 release and it will be difficult for me to migrate to CBL 2 without them.

Good to know.  We’ll keep track of how many people need this. I’m pretty sure that even if it’s not in 2.0 it’ll go into 2.1 … it’s such a useful feature for mobile apps in particular.

—Jens

[Brendan Duddridge]

I just thought of another feature I use regularly in CBL 1.4. Local Documents.

Will 2.0 still support them? I use it extensively for CloudKit syncing that I recently added. Plus I also store other bits of data there that don't need to be synced, such as a list of devices that are setup for peer-to-peer syncing.

Thanks,

Brendan

[Jens Alfke]

> On Mar 27, 2017, at 10:32 AM, Brendan Duddridge <bren...@gmail.com> wrote:
> I just thought of another feature I use regularly in CBL 1.4. Local Documents.
>
> Will 2.0 still support them? I use it extensively for CloudKit syncing that I recently added. Plus I also store other bits of data there that don't need to be synced, such as a list of devices that are setup for peer-to-peer syncing.

They exist at the LiteCore layer, but they’re not exposed in the 2.0 public API. It wouldn’t be hard to add them, though. What do you think, Pasin?

—Jens

[Brendan Duddridge]

That would be awesome. A simple way to query based on keys would be great too. Right now when I want to clear out a whole bunch of key/values from the local document store I have to fetch regular documents, loop through them, then delete the ones that have matching keys. I'm using this for storing CloudKit sync metadata. When you receive a response from CloudKit, it contains metadata that needs to be stored so that the next time you go to sync, you can create a CKRecord that already has the metadata in it and it knows to match it up with a CKRecord on the server. My keys all have a prefix and it would be great to be able to delete them all in one go via a prefixed search. But just the same local document store that we have today would be great to still have in 2.0 (or 2.1 when you can get to adding back geo queries).

[pa...@couchbase.com]

Thanks Brendan! Currently local document is not in 2.0.0 API. As Jens said, it's not difficult to add APIs for it. We are taking the approach to have minimum set of the API as much as possible. We would like to hear from developers about what they need to build up a richer set of the API. I think we need a way for developers to give us feedback for 2.0 features. I will bring this up to our PM.

You can also simply emulate the local document by creating a new database only for storing data that you don't want to sync. By doing that you get the query feature (even now with 1.x) :)

[devlist]

Jumping in to this thread …

Will CBLView and createQuery should be around?
I understand the wider appeal of introducing SQL type queries to NoSQL
However I prefer the way that CBLQuery is closer to the underlying model / indexes.

Thanks,

Martin

[pa...@couchbase.com]

CBLView and Map/Reduce are not in 2.0.0.

- Pasin

[Jens Alfke]

On Mar 28, 2017, at 9:58 AM, pa...@couchbase.com wrote:

CBLView and Map/Reduce are not in 2.0.0.

There were a couple of reasons for this decision:

• We found that by using declarative queries that translate into SQLite queries we get much better performance, especially for indexing. Map/reduce indexes have to be represented as SQLite tables, which is inefficient.
• Declarative queries can do things, like joins, that can’t be done (or are very awkward to do) with map/reduce.
• Most developers coming to Couchbase Mobile are unfamiliar with map/reduce, and there’s a learning curve (as reflected in the support requests we get.)
• We’re trying to bring the development experiences on mobile and server closer together, and N1QL is the primary query mechanism on server.

This is open for debate — part of why we’re seeding 2.0 so early is to get feedback on API/feature changes — but the reality is that we have a very small development team and have to constrain what features we support.

—Jens

[Brendan Duddridge]

Totally understand not supporting CBLView and Map/Reduce.

And you're right, I can manually support local documents with a separate database file just for that that's not synced. It was just convenient to call putLocalDocument on the current database.

I've just been patiently waiting for CBLModel, Replication and a migration function before I can start trying out version 2. I see you've been working on replication, so that's awesome.

[Jens Alfke]

On Mar 28, 2017, at 4:59 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I've just been patiently waiting for CBLModel, Replication and a migration function before I can start trying out version 2. I see you've been working on replication, so that's awesome.

Replication is coming in DP4, this Friday. We’ll be including a build of Sync Gateway that supports the new replication protocol.

Migration may be in the next build … I think joins will be my top priority, but if I get that done I’ll do the DB updater.

Models might be in the next build too, but that’s more Pasin’s department :)

—Jens

[Brendan Duddridge]

Replication is coming in DP4, this Friday. We’ll be including a build of Sync Gateway that supports the new replication protocol.

By "new replication protocol", will that still be compatible with CouchDB and Cloudant? Will 2.0 be able to sync with 1.4 over peer-to-peer?

Just curious what the requirements will be for syncing in 2.0. 

[Brendan Duddridge]

Hi Jens,

You may have missed seeing my question last week, so I thought I'd post it again:

You said:

We’ll be including a build of Sync Gateway that supports the new replication protocol.

By "new replication protocol", will that still be compatible with CouchDB and Cloudant? Will 2.0 be able to sync with 1.4 over peer-to-peer?

Just curious what the requirements will be for syncing in 2.0.  

Thanks,

Brendan

[devlist]

I am also keen to find out about peer to peer support…

Martin

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/246ebd6d-73f1-405d-8f77-abd06397198a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Traun Leyden]

Hey Brendan,

See below.

On Tue, Apr 4, 2017 at 8:58 PM, Brendan Duddridge <bren...@gmail.com> wrote:

By "new replication protocol", will that still be compatible with CouchDB and Cloudant?

It won't, in fact it won't use HTTP at all.  It will use BLIP-over-websockets. 

 

Will 2.0 be able to sync with 1.4 over peer-to-peer?

I don't believe there will be any 1.4 sync compatibility on the couchbase lite level (due to reasons mentioned above), although Sync Gateway will support syncing with both 1.4 and 2.0 couchbase lite clients.

Hope this helps!

Traun

 

Just curious what the requirements will be for syncing in 2.0.  

Thanks,

Brendan

--

You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

[Brendan Duddridge]

By "new replication protocol", will that still be compatible with CouchDB and Cloudant?

It won't, in fact it won't use HTTP at all.  It will use BLIP-over-websockets. 

Umm... are you serious? No support for syncing with anything other than a Couchbase Sync Gateway server? That's definitely a deal breaker for me. :(

Why would you do that? My app syncs with IBM Cloudant and Apache CouchDB 2.0 plus peer-to-peer syncing.

I guess I'm going to be on 1.4 forever then. Either that or I'll have to go through a whole new migration process. Maybe to Realm or YapDatabase or something like that. It was already a difficult transition to migrate from raw SQL code to Couchbase Lite. It took me nearly a year.

Please, I hope you're not serious.

Brendan

[Traun Leyden]

I'm chasing this down to make sure I have my facts straight -- will get back to you.

--

You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/738b9882-d178-431f-830e-706881b81987%40googlegroups.com.

[Brendan Duddridge]

Hi Traun,

Thanks. I appreciate that. I'm sure I'm not the only one that syncs Couchbase Lite to other CouchDB compatible services.

Thanks,

Brendan

On Wednesday, April 12, 2017 at 4:06:48 PM UTC-6, Traun Leyden wrote:

I'm chasing this down to make sure I have my facts straight -- will get back to you.

On Wed, Apr 12, 2017 at 2:35 PM, Brendan Duddridge <bren...@gmail.com> wrote:

By "new replication protocol", will that still be compatible with CouchDB and Cloudant?

It won't, in fact it won't use HTTP at all.  It will use BLIP-over-websockets. 

Umm... are you serious? No support for syncing with anything other than a Couchbase Sync Gateway server? That's definitely a deal breaker for me. :(

Why would you do that? My app syncs with IBM Cloudant and Apache CouchDB 2.0 plus peer-to-peer syncing.

I guess I'm going to be on 1.4 forever then. Either that or I'll have to go through a whole new migration process. Maybe to Realm or YapDatabase or something like that. It was already a difficult transition to migrate from raw SQL code to Couchbase Lite. It took me nearly a year.

Please, I hope you're not serious.

Brendan

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.

[Jens Alfke]

On Apr 12, 2017, at 2:35 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Umm... are you serious? No support for syncing with anything other than a Couchbase Sync Gateway server? That's definitely a deal breaker for me. :(

First off I want to say that I sympathize. I really appreciate the contributions you’ve made, Brendan, and I feel bad about causing problems for you. There are times when moving forward requires breaking compatibility, and that can suck for some users. I wish it didn’t have to be that way, but after working with the CouchDB replication protocol for years it became clear that we had to move to something else.

Why would you do that? My app syncs with IBM Cloudant and Apache CouchDB 2.0 plus peer-to-peer syncing.

In short: For reasons of performance, code size, reliability, and cross-platform compatibility.

We've been using the CouchDB replication protocol for over five years, and found that it has serious problems due to its design. A lot of that comes from being built on HTTP. REST is great for general-purpose APIs, but replication needs throughput, and the overhead of making all those HTTP calls adds up. Being dependent on platform HTTP libraries also means that we get subtly different behaviors depending on how they handle things like cookies, chunked encoding, MIME, authentication, etc. And the way the replication protocol handles attachments is really troublesome. (Just sneak up behind any Couchbase Lite engineer and say "revpos!", and watch them scream.)

I know that common protocols and interoperability are good things. I've been a big believer in compatibility and a voice for it at Couchbase. But we're being held back by it. As a big decentralized open-source project, Apache CouchDB moves slowly. (It took them four or five years to integrate the clustering extensions that Cloudant developed, and release CouchDB 2.0.) I don’t think it would have been effective to build a new protocol through that organization. The new protocol is open, and IMHO a lot easier to implement fully than the old one; we’d be happy for other software to adopt it.

[For what it's worth: Damien Katz, who created CouchDB and was a founder of Couchbase, is totally in favor of what we're doing. He told me (paraphrasing) "Oh yeah, I had no idea what I was doing when I made up that protocol ten years ago; you should totally make a second generation one."]

I guess I'm going to be on 1.4 forever then. Either that or I'll have to go through a whole new migration process. Maybe to Realm or YapDatabase or something like that.

We're going to be maintaining & supporting Couchbase Lite 1.4 for at least another year. (We have service contracts to customers that guarantee that.) And of course, since it’s open source you or anyone else can continue to maintain or develop it.

My (biased) opinions on migration:

Realm has a nice client API, but their replication support is brand-new and I have doubts about it. You have to install and run their server (it's not hosted AFAIK), and the server is not free and not open source. The replication protocol is a black box too. Another factor is that Realm's database has a rigid schema, like SQLite or Core Data — I don't know how they're going to reconcile that with replication, because if the client hasn't migrated to the same schema version as the server, data interchange is a mess.

I haven't looked at YapDatabase closely. On paper it sounds nice (and a lot like Couchbase Lite!) I'm not sure how well syncing via Apple's CloudKit will work in practice, especially with conflicts, and there's the obvious factor that it completely rules out supporting any non-Apple OS.

(A final thought:t it would be a Simple Matter Of Programming™ for someone to build a CouchDB-compatible replicator on top of Couchbase Lite 2.0. Just pull the replicator code out of any platform of 1.x, change the API calls to fit 2.0, and superglue it on. OK, it wouldn’t be simple, but not rocket science either.)

—Jens

[Brendan Duddridge]

Hi Jens,

Thanks for your reply. 

(A final thought:t it would be a Simple Matter Of Programming™ for someone to build a CouchDB-compatible replicator on top of Couchbase Lite 2.0. Just pull the replicator code out of any platform of 1.x, change the API calls to fit 2.0, and superglue it on. OK, it wouldn’t be simple, but not rocket science either.)

And is it possible I could pay you to write that compatibility layer?

I don't have a problem with moving forward. I just need a path that I can follow that lets me take my thousands of customers with me.

Will there be a cloud service my customers can subscribe to that provides compatibility with the 2.0 protocol? They would need to migrate their data from IBM Cloudant to whatever service that is.

Will Couchbase Sync Gateway have a proper web UI that lets customers setup and configure it? Although Apache CouchDB isn't super user friendly, I've managed to write some documentation and it has a web UI that allows some of my more advance customers to be able to setup their own sync servers. But the vast majority of my customers are using IBM Cloudant or peer-to-peer syncing.  But in my opinion IBM Cloudant is much easier for people because there's no requirement that the devices you're syncing with have to all be online at the same time. 

In an older version of my app, before switching to Couchbase, I was using raw SQL with FMDB and I integrated it with the Dropbox Sync API. Spent a bunch of time building that out. The day I released the update to support Dropbox, I got an email from them telling me they're cancelling support for the Dropbox Sync API. In fact, their Sync API servers are shutting down this June. So I spent about a year migrating my code over to Couchbase Lite because from your website you had a great sync story and you were actively working on this. It's been great being able to sync automatically between devices. You guys are far smarter than I am and have built a great syncable database.  But now I'm finding myself in the same situation as I was with Dropbox. 

If there's any way to have a compatibility layer at least to provide some sort of transition that would be great. I only need it for iOS and Mac. So if I can hire you to build that for me that would be great. Well, as long as it doesn't break my bank account :)

Thanks,

Brendan

[Dmitry Antonov]

+1 for Geo-queries

[Jens Alfke]

> On Aug 6, 2017, at 10:46 PM, Dmitry Antonov <anton...@gmail.com> wrote:
>
> +1 for Geo-queries

At this point it’s looking very unlikely we’ll have time to add geo for 2.0. :( But I’m hoping it will go into 2.1.

(SQLite’s R-tree extension does the hard work; but there’s still a bunch of glue required to integrate it into our schemaless document model and our N1QL-like query engine, plus the public API design, plus testing.)

—Jens

[Dmitry Antonov]

Hi Jens, Is Java SDK support for non-android apps in scope for 2.0?

Dmitry

[Jens Alfke]

On Oct 24, 2017, at 8:25 PM, Dmitry Antonov <anton...@gmail.com> wrote:

Hi Jens, Is Java SDK support for non-android apps in scope for 2.0?

I don’t work on Java, but I believe that’s going to be in 2.1.

—Jens

[Alex Zavatone]

I know this is very late in the thread, but regarding iOS peer to peer replication in Couchbase, I am looking for "best approach" advice on how to solve this peer to peer replication problem.

Our app is built in Swift and will be operating in an isolated hospital on iPads and there is no guarantee of a stable and reachable network.  Assuming a maximum of 9 devices, we opted to test the stability of peer to peer by using Apple's peer to peer communication and discovery to identify peers and join a "room".  

While this worked in the simple case, as soon as > 4 devices connected to one room, devices would randomly disconnect, reconnect to ghost instances, return error messages from deep in private classes, even with Apple's own sample code.   Using this multipeer framework would clearly create a nightmare situation when more than 4 devices were in range within another on the other side of the planet.

I've reviewed Pasin's Peer to Peer Photo example here:

https://blog.couchbase.com/photodrop/

What I'm unclear on is how is which approach is being used for device discovery.  How device discovery is performed when a device comes within range.  We need to handle this even before we start a replication.

Thanks in advance.

Alex Zavatone

On Thu, Feb 16, 2017 at 2:03 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Hi Jens,

Great work on 2.0! I'm super stoked about being able to work with it once it has CBLModel and replication support.

I know there's some debate about whether or not map/reduce will be available in 2.0. I do love the sound of the new query engine and I'm thrilled to convert from map/reduce over to it. Hopefully that'll be possible. I have some very complex map/reduce functions in my app.

But aside from map/reduce, are there any other things that are in 1.x that won't be coming to 2.0?

A couple things I hope are going to be there eventually are:

1. Peer-to-peer replication

2. This function from CBLDocument:

BOOL success = [newDoc putExistingRevisionWithProperties:jsonObject

attachments:attachmentProps

    revisionHistory:revIds

    fromURL:nil

                      error:&error];

I use putExistingRevisionWithProperties extensively for unarchiving and now for providing CloudKit syncing for my customers. So far it works really well for me.

Thanks!

Brendan

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchbase+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/dc9cbef5-2e57-4a79-b808-ff59d6d00c01%40googlegroups.com.

[Traun Leyden]

Alex, can you open a new topic for this?

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CAL%2Bx05OaO70wOSkztTMpj%3DQ4R2XbcSPbjr5qxo8qSaj_OnLJew%40mail.gmail.com.