ANN: TouchDB

[Jens Alfke]

TouchDB is a project I’ve been feverishly working on for a few weeks. It’s an investigation into the feasibility of a CouchDB-compatible database rewritten from the ground up for mobile apps. The comparison I like to make is that “if CouchDB is MySQL, then TouchDB is SQLite”. In fact, it uses SQLite as its underlying storage engine. You can read a longer justification for it on its wiki, as well as an FAQ and design document.

— It speaks CouchDB’s replication protocol. I’m pretty serious about that; I’m even documenting the protocol. 

— It also understands a large subset of the REST API, enough so that it works with CouchCocoa. I’ve got a clone of Grocery Sync working as one of the demo apps in the project.

— The current implementation is for iOS. Marty is hoping to start on an Android version soon.

TouchDB is certainly not ready for prime-time yet, but here are some current statistics to whet your appetite:

• Code size of an ‘empty’ iOS app with nothing in it but TouchDB: ~150k

• Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm)

• Size of source code: ~4000 lines of Obj-C (plus another ~2500 lines from some existing utility libraries)

What’s left to do? Probably a lot — that infamous “second 90%”. Prominently:

• Attachments

• Reduce functions and grouping

• Filters for views and replication

• Performance tuning

See the issue tracker for more.

So, what does this mean for Couchbase Mobile? Honestly, we don’t know yet. It may be that TouchDB turns out to be so awesome that it replaces embedded-CouchDB entirely in Couchbase Mobile on iOS and Android. It may be that there are still scenarios where embedded-CouchDB works better and is worth the extra overhead for some developers, in which case we’ll still support it. This is not a product announcement; it’s a technical announcement of something that isn’t a product yet, because we like to do our development in the open. We’d love your feedback or even contributions.

—Jens

[Aaron Vegh]

Hi Jens,

> • Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm)
>

I think I just wet myself.

My needs are *very* modest for my current iOS project. I'm eager to see that fever continue and have this to work with!

You go Jens!
Cheers,
Aaron.

[Layton Duncan]

Awesome! We took a look at TouchDB yesterday when it appeared (evaluating it for including in our Air Forms product). Great work, I personally love this direction. To us, the true value of CouchDB is in sync. The on disk storage mechanisms, particularly on a mobile "client" is less of a concern. As long as it retains sync compatibility with CouchDB we'd be happy.

The question we had was, is this likely to form the basis of SyncPoint? It seems many of the rationals and goals are aligned here?

Layton Duncan | Founder | Polar Bear Farm Ltd 
10 Poplar St, Christchurch 8001, NEW ZEALAND

www.PolarBearFarm.com

[Dali Kilani]

Exciting! The use case I am most interested in personally is : phonegap/callback + touchDB  + kanso/couchapp + backend sync.

This would be the winning combination for me. The biggest hurdles for doing this so far with the couchdb-based couchbase mobile are : App size and the long  "Time to interactive" barrier. TouchDB seems to be going in the right direction in solving those.

Dali

--

Dali Kilani
===========
Twitter :  @dadicool
Web : http://linkd.in/dalikilani

Phone :  (650) 492-5921 (Google Voice)
E-Fax  :  (775) 552-2982

[Iain Stubbs]

very interested ... particulaly as it will support replication

--
Cheers



Iain Stubbs
0404873041

"We are ready for any unforeseen event that may or may not occur" - George W Bush

[Pierre Metrailler]

Awesome! I truly believe this radical move is going into the right
direction. The small footprint and the startup time are very
impressive.
This being said, I see 2 controversial design decisions IMHO:
- View function implemented in native code- Lack of (plan for) HTTP
API
While I do understand the rationale behind it (simplicity,
performance, no additional JS interpreter, library VS server), I think
it nixes 2 important strengths of Couchbase Mobile. Portable views are
elegant - with Couchbase Mobile we are able to architect our database
structure and underlying views in a completely platform-agnostic
fashion. And the lack of HTTP API means no direct interaction between
UIWebView and the database - and no CouchApp - which is a major loss.
A provision for JS view functions and HTTP API would be a game changer
here.
But overall - a promising move.

On Dec 19, 8:14 pm, Jens Alfke <j...@couchbase.com> wrote:
> TouchDB<https://github.com/couchbaselabs/TouchDB-iOS/> is a project I’ve been feverishly working on for a few weeks. It’s an investigation into the feasibility of a CouchDB-compatible database rewritten from the ground up for mobile apps. The comparison I like to make is that “if CouchDB is MySQL, then TouchDB is SQLite”. In fact, it uses SQLite as its underlying storage engine. You can read a longer justification for it<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Why-TouchDB?> on its wiki, as well as an FAQ<https://github.com/couchbaselabs/TouchDB-iOS/wiki/FAQ> and design document<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-S...>.
>
> — It speaks CouchDB’s replication protocol. I’m pretty serious about that; I’m even documenting the protocol<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Replication-Algorithm>.

> — It also understands a large subset of the REST API, enough so that it works with CouchCocoa. I’ve got a clone of Grocery Sync working as one of the demo apps in the project.
> — The current implementation is for iOS. Marty is hoping to start on an Android version soon.
>
> TouchDB is certainly not ready for prime-time yet, but here are some current statistics to whet your appetite:
>
> • Code size of an ‘empty’ iOS app with nothing in it but TouchDB: ~150k
> • Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm)
> • Size of source code: ~4000 lines of Obj-C (plus another ~2500 lines from some existing utility libraries)
>
> What’s left to do? Probably a lot — that infamous “second 90%”. Prominently:
> • Attachments
> • Reduce functions and grouping
> • Filters for views and replication
> • Performance tuning

> See the issue tracker<https://github.com/couchbaselabs/TouchDB-iOS/issues> for more.

[Dave Feldman]

Rock on.

[Peter Goldsmith]

My sentiments are in align with everyone else's here, awesome

[Alexander Gabriel]

sounds awesome but since I'm developping a couchapp I hope it will not replace CouchDb for Android or iOS

Alex

2011/12/19 Peter Goldsmith <peter.n....@gmail.com>

[Nate Todd]

Yes, this is excellent.

I understand many people on this mailing list do use the http API and javascript features from both native and phonegap apps, but that audience may be large because that was a primary feature of couch and the current mobile Couchbase implementation.  I believe this project could gain a much wider audience in the iOS developer community if TouchDB (and Mobile Syncpoint) keeps going down the path of high-performance, small footprint and tight cocoa integration even if that means breaking backwards compatibility.

While dropping to lower-level technologies can often be a huge pain, the payoffs in performance, battery life and application footprint can be enormous.  Many native developers would never even consider including the current Mobile Couchbase because the tradeoffs do not align with the goals of good iOS apps (user experience always trumps developer sanity).

Thanks and keep up the great work!

[stubbsi]

Have tried incorporating touchdb into my app which is ARC ,,,

Get the following message:

ARC forbids Objective-C objects in Structs ... used
_strong_nonretained to get around it...

On linking the app, get the following messages

"_OBJC_CLASS_$_FMDatabase", referenced from:

objc-class-ref in TouchDB(TDDatabase.o)

"_OBJC_CLASS_$_CouchbaseMobile", referenced from:

objc-class-ref in CouchCocoa(CouchEmbeddedServer.o)

ld: symbol(s) not found for architecture i386

clang: error: linker command failed with exit code 1 (use -v to see
invocation)

Not sure how to make this work

/** Standard query options for views. */
typedef struct TDQueryOptions
{
NSString* startKey;
NSString* endKey;
int skip;
int limit;
BOOL descending;
BOOL includeDocs;
BOOL updateSeq;
} TDQueryOptions;

On Dec 20, 6:14 am, Jens Alfke <j...@couchbase.com> wrote:
> TouchDB<https://github.com/couchbaselabs/TouchDB-iOS/> is a project I’ve been feverishly working on for a few weeks. It’s an investigation into the feasibility of a CouchDB-compatible database rewritten from the ground up for mobile apps. The comparison I like to make is that “if CouchDB is MySQL, then TouchDB is SQLite”. In fact, it uses SQLite as its underlying storage engine. You can read a longer justification for it<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Why-TouchDB?> on its wiki, as well as an FAQ<https://github.com/couchbaselabs/TouchDB-iOS/wiki/FAQ> and design document<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-S...>.
>

> — It speaks CouchDB’s replication protocol. I’m pretty serious about that; I’m even documenting the protocol<https://github.com/couchbaselabs/TouchDB-iOS/wiki/Replication-Algorithm>.

> — It also understands a large subset of the REST API, enough so that it works with CouchCocoa. I’ve got a clone of Grocery Sync working as one of the demo apps in the project.
> — The current implementation is for iOS. Marty is hoping to start on an Android version soon.
>
> TouchDB is certainly not ready for prime-time yet, but here are some current statistics to whet your appetite:
>
> • Code size of an ‘empty’ iOS app with nothing in it but TouchDB: ~150k
> • Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm)
> • Size of source code: ~4000 lines of Obj-C (plus another ~2500 lines from some existing utility libraries)
>
> What’s left to do? Probably a lot — that infamous “second 90%”. Prominently:
> • Attachments
> • Reduce functions and grouping
> • Filters for views and replication
> • Performance tuning

> See the issue tracker<https://github.com/couchbaselabs/TouchDB-iOS/issues> for more.

[stubbsi]

clarification

The errors I received on building are:

On linking the app, get the following messages
"_OBJC_CLASS_$_FMDatabase", referenced from:
      objc-class-ref in TouchDB(TDDatabase.o)
"_OBJC_CLASS_$_CouchbaseMobile", referenced from:
    objc-class-ref in CouchCocoa(CouchEmbeddedServer.o)

[Jens Alfke]

If you added the TDb source files to your project, don't forget to add the sources in vendor/ too -- that's where FMDatabase comes from.

Also, CouchEmbeddedServer doesn't work with TDb yet. Look at the iOS demo's app delegate to see how it sets up the 'server'.

--Jens [via iPhone]

[Iain Stubbs]

thanks

I am not acutally using couchembeddedserver --- will sift through my code...

[Iain Stubbs]

Not referencing couchembeddedserver at all.  But still in couchcocoa ... not sure where to go from here...

[Dave Cottlehuber]

Hi Jens,

this is really exciting, and the name is cool too.

I'd always thought that CouchDB needed an RFC-like approach for JSON
sync, to enable as many diverse implementations as possible. I'll be
keen to work with you on documenting the replicator as well.

A+
Dave

[Jens Alfke]

On Dec 19, 2011, at 2:05 PM, Pierre Metrailler wrote:

Portable views are
elegant - with Couchbase Mobile we are able to architect our database
structure and underlying views in a completely platform-agnostic
fashion.

These could be implemented as an add-on incorporating SpiderMonkey.

And the lack of HTTP API means no direct interaction between
UIWebView and the database - and no CouchApp - which is a major loss.

It should work in-process if you point the WebView at a “touchdb:” URL, but I haven’t tried it. But for this to really be useful you’d also need to implement support for design documents, which again could be done as an add-on.

—Jens

[Jens Alfke]

On Dec 19, 2011, at 10:58 PM, Iain Stubbs wrote:

Not referencing couchembeddedserver at all.  But still in couchcocoa ... not sure where to go from here...

Ah, you’re probably using the -objc linker flag, which prevents classes (in this case CouchEmbeddedServer) from being dead-stripped.

… OK, I’ve just pushed a new revision to the touchdb branch of CouchCocoa that no longer puts CouchEmbeddedServer in the iOS framework. That should fix this.

I’ve also added some other conveniences for using TouchDB with CouchCocoa, but I’ll save that for a new thread.

—Jens