Node.js bindings for couchbase-lite-core?

[Matias Piipari]

Is anybody in or outside Couchbase working on couchbase-lite-core bindings for Node.js? I'd be interested of starting such an effort / joining to contribute.

Matias

[Jens Alfke]

> On Oct 17, 2017, at 4:30 AM, Matias Piipari <matias....@gmail.com> wrote:
>
> Is anybody in or outside Couchbase working on couchbase-lite-core bindings for Node.js? I'd be interested of starting such an effort / joining to contribute.

I’m not aware of it, but that sounds like a cool project! What do you envision the JS API looking like?

(In the long run we do want to have a JS API binding for Couchbase Lite 2, but there hasn’t been any design work yet.)

—Jens

[Matias Piipari]

Perhaps I'll mull on it a bit and put together some ideas. My naive idea was to look at the Swift API for inspiration. I'm really at the stage of thinking through options of how to proceed with CBL2 overall – the driver for me to think about Node.js bindings is that if CBL2 will not have CouchDB replication support (nor a HTTP listener with CouchDB like REST API?) I have some extra work ahead ahead of porting my app Manuscripts to Windows and Linux (I'm doing it in Electron) in a way that still retains offline manuscript editing support. I could of course just use the CBL2 C# API (a decent Windows and Linux proof C# / Node.js bridge exists), but that stinks too.

Matias 

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/4B6RDHriNak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/EFCB7EE0-6B10-44BC-A96F-40AF12F86681%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

--

- - - - -
Matias Piipari, PhD

http://twitter.com/mz2

Manuscripts - A writing tool like nothing you have seen before

http://manuscriptsapp.com

[Matias Piipari]

Having had a look at the feature/2.0 branch of couchbase-lite-ios, the Swift / Objective-C implementations certainly look to be relatively thin wrappers for the C++ code, as you guys have advertised. Creating an equivalent higher level API in Node.js executable ES7 (or TypeScript) which behind the scenes calls to the same C++ APIs, using the public Swift API as a guide for API design should not be impossible. I'd probably use ES6 style destructuring with APIs to approximate Swift style readability, but I guess a key question is whether it's a sensible time to try such a thing (the README in the CBL repo states that the CBL API is in flux, in early stage of development etc). Should I wait?

Matias 

[Jens Alfke]

On Oct 17, 2017, at 4:13 PM, Matias Piipari <matias....@gmail.com> wrote:

Creating an equivalent higher level API in Node.js executable ES7 (or TypeScript) which behind the scenes calls to the same C++ APIs, using the public Swift API as a guide for API design should not be impossible.

I agree. I’m not sure how the actual inter-language glue would work; I have some experience with binding V8 to C++ but that was back in 2009. Hopefully it can be automated to the extent of exposing JS functions corresponding to the C API.

I guess a key question is whether it's a sensible time to try such a thing (the README in the CBL repo states that the CBL API is in flux, in early stage of development etc). Should I wait?

The CBL 2 API has mostly settled down by now. There are some changes to the Document API we’re discussing internally, to make thread-safety easier to achieve and to simplify naming, but they shouldn’t have a big impact on a JS binding.

—Jens

[Matias Piipari]

On Wed, Oct 18, 2017 at 6:10 AM, Jens Alfke <je...@couchbase.com> wrote:

On Oct 17, 2017, at 4:13 PM, Matias Piipari <matias....@gmail.com> wrote:

Creating an equivalent higher level API in Node.js executable ES7 (or TypeScript) which behind the scenes calls to the same C++ APIs, using the public Swift API as a guide for API design should not be impossible.

I agree. I’m not sure how the actual inter-language glue would work; I have some experience with binding V8 to C++ but that was back in 2009. Hopefully it can be automated to the extent of exposing JS functions corresponding to the C API.

The two options that strike as the best ones for me are:

1. Writing the JS bridge in C++ using the N-API (https://nodejs.org/api/n-api.html), i.e. to expose objects to the JavaScript environment that are backed by native code. This should work with the Chackra JS engine as well, not just V8 (there is of course also a C++ API still for V8 that would let one do the same thing).

2. Using the couchbase-lite-core C API and an FFI bridge. 

Option #2 likely yields a prototype easier and faster and #1 is a better long term solution.

What's the status of the C / C++ APIs, in that they are at least not noted to intended as the "public" API? Does that mean that they are to go substantial changes and some stabler / smaller subset is to be introduced still? Or have I just hit some out of date documentatioN/

 

I guess a key question is whether it's a sensible time to try such a thing (the README in the CBL repo states that the CBL API is in flux, in early stage of development etc). Should I wait?

The CBL 2 API has mostly settled down by now. There are some changes to the Document API we’re discussing internally, to make thread-safety easier to achieve and to simplify naming, but they shouldn’t have a big impact on a JS binding.

The precise way in which V8 handles thread isolation I bet will actually cause some pain. If the threading behaviour was like in CBL for iOS / macOS 1.x like I have potential solutions in mind. Could you explain more about the threading model, and what you describe as making threadsafety easier, in couchbase-lite-core?

 

—Jens

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/4B6RDHriNak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/D20031DF-FA96-4C19-BAA5-42E35F624CAB%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Oct 18, 2017, at 5:42 AM, Matias Piipari <matias....@gmail.com> wrote:

What's the status of the C / C++ APIs, in that they are at least not noted to intended as the "public" API? Does that mean that they are to go substantial changes and some stabler / smaller subset is to be introduced still? Or have I just hit some out of date documentatioN/

LiteCore’s C API is not considered public; it’s an internal API for use by implementations of Couchbase Lite, it can change without notice, etc. That being said, we try to limit incompatible changes because they require coordination between the three current CBL implementations. (I did just commit one yesterday, though.)

We would like to have a public C/C++ API for developers to use, which would be considered a new platform of Couchbase Lite, but that’s something for the future.

The precise way in which V8 handles thread isolation I bet will actually cause some pain. If the threading behaviour was like in CBL for iOS / macOS 1.x like I have potential solutions in mind. Could you explain more about the threading model, and what you describe as making threadsafety easier, in couchbase-lite-core?

I just finished writing a wiki document on LiteCore thread safety. In a nutshell: LiteCore doesn’t care what threads it’s called on, but generally doesn’t allow multiple concurrent calls involving the same database (or objects derived from the database, like documents or queries.) The document spells out the concurrency limitations of every API call.

—Jens

[Matias Piipari]

On Wed, Oct 18, 2017 at 4:57 PM, Jens Alfke <je...@couchbase.com> wrote:

On Oct 18, 2017, at 5:42 AM, Matias Piipari <matias....@gmail.com> wrote:

What's the status of the C / C++ APIs, in that they are at least not noted to intended as the "public" API? Does that mean that they are to go substantial changes and some stabler / smaller subset is to be introduced still? Or have I just hit some out of date documentatioN/

LiteCore’s C API is not considered public; it’s an internal API for use by implementations of Couchbase Lite, it can change without notice, etc. That being said, we try to limit incompatible changes because they require coordination between the three current CBL implementations. (I did just commit one yesterday, though.)

This sounds good. Do you follow some pattern for noting commits or release note entries where C API changes are introduced? 

I've started experimentally putting together a node-ffi based wrapper using the C API, thus far things are working (on macOS anyway – I've logged some issues regarding compilation on Linux presently) with a few API calls mapped as a test. I think I'll write the real thing in TypeScript so that I can use enums for the typedef equivalents and for the Swift / Objective-C API higher level type equivalents (many of the same of which make sense to provide).

 

We would like to have a public C/C++ API for developers to use, which would be considered a new platform of Couchbase Lite, but that’s something for the future.

Nice, and also understandable why that's for the future.

The precise way in which V8 handles thread isolation I bet will actually cause some pain. If the threading behaviour was like in CBL for iOS / macOS 1.x like I have potential solutions in mind. Could you explain more about the threading model, and what you describe as making threadsafety easier, in couchbase-lite-core?

I just finished writing a wiki document on LiteCore thread safety. In a nutshell: LiteCore doesn’t care what threads it’s called on, but generally doesn’t allow multiple concurrent calls involving the same database (or objects derived from the database, like documents or queries.) The document spells out the concurrency limitations of every API call.

Great, this may even work. :-)

 

—Jens

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/4B6RDHriNak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/ADA3DF60-C429-49A7-A489-CDF73CA8F601%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

[Erik Porsche]

Hi there,

I have a conceptual question regarding this topic. Instead of building a wrapper using node-ffi or edge-js (when building on top of couchbase-lite-net) - is it maybe better to wait for this issue to be solved: https://github.com/couchbase/couchbase-lite-core/issues/19 ?

Maybe in the future couchbase-lite-core will offer a websocket like the sync-gateway does with the extra benefit of a local no-sql database and a replicator/conflict resolver working in the background?

My dream machine is a cross desktop compiling app with online synchronization and conflict resolving functionality and I am not really sure how to do this today. Couchbase-lite-net can be build against Core 2.0 - therefore it is cross platform, but then I have the problem of building a proper UI - since XAML Standard is not ready yet for cross desktop compiling (and who knows if it ever will be). Node.js together with a electron container seems to be a feasible alternative. Fingers crossed that one day this scenario can be implemented.

Best regards 

Erik

Am Montag, 23. Oktober 2017 02:20:40 UTC+2 schrieb Matias Piipari:

On Wed, Oct 18, 2017 at 4:57 PM, Jens Alfke <je...@couchbase.com> wrote:

On Oct 18, 2017, at 5:42 AM, Matias Piipari <matias....@gmail.com> wrote:

What's the status of the C / C++ APIs, in that they are at least not noted to intended as the "public" API? Does that mean that they are to go substantial changes and some stabler / smaller subset is to be introduced still? Or have I just hit some out of date documentatioN/

LiteCore’s C API is not considered public; it’s an internal API for use by implementations of Couchbase Lite, it can change without notice, etc. That being said, we try to limit incompatible changes because they require coordination between the three current CBL implementations. (I did just commit one yesterday, though.)

This sounds good. Do you follow some pattern for noting commits or release note entries where C API changes are introduced? 

I've started experimentally putting together a node-ffi based wrapper using the C API, thus far things are working (on macOS anyway – I've logged some issues regarding compilation on Linux presently) with a few API calls mapped as a test. I think I'll write the real thing in TypeScript so that I can use enums for the typedef equivalents and for the Swift / Objective-C API higher level type equivalents (many of the same of which make sense to provide).

 

We would like to have a public C/C++ API for developers to use, which would be considered a new platform of Couchbase Lite, but that’s something for the future.

Nice, and also understandable why that's for the future.

The precise way in which V8 handles thread isolation I bet will actually cause some pain. If the threading behaviour was like in CBL for iOS / macOS 1.x like I have potential solutions in mind. Could you explain more about the threading model, and what you describe as making threadsafety easier, in couchbase-lite-core?

I just finished writing a wiki document on LiteCore thread safety. In a nutshell: LiteCore doesn’t care what threads it’s called on, but generally doesn’t allow multiple concurrent calls involving the same database (or objects derived from the database, like documents or queries.) The document spells out the concurrency limitations of every API call.

Great, this may even work. :-)

 

—Jens

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/4B6RDHriNak/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/ADA3DF60-C429-49A7-A489-CDF73CA8F601%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Oct 22, 2017, at 5:20 PM, Matias Piipari <matias....@gmail.com> wrote:

This sounds good. Do you follow some pattern for noting commits or release note entries where C API changes are introduced? 

I try to call this out in the commit message. So far it’s been informal because it’s just three co-workers using this API to implement CBL, but if there’s more development going on, we may want to establish clearer rules.

Also, you can do a git diff of C/include/ to focus on changes to the API headers.

—Jens

[Jens Alfke]

On Oct 24, 2017, at 3:58 AM, Erik Porsche <e.po...@gmail.com> wrote:

I have a conceptual question regarding this topic. Instead of building a wrapper using node-ffi or edge-js (when building on top of couchbase-lite-net) - is it maybe better to wait for this issue to be solved: https://github.com/couchbase/couchbase-lite-core/issues/19 ?

That would be another approach, but going through the REST API is (comparatively) slow. JS API calls have to be marshaled into HTTP requests, then unmarshaled into native API calls, then the same process in reverse for the response.

(TouchDB, the ancestor of Couchbase Lite, also worked this way. It got a lot faster when we refactored it to connect the front-end API directly to the engine without the HTTP layer.)

—Jens

[Matias Piipari]

Cool. I got somewhere with prototyping using what I am calling couchbase-lite-node without a better term (couchbase-lite-core APIs mapped in node, using fastcall (an ffi package alternate to node-ffi) as a basis. Quickly became obvious though that although that works, debugging anything with the bindings is painful using it (sigsegv as only error info gets old real fast) and it’s nice to also get compile time information of any further couchbase-lite-core C API changes too. So I’m starting over with n-api.

Matias

- - - - -
Matias Piipari, PhD
http://twitter.com/mz2

Manuscripts - A writing tool like nothing you have seen before
http://manuscriptsapp.com

[Matias Piipari]

For anyone interested, I've put my embryonic, experimental couchbase-lite-node project over to this public repository:

https://gitlab.com/mpapp-public/couchbase-lite-node

This has become a fun evening hacking project. I intend to wrap the couchbase-lite-core C API with a Node.js API that can act as a basis for a higher level Couchbase Lite Node.js API implementation.

Matias

[Matias Piipari]

I've continued to work on the couchbase-lite-core Node.js mappings and got my implementation to a stage where nearly all APIs in C4Database.h (open / close db, get / put raw documents etc) and the key stuff in C4Base.h (logging domains, levels) to help debugging are working and there's a test suite written with near full coverage of the bindings as they expand. Still C4Document  and its Fleece extensions to work through, and C4Query.h, before it makes sense to even think of using any of this for a couchbase-lite like higher level API on top of this. However, given the code already there, building these bindings is now mostly very straightforward, if tedious given the amount of C boilerplate involved.

Two questions I have come up with based on reading some of the source that I wanted to be certain about: 

1. Are there any asynchronous callbacks involved in the C API level (for example from database observers) which would fire (as a feature, not a bug) on a different thread than the thread on which the database API calls are otherwise executing? This would lead to some special considerations given the V8 model of thread isolation.

2. Is it correct that for any C4Slice / C4String / ... with malloced buf to it, the C4 APIs I passed them to do not take ownership (i.e. that I am responsible for freeing it at the end I created it)? I'm creating enough of slices that I'd like to now get their memory management right. Those C4 objects that C4 APIs themselves hand and which map to objects exposed to the JS context, like C4Database *, are already memory managed seemingly alright in my wrapper (N-API provides garbage collection finalization callbacks where I free up the native resources). 

Matias

[Jens Alfke]

On Nov 13, 2017, at 5:24 PM, Matias Piipari <matias....@gmail.com> wrote:

1. Are there any asynchronous callbacks involved in the C API level (for example from database observers) which would fire (as a feature, not a bug) on a different thread than the thread on which the database API calls are otherwise executing? This would lead to some special considerations given the V8 model of thread isolation.

Database/document observers can be called on another thread (specifically, on a thread on which a different C4Database is committing a transaction.)

Replication callbacks are called on background threads belonging to the replicator.

2. Is it correct that for any C4Slice / C4String / ... with malloced buf to it, the C4 APIs I passed them to do not take ownership (i.e. that I am responsible for freeing it at the end I created it)?

Yes. C4Slices are ephemeral. If you pass one to a function, LiteCore won't access that memory after the function returns.

—Jens

[Matias Piipari]

On Nov 13, 2017, at 5:24 PM, Matias Piipari <matias....@gmail.com> wrote:

1. Are there any asynchronous callbacks involved in the C API level (for example from database observers) which would fire (as a feature, not a bug) on a different thread than the thread on which the database API calls are otherwise executing? This would lead to some special considerations given the V8 model of thread isolation.

Database/document observers can be called on another thread (specifically, on a thread on which a different C4Database is committing a transaction.)

Replication callbacks are called on background threads belonging to the replicator.

From the looks of it, given no shared memory is involved (?) I need to write just a bit of boilerplate to copy the "message" from an observer / replicator callback that is internal to my API wrapper to a callback that the Node.js wrapper's client passed in, and executing that client provided callback on the Node / libuv main thread asynchronously with that copy of the message (argument values).

2. Is it correct that for any C4Slice / C4String / ... with malloced buf to it, the C4 APIs I passed them to do not take ownership (i.e. that I am responsible for freeing it at the end I created it)?

Yes. C4Slices are ephemeral. If you pass one to a function, LiteCore won't access that memory after the function returns.

Great, thanks. I have a fair amount of reviewing to do to fix up all the code created thus far so I don't leak, but at least it's 100% repetitive an issue.

 

—Jens

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/4B6RDHriNak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/F24D749D-F52E-4FF9-9A62-287786F32132%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

--