Python SDK plans

[Volker Mische]

Hi all,

as some of you might know, I'll push the Python SDK to the next level
the next few weeks. What this means is outlined below.

I'm really in favour of an open process and I appreciate any
comments/suggestions. Especially I'd like to hear about what the higher
level API should look like.

Looking forward to fruitful discussions here on the mailing list or on
IRC in #libcouchbase (I'm vmx there).

Cheers,
Volker


Plans for the Python SDK
========================

The following steps are in chronological order, but they might
overlap. No estimates are given as it doesn't contain any optional
things, but only the ones that need to be done. Hence it's done when
it's done.

Initially the Python SDK will use the synchronuous mode of
libcouchbase, the asynchronuous mode can be added later on.


Playing with technology
-----------------------

Decide which technology to use. So far it looks like we go for
[libcouchbase][1] wrapped with [Cython][2]. The advantage of
libcouchbase is performance and with Cython we should get it
working on all supported platforms (Linux, OSX, Windows). It
potentially even works with [PyPy][3].


Designing the API
-----------------

Whether the API will be like the Ruby one [5], fully compatible to the
old Python one, both or completely different needs to be decided.

The design should be open and very community driven. Community in this
case means people outside Couchbase, but also developers/evangelists
inside Couchbase. Just everyone who knows Python and likes to have a
nice API for it.


Build a protoype
----------------

Build a prototype that does gets and sets. Try it on all supported
platforms. It doesn't need to have a decent build system yet, but the
process of getting it running should be documented and reproducable.

For Cython you need special header files. I've spent a significant
amount of having this conversion automated. The work is a fork of
cwrap which can be found on [my personal Github repo][4].

So far it looks good and compiles without errors. Though there's still
some minor cleanup needed. I really want to have it outpue a file that
can be used without any changes.


Wrap the low-level functionality
--------------------------------

Wrap the libcouchbase functionality at a very low level. A nicer
Python API can then be built on top of that.


Sane build system
-----------------

Take the time to make the build easy on all supported platforms.


[1]: https://github.com/couchbase/libcouchbase
[2]: http://cython.org/
[3]: http://pypy.org/
[4]: https://github.com/vmx/cwrap/tree/vmx
[5]: http://rubydoc.info/gems/couchbase/

[Michael Salmon]

Well as a new user this past week to couchbase, I can say that the python module is definitely either lacking features or poorly undocumented. So I commend you on these efforts to revamp it! I'd really like to track this progress since our new mobile app backend db client be built with python.  I'd like to offer help with writing new docs and examples once a new API is sketched out.

-Michael

--
You received this message because you are subscribed to the Google Groups "Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yves Dorfsman]

On 2013-03-20 12:59, Volker Mische wrote:
> Hi all,
>
> as some of you might know, I'll push the Python SDK to the next level
> the next few weeks. What this means is outlined below.

Are you talking about "couchbase-python-client"?

If so, are you planning on making the current one evolved, or starting from
scratch?

I have started some work to make it work in both python 2 and python 3, but
haven't submitted anything for review yet (setting up a test environment for
python 2 was less trivial than I thought!). Whatever is done to the python
client, I think efforts should be made into having clients for both python 2
and 3 (one unique client if it is simple enough that we can make it work in
both environments, two clients, possibly using a translation script, if it is
more complicated).


--
Yves. http://www.SollerS.ca/
Unix/Linux and Python specialist in Calgary.
http://blog.zioup.org/

[Volker Mische]

On 03/20/2013 10:17 PM, Yves Dorfsman wrote:
> On 2013-03-20 12:59, Volker Mische wrote:
>> Hi all,
>>
>> as some of you might know, I'll push the Python SDK to the next level
>> the next few weeks. What this means is outlined below.
>
>
> Are you talking about "couchbase-python-client"?
>
> If so, are you planning on making the current one evolved, or starting
> from scratch?
>
> I have started some work to make it work in both python 2 and python 3,
> but haven't submitted anything for review yet (setting up a test
> environment for python 2 was less trivial than I thought!). Whatever is
> done to the python client, I think efforts should be made into having
> clients for both python 2 and 3 (one unique client if it is simple
> enough that we can make it work in both environments, two clients,
> possibly using a translation script, if it is more complicated).

Yes, I definitely want Python 3 support. I need to look into it, what
the best approach is these days.

Cheers,
Volker

[Volker Mische]

Hi Michael,

which features are you missing?

It's great to get support for the docs :)

Cheers,
Volker

> <mailto:couchbase%2Bunsu...@googlegroups.com>.

[Michael Salmon]

Well for example this page mentions a bucket.view(..) method - http://www.couchbase.com/develop/python/next

And the API doc, doesn't  - http://www.couchbase.com/docs/couchbase-sdk-python-1.0/couchbase-sdk-python-summary.html 

Makes me think one of the docs is old or not complete.  Am I missing something obvious?

[Volker Mische]

Good call. I think this part is from the old SDK for Couchbase 1.8 where
there were no MapReduce Views. Because the View page mentions the
method:
http://www.couchbase.com/docs/couchbase-sdk-python-1.0/workingwithviews.html

Cheers,
Volker

> > <mailto:volker...@gmail.com <mailto:volker...@gmail.com>>>

> > <mailto:couchbase%2Bunsu...@googlegroups.com
> <mailto:couchbase%252Buns...@googlegroups.com>>.

[Michael Salmon]

Well at least I'm not crazy then. That's good. The docs are organized in a spider-web between 1.8 and 2.0, so it's frustrating. Another confusing thing is the versions and doc names.couchbase-sdk-python-1.0, Couchbase 1.8, Couchbase 2.0. Would be good to keep the SDK Version # in sync with the version of the DB they are tested against. e.g. "SDK 2.0 supports Couchbase 2.0. SDK 1.x supports 1.8, and hence doesn't support <link to what's new in 2.0...>"

[Matt Ingenthron]

Hi Michael,

Thanks for joining the hangout by the way.  :)

On 3/20/13 3:43 PM, "Michael Salmon" <mic...@hahla.com> wrote:

Well at least I'm not crazy then. That's good. The docs are organized in a spider-web between 1.8 and 2.0, so it's frustrating. Another confusing thing is the versions and doc names.couchbase-sdk-python-1.0, Couchbase 1.8, Couchbase 2.0. Would be good to keep the SDK Version # in sync with the version of the DB they are tested against. e.g. "SDK 2.0 supports Couchbase 2.0. SDK 1.x supports 1.8, and hence doesn't support <link to what's new in 2.0...>"

The version numbers diverging is somewhat intentional, as each has it's own release cycle.  We do address this both at couchbase.com/develop and in the individual SDK documentation.  Examples:

http://www.couchbase.com/develop

http://www.couchbase.com/docs/couchbase-sdk-ruby-1.2/index.html

http://www.couchbase.com/docs/couchbase-sdk-ruby-1.0/index.html

For example, we had a reason to update .NET with some new features and that called for a new .minor.  Thus the 1.0 of .NET became 1.1.  The new features for 2.0 server were compatible with the 1.1 features, thus it became version 1.2.

And… we do actually cross test because we want  you to be able to use the latest client library with an older cluster and vice versa.  Bug fixes, for example, we try to get only into the latest release unless there's some reason to backport.

Similarly, we'll probably ship a 2.1 of Couchbase Server someday, and there may not be a reason to have a new Python client depending on the features.

We follow the recommendations at semver.org.  One omission on this is that I started python at version 0.5 instead of 0.1.  That's my fault.

Agreed that the python docs are a bit twisted about-- we'll look to fix that very soon too.

I hope that clarifies-- I mean to write a blog on the versioning soon.

Thanks,

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Matt Ingenthron - Director, Developer Solutions

Couchbase, Inc.

[Michael Salmon]

I would also suggest on updates to Python API is to have it consistent with PHP or Ruby (I don't know how close/off they are exactly, but..) because I just noticed that php uses startKey/endKey for view params, and the client.view() uses start_key, end_key (at least that's what it looks like from in the src).

I also just noticed that all the requests go through REST, so yes making this tie direct to the c-api would be the right path. Ideally I could use it with twisted python event system to get max performance too.

[Matt Ingenthron]

On 3/20/13 6:26 PM, "Michael Salmon" <mic...@hahla.com> wrote:

I would also suggest on updates to Python API is to have it consistent with PHP or Ruby (I don't know how close/off they are exactly, but..) because I just noticed that php uses startKey/endKey for view params, and the client.view() uses start_key, end_key (at least that's what it looks like from in the src).

We try to follow what's idiomatic for the language, and here we'll probably follow the PEP8 recommendations (which prescribes underscores for many things), unless there's some reason not to.

I also just noticed that all the requests go through REST, so yes making this tie direct to the c-api would be the right path. Ideally I could use it with twisted python event system to get max performance too.

Not all requests go through REST.  Some are going through memcached binary protocol with some Couchbase extensions.  Our intent with the SDK's API is to abstract all of that away, though both protocols are exposed as public interfaces for those who need to be closer to the bits on the wire.

I'd expect as we pull this API together, it'll probably be pretty unified much like Ruby/Java and the like.

Thanks,

Matt

On Wed, Mar 20, 2013 at 3:54 PM, Matt Ingenthron <ma...@couchbase.com> wrote:

Hi Michael,

Thanks for joining the hangout by the way.  :)

On 3/20/13 3:43 PM, "Michael Salmon" <mic...@hahla.com> wrote:

Well at least I'm not crazy then. That's good. The docs are organized in a spider-web between 1.8 and 2.0, so it's frustrating. Another confusing thing is the versions and doc names.couchbase-sdk-python-1.0, Couchbase 1.8, Couchbase 2.0. Would be good to keep the SDK Version # in sync with the version of the DB they are tested against. e.g. "SDK 2.0 supports Couchbase 2.0. SDK 1.x supports 1.8, and hence doesn't support <link to what's new in 2.0...>"

The version numbers diverging is somewhat intentional, as each has it's own release cycle.  We do address this both at couchbase.com/develop and in the individual SDK documentation.  Examples:

http://www.couchbase.com/develop

http://www.couchbase.com/docs/couchbase-sdk-ruby-1.2/index.html

http://www.couchbase.com/docs/couchbase-sdk-ruby-1.0/index.html

For example, we had a reason to update .NET with some new features and that called for a new .minor.  Thus the 1.0 of .NET became 1.1.  The new features for 2.0 server were compatible with the 1.1 features, thus it became version 1.2.

And… we do actually cross test because we want  you to be able to use the latest client library with an older cluster and vice versa.  Bug fixes, for example, we try to get only into the latest release unless there's some reason to backport.

Similarly, we'll probably ship a 2.1 of Couchbase Server someday, and there may not be a reason to have a new Python client depending on the features.

We follow the recommendations at semver.org.  One omission on this is that I started python at version 0.5 instead of 0.1.  That's my fault.

Agreed that the python docs are a bit twisted about-- we'll look to fix that very soon too.

I hope that clarifies-- I mean to write a blog on the versioning soon.

Thanks,

Matt

-- 

Matt Ingenthron

Couchbase, Inc.

[fr...@tower3.io]

I am very happy to see that an overhaul of the couchbase python client is in the works.  I have been battling with it since last August.  This past week I finally decided to give up and either re-write it or try wrapping libcouchbase.  My first choice is to wrap libcouchbase.

I considered using Cython, but decided that building the pdx files was going to be difficult and hard to maintain.  I did not make a search for automated translation tools, so I'm intrigued to see that maybe they can be auto-generated.

Instead, a couple of days ago I started building a python extension module to wrap libcouchbase.  Extension module code is somewhat convoluted, and made more so by the need for callbacks, but the result is fairly straightforward.  I've run into a snag, which you can read about here, but I'm pretty confident I can build something that will suit my purposes in a few days. I do expect to use libcouchbase asynchronously, and am building the extension accordingly.  I don't currently plan to wrap the entire libcouchbase API.

I will try out your updated cwrap and give Cython another look.

Your plan looks like a good one to me.  I agree that it's a waste of time to start from the current couchbase-python-client.  Although it is potentially a pain in the neck for developers who are trying to make use of your new client while you develop it, I would hold off on making any firm commitments in the "Design the API" step until after your prototype has a few miles on it.

- Frank

[Michael Salmon]

Yeah thanks, the redalyzer was quite cool. I can't say the .Net plugin was too interesting, but for some people maybe. 

I'm curious if there are any survey's of what languages/sdks are most popular to interface with couchbase, especially in high performance / mobile/web) type apps. 

Thanks,

Michael

On Wed, Mar 20, 2013 at 3:54 PM, Matt Ingenthron <ma...@couchbase.com> wrote: