some errors with python client under concurrency

[Chen Sun]

I am a new guy to couchbase.I try to use couchbase and couchbase's python client in my python-base web APIs.But I find some errors when I use ab(a tool to test the web api performence) to test the api with some concurrecny http requests. The errors of the python clinet throwed bellow here.I want know why these errors occoured,and how to solve this problem.

The errors
---------------------------------------------------------------------------------------------------------------------------

uwsgi: src/server.c:666: lcb_server_purge_implicit_responses: Assertion `c->cmd_log.nbytes' failed.
python-couchbase: self->nremaining == 0 at src/oputil.c:77. AbortTraceback (most recent call last):
  File "api.py", line 111, in falcon.api.API.__call__ (falcon/api.c:1389)
    #                                             startkey=start, endkey=end,
  File "api.py", line 96, in falcon.api.API.__call__ (falcon/api.c:1210)
    #                 end = '/' + adid + '/' + end
  File "api.py", line 174, in on_post
    query_results = couchbase_category.get_multi(keys, quiet=True)
  File "/usr/local/lib/python2.7/dist-packages/couchbase/connection.py", line 1053, in get_multi
    return _Base.get_multi(self, keys, ttl=ttl, quiet=quiet, replica=replica, no_format=no_format)
couchbase.exceptions._TimeoutError_0x17 (generated, catch TimeoutError): <Key=u'com.htc.fm', RC=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], Operational Error, Results=34, C Source=(src/multiresult.c,286)>

------------------------------------------------------------------------------------------------------------------------------------

My python code

couchbase_category = Couchbase.connect(
            bucket='Category', host='127.0.0.1', timeout=5)
class Category:
 
    def on_post(self, req, resp):
 
        t1 = datetime.datetime.now()
        content = {}
        # try:
        params = json.load(req.stream, 'utf-8')
        keys = []
        for app in params['apps']:
            keys.append(app['bundle_id'])
 
        unknown_apps = []
        result = {}
        result['known_bundle_ids'] = []
        result['unknown_bundle_ids'] = []
        result['apps'] = []
        result['categorys'] = []
 
        query_results = couchbase_category.get_multi(keys, quiet=True)
        for key in query_results:
            if query_results[key].success:
                result['apps'].append(query_results[key].value)
                result['known_bundle_ids'].append(key)
                result['categorys'].append(
                    query_results[key].value['categoryName'])
            else:
                result['unknown_bundle_ids'].append(key)
        result['categorys'] = list(set(result['categorys']))
        for item in params['apps']:
            if item['bundle_id'] in result['unknown_bundle_ids']:
                unknown_apps.append(item)
        timestamp = time.time()
        pipe = r.pipeline()
        for app in unknown_apps:
            pipe.zadd('uncached_apps', timestamp, app)
        pipe.execute()
        content['is_ok'] = True
        content['result'] = result
        t2 = datetime.datetime.now()
        content['total_time'] = (t2 - t1).total_seconds()
        # except Exception, err:
     #   content['is_ok'] = False
        # finally:
        resp.status = falcon.HTTP_201
        resp.body = json.dumps(content)

[M. Nunberg]

Hi

Such errors would fall under the domain of the underlying libcouchbase.
What version of libcouchbase are you using? Version 2.3.1 and earlier
has a bug which may manifest itself in the symptom you are describing,
so please try 2.3.2 (http://www.couchbase.com/communities/c-client-library)

I would also suggest you try to run your tests with the new 2.4.0-beta
version to see how that goes. Get it here
http://packages.couchbase.com/clients/c/index.html

On 07/17/2014 04:23 AM, Chen Sun wrote:
> I am a new guy to couchbase.I try to use couchbase and couchbase's
> python client in my python-base web APIs.But I find some errors when I
> use ab(a tool to test the web api performence) to test the api with some
> concurrecny http requests. The errors of the python clinet throwed
> bellow here.I want know why these errors occoured,and how to solve this
> problem.
>
> The errors
> ---------------------------------------------------------------------------------------------------------------------------
>
> uwsgi: src/server.c:666: lcb_server_purge_implicit_responses: Assertion `c->cmd_log.nbytes' failed.
> python-couchbase: self->nremaining == 0 at src/oputil.c:77. AbortTraceback (most recent call last):
> File "api.py", line 111, in falcon.api.API.__call__ (falcon/api.c:1389)
> # startkey=start, endkey=end,
> File "api.py", line 96, in falcon.api.API.__call__ (falcon/api.c:1210)
> # end = '/' + adid + '/' + end
> File "api.py", line 174, in on_post
> query_results = couchbase_category.get_multi(keys, quiet=True)
> File "/usr/local/lib/python2.7/dist-packages/couchbase/connection.py", line 1053, in get_multi
> return _Base.get_multi(self, keys, ttl=ttl, quiet=quiet, replica=replica, no_format=no_format)
> couchbase.exceptions._TimeoutError_0x17 (generated, catch TimeoutError): <Key=u'com.htc.fm', RC=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], Operational Error, Results=34, C Source=(src/multiresult.c,286)>
>
> ------------------------------------------------------------------------------------------------------------------------------------
>
> My python code
>

> couchbase_category= Couchbase.connect(

> bucket='Category', host='127.0.0.1', timeout=5)
> class Category:
>
> def on_post(self, req, resp):
>
> t1= datetime.datetime.now()
> content= {}
> # try:
> params= json.load(req.stream, 'utf-8')
> keys= []

> for appin params['apps']:

> keys.append(app['bundle_id'])
>
> unknown_apps= []
> result= {}
> result['known_bundle_ids'] = []
> result['unknown_bundle_ids'] = []
> result['apps'] = []
> result['categorys'] = []
>
> query_results= couchbase_category.get_multi(keys, quiet=True)

> for keyin query_results:

> if query_results[key].success:
> result['apps'].append(query_results[key].value)
> result['known_bundle_ids'].append(key)
> result['categorys'].append(
> query_results[key].value['categoryName'])
> else:
> result['unknown_bundle_ids'].append(key)
> result['categorys'] = list(set(result['categorys']))

> for itemin params['apps']:

> if item['bundle_id'] in result['unknown_bundle_ids']:
> unknown_apps.append(item)
> timestamp= time.time()
> pipe= r.pipeline()

> for appin unknown_apps:

> pipe.zadd('uncached_apps', timestamp, app)
> pipe.execute()
> content['is_ok'] = True
> content['result'] = result

> t2= datetime.datetime.now()

> content['total_time'] = (t2 - t1).total_seconds()
> # except Exception, err:
> # content['is_ok'] = False
> # finally:
> resp.status = falcon.HTTP_201
> resp.body = json.dumps(content)
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Couchbase" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to couchbase+...@googlegroups.com
> <mailto:couchbase+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[孙如镇]

Hi Nunberg,

  

     Thanks for your replay. I will follow you advices and do some experiments.

Best regards.

an email to couchbase+unsubscribe@googlegroups.com
<mailto:couchbase+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/couchbase/mhiJsqPkk7g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to couchbase+unsubscribe@googlegroups.com.

[孙如镇]

Hi Nunberg,

     My libcouchbase-dev is 2.3.2. I have tried to use your libcouchbase-dev 2.4.0, but situation even be much worsen.A lot "timeout" were throw out,but no "src/server.c:666: lcb_server_purge_implicit_responses: Assertion `c->cmd_log.nbytes' failed." errors. Now I am sure that the error " src/server.c:666: lcb_server_purge_implicit_responses: Assertion `c->cmd_log.nbytes' failed." makes my python application container(uwsgi) down before. By the way some time another error "evmap.c:401: Assertion ctx failed in evmap_io_active", So really need your help.

Best regards.

[Mark Nunberg]

Just how many keys are you passing to get_multi/set_multi?

Mentioning uwsgi, what form of thread/process configuration are you using? Keep in mind that in the case of forking, you must ensure the Connection object is create in the child process, not the parent process.

To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.

[孙如镇]

33 keys were used by get_multi, my uwsgi command is "sudo uwsgi -s /tmp/uwsgi.sock  --wsgi-file api.py --callable app --master --process 4  -R 10000", I didn't use threads.

To unsubscribe from this group and all its topics, send an email to couchbase+...@googlegroups.com.

[Mark Nunberg]

Unfortunately I am not very well versed in uwsgi, however from from what I see, you are instantiating the Connection object in global scope. Presumably the Category.on_post method is called inside the workers which are already spawned once the application has been loaded (inheriting the Connection object). This may lead to unwanted behavior, ranging from data corruption to random crashes.

The crashes you are seeing seem to be coming from libevent which is obviously not happy about being forked (in the 2.4.0 client). The 2.3.x client crashes because it is receiving garbage data from the socket (since you have multiple processes reading from the same socket).

Please set LCB_LOGLEVEL=5 in the environment before spawning your program. This will show a lot of debug output about the client library (continue using 2.4 for this); the output will tell us if you are actually trying to use the same client from multiple processes.

If indeed this is the case you will need to figure out a way to ensure that the client object is never inherited from a parent to a child process. There is the Python `atfork` (https://pypi.python.org/pypi/atfork/0.1.2) module which you can use to help with this.

Regards,

Mark

[孙如镇]

I did set the LCB_LOGLEVEL to 5 and change the libcouchbase to 2.4.0 again, but unfortunately  uwsgi hide the debug output of the liccouchbase. And I got a lot "[err] evmap.c:401: Assertion ctx failed in evmap_io_active " this cycle. I wonder if there are some other methods to get the libcouchbase debug output just like a log file or others.

[Mark Nunberg]

Currently it only logs to standard error. I would guess it would log elsewhere though - perhaps a file or similar. Still the more I think about your issue the more I am inclined to assume that you are passing the client object to the child processes.

My advice would be to place the initialization (e.g. cb = Couchbase.connect()) inside the function that is first called when your worker process is invoked, so e.g:

import os

from warnings import warn

from couchbase import Couchbase

cbclients = {}

def get_cb_client(**cbkw):

    curpid = os.getpid()

    # You might want to modify this if dealing with multiple buckets

    if cbclients:

        # Purge the dictionary from older clients

        for pid in cbclients.keys():

            if pid != curpid:

                warn("Found leaked client object from PID {0}. We are {1}".format(pid, curpid))

                del cbclients[pid]

    return cbclients.setdefault(curpid, Couchbase.connect(**cbkw))

class Category:

    def on_post(self, req, resp):

        bucket = get_cb_client(bucket='Category', host='127.0.0.1', timeout=5)

        # Rest of the code goes here..

Note that I haven't tested this example, nor would I recommend it as a replacement for proper design of applications that are to be forked, but it should at least help debug/fix some of the issues you are having.

[孙如镇]

I solved my problem by using a uwsgi param (--lazy) to set the uwsgi loading code after forking new worker-processes , that reason of the problem is that uwsgi loads python code first ,and than fork new work-processes .

[孙如镇]

Thanks for your help again. Your are  so conscientious in your work which makes me a deep  impression. Thank you again. Wish you a happy work.