production to handle over 1000 request fail

[Cato Yeung]

Dear experts,

I am doing a loadtest for my pycassa. I found that it raise Exceptions if my application do over 1000 write at the same time.

My app:

import pycassa

from multiprocessing import Process

def singleton(cls):

    instances = {}

    def get_instance(*args, **kwargs):

        if cls not in instances:

            instances[cls] = cls(*args, **kwargs)

        return instances[cls]

    return get_instance

@singleton

class Connection(pycassa.ConnectionPool):

    def __init__(self, keyspace='InviteAppRevamp2', server_list=['192.168.1.41:9160'], pool_size=20):

        pycassa.ConnectionPool.__init__(self,

                                        keyspace,

                                        server_list=server_list,

                                        pool_size=pool_size)

def write():

    pool = Connection()

    cf = pycassa.ColumnFamily(pool, 'test2')

    attr = {('xyz', "abc", '123'): "colval"}

    cf.insert('haha', attr)

if __name__ == '__main__':

    for i in range(1, 10):

        p = Process(target=write, args=())

        p.start()

How can I solve the issue?

Cheers,

Cato

[Tyler Hobbs]

What kind of Exceptions?

Did you check your open file handle limit? (use ulimit -a)

--
Tyler Hobbs
DataStax

[Cato Yeung]

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 88, in run

    self._target(*self._args, **self._kwargs)

  File "cassandra.py", line 21, in write

    pool = Connection()

  File "cassandra.py", line 8, in get_instance

    instances[cls] = cls(*args, **kwargs)

  File "cassandra.py", line 18, in __init__

    pool_size=pool_size)

  File "/Library/Python/2.7/site-packages/pycassa/pool.py", line 359, in __init__

    self.fill()

  File "/Library/Python/2.7/site-packages/pycassa/pool.py", line 420, in fill

    conn = self._create_connection()

  File "/Library/Python/2.7/site-packages/pycassa/pool.py", line 408, in _create_connection

    (exc.__class__.__name__, exc))

AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: Could not connect to 192.168.1.41:9160

This error occurs.

And file limit is 1024

Cheers,

Cato

[Cato Yeung]

I found that the error can be solved by changing the pool size to 1.

However, another error occurs:

[Fri Sep 21 10:12:20 2012] [alert] (13)Permission denied: mod_wsgi (pid=12374): Couldn't bind unix domain socket '/private/var/run/wsgi.12374.0.1.sock'.

[Fri Sep 21 10:12:20 2012] [notice] Digest: generating secret for digest authentication ...

[Fri Sep 21 10:12:20 2012] [notice] Digest: done

[Fri Sep 21 10:12:20 2012] [notice] Apache/2.2.21 (Unix) PHP/5.3.6 mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 mod_wsgi/3.4 Python/2.7.1 configured -- resuming normal operations

INFO:pycassa.pool:Connection 4329685632 (192.168.1.41:9160) in pool 4495557264 failed: Could not connect to 192.168.1.41:9160

This is the error log of my apache

Cheers,

Cato

[Cato Yeung]

I have already fixed that problem too. Just follow the document here:

http://code.google.com/p/modwsgi/wiki/ConfigurationIssues

section 

Location Of UNIX Sockets

However, some of my http request will generate error, but no error is shown in apache error log.

Those http requests return something like this:

<urlopen error [Errno 54] Connection reset by peer>

<urlopen error [Errno 32] Broken pipe>

Cheers,

Cato

[Tyler Hobbs]

If your open file handle limit is 1024, that's probably the issue.  You need to increase that using ulimit, which is temporary, or by editing /etc/security/limits.conf, which requires logging out and logging back in to take effect, and may be located somewhere else on non-debian based distros.

There's an example of what to put in /etc/security/limits.conf here: http://www.datastax.com/docs/1.0/troubleshooting/index#java-reports-an-error-saying-there-are-too-many-open-files

--
Tyler Hobbs
DataStax