NotFound exception in delete() -- is beanstalk thread safe?

[Roy Smith]

We're using pybeanstalk 0.11.1, python 2.6.5, Ubuntu 10.04.3 LTS.  The main loop of our multi-threaded beanstalk client looks like:

    while True:

        m = mq.reserve()

        jid = m['jid']

        try:

            # process job                                                                                                                                                   

        except Exception:

            logger.error("skipped job %d" % jid)

        finally:

            logger.info("deleted job %d" % jid)

            mq.delete(jid)

Every once in a while (yesterday it happened once in 60k jobs processed), the delete() call in the finally block raises a NotFound exception.  I've included a stack dump below.

2012-04-02T04:56:57+00:00 cluster1.songza.com 2012-04-02 04:56:57,497 - scrobble_mill - Caught NotFound('Server returned: NOT_FOUND',) in <Thread(Thread-9, started 139969929115392)>

Traceback (most recent call last):

  File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 98, in listen_wrapper

    listen(host, port)

  File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 162, in listen

    mq.delete(jid)

  File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 94, in caller

    *getattr(protohandler, 'process_%s' % attr)(*args, **kw))

  File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 68, in _do_interaction

    return self._get_response(handler)

  File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 58, in _get_response

    res = handler(recv)

  File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 56, in __call__

    return self.__h(val)

  File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 68, in handler

    checkError(response)

  File "/usr/lib/pymodules/python2.6/beanstalk/errors.py", line 35, in checkError

    raise err

NotFound: Server returned: NOT_FOUND

Looking over the logs, it appears that two different threads got the same job in their reserve() calls.  Is this a bug?  I was assuming that beanstalk was thread safe, but maybe not?

[Roy Smith]

I think we've got this figured out.  It looks like what probably happened was the job timed out so it went back on the queue.

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/GMkJoHdsoOgJ.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

---

Roy Smith

r...@panix.com