Deletion of watched node causes async exception (demo and sample patch included)

[Marc Munro]

Setting a children watch on a node results in an asynchronous failure if the node is deleted.  Since the exception is async, there is no simple way to trap this.

This is on version 2.0 and also in HEAD from github.
My environment is debian jessie: Linux bloodnok 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) x86_64 GNU/Linux
My python version is: 2.7.9

Here is a simple test case:
-------------------------------------------------
from kazoo.client import KazooClient

def callback(children):
    print "CALLBACK: %s" % str(children)

import logging
logging.basicConfig()

zk = KazooClient('localhost:2181')
zk.start()
zk.ensure_path("/testpath")
watcher = zk.ChildrenWatch("/testpath", callback)
zk.ensure_path("/testpath/wibble")
zk.delete("/testpath/wibble")
print "OK SO FAR"
zk.delete("/testpath")
-----------------------------------------------

Which generates:

marc:zkmonitor$ test/demo.py
CALLBACK: []
CALLBACK: [u'wibble']
CALLBACK: []
marc:zkmonitor$ test/demo.py
CALLBACK: []
CALLBACK: [u'wibble']
OK SO FAR
CALLBACK: []
ERROR:kazoo.handlers.threading:Exception in worker queue thread
Traceback (most recent call last):
  File "/home/marc/kazoo/kazoo/kazoo/handlers/threading.py", line 99, in _thread_worker
    func()
  File "/home/marc/kazoo/kazoo/kazoo/handlers/threading.py", line 196, in <lambda>
    self.callback_queue.put(lambda: callback.func(*callback.args))
  File "/home/marc/kazoo/kazoo/kazoo/recipe/watchers.py", line 341, in _watcher
    self._get_children(event)
  File "/home/marc/kazoo/kazoo/kazoo/recipe/watchers.py", line 36, in wrapper
    return func(*args, **kwargs)
  File "/home/marc/kazoo/kazoo/kazoo/recipe/watchers.py", line 319, in _get_children
    self._path, self._watcher)
  File "/home/marc/kazoo/kazoo/kazoo/client.py", line 272, in _retry
    return self._retry.copy()(*args, **kwargs)
  File "/home/marc/kazoo/kazoo/kazoo/retry.py", line 123, in __call__
    return func(*args, **kwargs)
  File "/home/marc/kazoo/kazoo/kazoo/client.py", line 1078, in get_children
    return self.get_children_async(path, watch, include_data).get()
  File "/home/marc/kazoo/kazoo/kazoo/handlers/utils.py", line 78, in get
    raise self._exception
NoNodeError: ((), {})
marc:zkmonitor$

The following patch seems to make things better:
---------------------------------------------
diff --git a/kazoo/recipe/watchers.py b/kazoo/recipe/watchers.py
index ad585da..60e9232 100644
--- a/kazoo/recipe/watchers.py
+++ b/kazoo/recipe/watchers.py
@@ -315,8 +315,15 @@ class ChildrenWatch(object):
             if self._stopped:
                 return
 
-            children = self._client.retry(self._client.get_children,
-                                          self._path, self._watcher)
+            try:
+                children = self._client.retry(self._client.get_children,
+                                              self._path, self._watcher)
+            except NoNodeError:
+                # Looks like watched node has been deleted, let's disable
+                # ourselves and pretend nothing happened
+                self._stopped = True
+                return
+           
             if not self._watch_established:
                 self._watch_established = True