Reporting Handler Exceptions

[Michael Wilson]

Hi,

As I am getting closer to Alpha with our Tornado app, I'd like a way to catch all uncaught exceptions in request handlers and supply a handler to do the appropriate notifications (send email, ring alarms, etc).

Is there a good way to do this? By using a common base class for the handlers somehow?

Thanks!

[Didip Kerabat]

You can extend _handle_request_exception(self, e) in your base handler. Example:

    def _handle_request_exception(self, e):

        # Do Async HTTP to a different handler that emails/SMS/tweet you stacktrace.

- Didip -

[wataka]

Nice Didip, one question though, how does one route the output meant
for the commandline to the controller context?

[Michael Wilson]

Hi All,

Good suggestion.

What I ended up doing was writing my own tiny logging handler and adding
it to the root logger.

In the handler's 'handle' method, I look for the text 'Uncaught
exception'. If it's there, that log record has everything I need,
including the exception record.

From that I can compose an email/SMS/etc and send it out.

Of course, this is a little brittle since it depends on 'Uncaught
exception', but that's easily managed, I think.

Thanks for the suggestions!

[Phil Plante]

From my experiences the custom logging handler is the way to go.  Its the easiest method to plug into the existing logging throughout the framework and requires very little work if you are just wanting to send an email.  The python logging framework already comes with a few different handlers to get you started.

[Peter Bengtsson]

Silly question perhaps but if you plug into the logging will you be able to distinguish between a 4xx raise exception and a 5xx one?

[wataka]

Hey Michael, could you post a snippet of your logging handler? Thanks

[Michael Wilson]

Sure. This uses Amazon SES to do the job.

class LoggingHandler(Handler):
"""
This is our logging handler. It's whole purpose in life is to look
for severe
errors, and if it sees one, make the appropriate notifications.
"""

def __init__(self):
print "LoggingHandler::__init__"

def handle(self, record):
"""
"""
msg = None

if record != None:
msg = record.getMessage()

if msg.startswith('Uncaught exception') != True:
return

# Extract the interesting bits
(type, value, tb) = record.exc_info

amazonSes = AmazonSES(amazon_ses_AccessKeyID,
amazon_ses_SecretAccessKey)

message = EmailMessage()
message.subject = "Service Failure "
message.bodyText = "Service Failure detected ' + \
strftime("%Y-%m-%d %H:%M:%S", gmtime()) + ' GMT' + \
', ' + strftime("%Y-%m-%d %H:%M:%S", localtime()) + ' Local' + \
'\r\n\r\n'

message.bodyText += 'Message: ' + msg + '\r\n\r\n'

if tb != None:
message.bodyText += 'Exception: ' + '\r\n\r\n'

for tb_line in traceback.format_exception(type, value, tb):
message.bodyText += tb_line + '\r\n'

fromEmailAddress = 'no-r...@service.com'
toEmailAddress = 'vic...@service.com'

try:
result = amazonSes.sendEmail(str(fromEmailAddress),
str(toEmailAddress),
message, replyToAddresses=str(fromEmailAddress))
except AmazonError, e:
print "LoggingHandler::handle Uncaught Error sending email to " +
repr(toEmailAddress) + \
", errorType " + repr(e.errorType) + \
", code " + repr(e.code) + \
", message " + repr(e.message)
return
except:
print "LoggingHandler::handle Uncaught Error sending email to " +
repr(toEmailAddress) + \
", Exception " + repr(sys.exc_info()[0])
return

print "LoggingHandler::handle Result.requestId " +
repr(result.requestId) + \
", result.messageId " + repr(result.messageId)
print "LoggingHandler::handle Message: " + msg

[Ivan]

I like using hoptoadapp.com to manage logging exceptions. It adds
some nice structure around the exceptions, handles the emailing when
errors come up and other niceties.

I wrote a simple connector, which can be mixed in to your tornado
request handler to pass off all exceptions to the hoptoad service
asynchronously.

https://github.com/ephramzerb/tornadotoad

>      fromEmailAddress  = 'no-re...@service.com'