Request finalizer runs *after* tweens finish

66 views
Skip to first unread message

Theron Luhn

unread,
Apr 28, 2017, 2:20:25 AM4/28/17
to pylons-discuss
I have a pretty standard setup of Pyramid and SQLAlchemy, based off http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/database/sqlalchemy.html

However, this has a problem:  The `cleanup` callback executes *after* the tween has returned.  I use tweens for both error logging (Rollbar) and request logging (output via python-fluent-loggerd).  This means that if the commit and associated flush fail, neither my request logger nor error logging catch it.

For example, if my tween looked like this:


class Tween:
    def __init__(self, handler, register):
        self.handler = handler

    def __call__(self, request):
        start = time()
        response = self.handler(request)
        duration = time() - start
        log(duration, response.status_code)  # This shows a success
        return response  # session.commit() fails sometime after this


The log call would report an HTTP 200, even the end result may be a HTTP 500 due to a failure in session.commt()

The obvious solution is to move logging to the WSGI middleware layer, but then I lose all the Pyramid-specific details, which I am loath to do.  I could also manually add a try/except to handle errors in the cleanup and report to Rollbar, but that still means my request logs show a success.

Is there a way I can force SQLAlchemy to commit/rollback (or at least flush) before the passing the baton back to the tweens?

Bert JW Regeer

unread,
Apr 28, 2017, 3:09:51 AM4/28/17
to pylons-...@googlegroups.com
Your Rollbar tween should live over the pyramid_tm tween. In the future it should probably also live above the pyramid_retry tween (that is if you want retry logic).

In that case if session.commit() fails your tween would correctly get back the exception view instead of the the non-exception view. 


--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/a98bf35f-d120-40b4-8168-fe4fc2f09f84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Bert JW Regeer

unread,
Apr 28, 2017, 3:12:49 AM4/28/17
to pylons-...@googlegroups.com
Sorry, I fired a little too soon…

You might want to look at the current state of the art in our sqlalchemy cookie cutter:


Request finalized callbacks are explicitly outside of tweens, and outside of any encapsulation.

You’ll want to look at pyramid_tm or build a tween yourself that handles things in a similar fashion.

On Apr 28, 2017, at 00:20, Theron Luhn <the...@luhn.com> wrote:

Theron Luhn

unread,
Apr 28, 2017, 2:25:42 PM4/28/17
to pylons-...@googlegroups.com
Thanks for pointing me to that SQLAlchemy cookie cutter, it was very helpful.  I was able to get something up and working with pyramid_tm and zope.sqlalchemy.

Do you know the motivation for pyramid_tm to be above EXC_VIEW by default?  I moved it below EXC_VIEW since I wanted commit errors to be handled by the exception views.  Is there any reason why I shouldn’t do this?

— Theron



You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/s5frQK7ES-s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylons-discus...@googlegroups.com.

To post to this group, send email to pylons-...@googlegroups.com.

Bert JW Regeer

unread,
Apr 28, 2017, 2:45:08 PM4/28/17
to pylons-...@googlegroups.com
Here is the motivation for that:


When Pyramid 1.9 drops, you’ll want to have the following:

rollbar -> pyramid_retry -> pyramid_tm -> EXC_VIEW -> your_app

Bert

Michael Merickel

unread,
Apr 28, 2017, 2:59:31 PM4/28/17
to Pylons
Sorry this isn't clear but pyramid_retry is not a tween.


— Theron



To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.

-- 
You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/s5frQK7ES-s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/E189DD60-2F5A-4D44-96AF-FB5A5D43670D%400x58.com.

Michael Merickel

unread,
Apr 28, 2017, 3:03:12 PM4/28/17
to Pylons
On Fri, Apr 28, 2017 at 1:25 PM, Theron Luhn <the...@luhn.com> wrote:
Do you know the motivation for pyramid_tm to be above EXC_VIEW by default?  I moved it below EXC_VIEW since I wanted commit errors to be handled by the exception views.  Is there any reason why I shouldn’t do this?

The goal is to allow exception views to use managed objects loaded by database queries (such as request.user). If pyramid_tm is under the excview then these objects are expired. It is possible to invoke exception views at any time, not just via the excview tween. You can add another tween over pyramid_tm that will use request.invoke_exception_view but alternatively in Pyramid 1.9 pyramid_retry will do this for you automatically -> meaning that if an exception hits the top of the stack, pyramid_retry will perform a last-ditch effort to render an exception response for you. It is true that in Pyramid 1.8 nothing, by default, will catch commit errors now that pyramid_tm is over the excview.

- Michael

Theron Luhn

unread,
Apr 29, 2017, 12:43:14 PM4/29/17
to pylons-...@googlegroups.com
Hmm, complicating things is that Rollbar doesn’t work when over EXCVIEW, since it tries to catch the exception which EXCVIEW swallows.

Might try to change that behavior and submit a PR, since it seems it’d be most desirable to have Rollbar as far up the stack as possible.

— Theron



Theron Luhn

unread,
Jun 5, 2017, 5:11:51 PM6/5/17
to pylons-...@googlegroups.com
Thank you all for your help.  Just a quick update on what happened:

I copied my implementation from SQLAlchemy cookie cutter.

In order to catch errors raised when committing a transactions, I modified Rollbar to work above EXC_VIEW and submitted a PR.  I haven’t received any response from the Rollbar team yet.  https://github.com/rollbar/pyrollbar/pull/162

— Theron



Reply all
Reply to author
Forward
0 new messages