a friendly warning about upcoming changes to pyramid_tm

[Michael Merickel]

Hey folks, I want to warn the community about an upcoming change to pyramid_tm that has been a long time in the making. I'm excited about it - but I also don't want to blindside anyone with a large-ish backward incompatible change.

TLDR - You may want to pin 'pyramid_tm < 2.0' until you have a chance to understand the changes.

There have been 3 major issues with pyramid_tm since its inception:

1) pyramid_tm doesn't cover exception view code so anything you do in an exception view should not be touching transactional objects or performing create / update / delete (non-readonly) queries.

2) Retry support has never fully torn down the request so if you're storing any transactional data in a request property (like request.user) and you have tm.attempts > 1, you may be leaking objects across transactions leading to subtle issues.

3) There is lots of room for subtle bugs when you perform changes to a transaction outside of the scope of pyramid_tm (it only covers part of the request lifecycle).

The resolution of these issues has involved several steps including a new library, a new concept in pyramid and changes to the transaction library and pyramid_tm.

1) The pyramid_tm tween was moved in the tween hierarchy to be OVER the EXCVIEW tween meaning transactions are kept alive during exception views and aborted later.

2) The retry support in pyramid_tm was removed and a new library was created called pyramid_retry. This library only works with pyramid 1.9 which is unreleased. The new retry library can will create a new request object for every attempt, throwing away any transactional state stored on it, leading to cleaner retries.

3) The transaction package now supports explicit transaction managers which will error when they are used after a commit/abort. You should use one by defining a "tm.manager_hook" that returns "transaction.TransactionManager(explicit=True)".

I hope this was helpful. We are still reviewing the changes to pyramid_tm and 2.0 does not have a release date but let's say that I promise it won't happen before March 15. Once that goes live you won't have any retry support, if you're relying on it (but I doubt you are because of the issues above) until pyramid 1.9 which also does not have a release date. :-)

Thanks!

- Michael

[Mike Orr]

The biggest problem with pyramid_tm I've run into is wanting to
rollback or commit the accumulated changes within the view without
messing up the global transaction outside the view. Normally when you
commit SQLAlchemy starts a new transaction afterward, but with
pyramid_tm you have to use 'transaction.commit' and that messes up its
transaction for the rest of request processing. Likewise if you start
to do some things and then can't (because a non-transactional resource
like a file can't be written or deleted), you want to roll back what
you did but without messing up the entire request. Again you have to
use 'transaction.abort()' or 'transaction.doom()', but that ultimately
causes a 500 error which you don't want. (Then there's the case of
logging, but I use separate autocommit connections outside the
transaction for that so they succeed even if the request has an
exception.)

Do the 'pyramid_tm' revisions offer any help with these cases?

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discus...@googlegroups.com.
> To post to this group, send email to pylons-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwEngaXn-%3DQZe13oKgw81_XRUJn-AN_3S_NPOipNgxsG-w%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.



--
Mike Orr <slugg...@gmail.com>

[Michael Merickel]

On Fri, Mar 3, 2017 at 10:10 AM, Mike Orr <slugg...@gmail.com> wrote:

The biggest problem with pyramid_tm I've run into is wanting to
rollback or commit the accumulated changes within the view without
messing up the global transaction outside the view. Normally when you
commit SQLAlchemy starts a new transaction afterward, but with
pyramid_tm you have to use 'transaction.commit' and that messes up its
transaction for the rest of request processing.

Isn't this what savepoints are for?

def view(request):

    sp = request.tm.savepoint()

    try:

        # view things ...

        # finally flush to push changes to the db without committing... this will

        # raise any IntegrityError, etc

        request.dbsession.flush()

    except Exception:

        # go back to where we were before the view executed

        sp.rollback()

 

Do the 'pyramid_tm' revisions offer any help with these cases?

pyramid_tm's goal is to provide a path for "normal" requests. This means throughout the request you add data to the transaction created by pyramid_tm and it will commit it eventually provided no errors occur. You should try *really really hard* not to commit that transaction on your own because obviously when you do you're making a lot of assumptions about what code comes next.

Obviously a lot of people are confused on this point, and that's an issue. The most common cases are usually solved by some combination of savepoints and flushing the underlying database session to catch errors earlier. For example, SQLAlchemy buffers all of your edits until a flush, and if you don't do it manually then it waits until commit. You could catch them earlier if you did a request.dbsession.flush() in a try/except (probably around a savepoint).

As far as your logging comment, this has always been the case that you may have things you want to commit to the database outside of the "normal" request. For example, errors, activity / login times, etc. These don't belong on pyramid_tm because they aren't contingent on whether the request succeeds or not. This has been a large driver for why the alchemy scaffold / cookiecutter was rewritten to get rid of a global database session. You can now create as many sessions as you want, and you don't have to hook them all to pyramid_tm - you can manage them individually in their own lifecycles.

Hope this helps!

- Michael

[Mike Orr]

On Fri, Mar 3, 2017 at 8:34 AM, Michael Merickel <mmer...@gmail.com> wrote:
> On Fri, Mar 3, 2017 at 10:10 AM, Mike Orr <slugg...@gmail.com> wrote:
>>
>> The biggest problem with pyramid_tm I've run into is wanting to
>> rollback or commit the accumulated changes within the view without
>> messing up the global transaction outside the view. Normally when you
>> commit SQLAlchemy starts a new transaction afterward, but with
>> pyramid_tm you have to use 'transaction.commit' and that messes up its
>> transaction for the rest of request processing.
>
>
> Isn't this what savepoints are for?
>
> def view(request):
> sp = request.tm.savepoint()
> try:
> # view things ...
>
> # finally flush to push changes to the db without committing... this
> will
> # raise any IntegrityError, etc
> request.dbsession.flush()
> except Exception:
> # go back to where we were before the view executed
> sp.rollback()

Ah, I didn't realize you could do that.

> Obviously a lot of people are confused on this point, and that's an issue.
> The most common cases are usually solved by some combination of savepoints
> and flushing the underlying database session to catch errors earlier. For
> example, SQLAlchemy buffers all of your edits until a flush, and if you
> don't do it manually then it waits until commit. You could catch them
> earlier if you did a request.dbsession.flush() in a try/except (probably
> around a savepoint).

The issue is that you may set up 'pyramid_tm' because the normal case
applies 99% of the time and seemingly always, and then you already
have an application built and discover that one view needs something
more complex.

I'll try savepoints and see if that solves it. As for flush, there's a
problem in the opposite direction. If you don't flush then you can
expunge pending adds/deletes from the session and it's like they never
were there. But if you do flush (e.g., to get an autoincrement ID or
fill in server-side defaults), then you can't simply expunge them, you
have to roll back. And if you've written a library function that
flushes, you can't get around the flush without bypassing or rewriting
the function.

[Bert JW Regeer]

I don’t follow this at all… either way you don’t delete because the transaction is rolled back.

-- 
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurQY%2BS2Pmccq2uQzT5gED%3D0NARLoQxmTYMpNKVSY5umSA%40mail.gmail.com.

[Mike Orr]

I don't want the global transaction to be rolled back because that
would cause a 500 error for the user. I want the request to succeed; I
just want to roll back my little work and give the user some kind of
status message on the page.

[Bert JW Regeer]

The transaction is rolled back when you raise from your standard view code, that exception that is raised will then be handled by the exception view tween, which will attempt to render the exception much like a normal view.

For example, given this view:

@view_config(renderer='../templates/index.mako',

             accept='text/html')

def create_new_user(request):

    user = create_user(request.POST[‘username’])

    return {‘msg’: ’Success, user created'}

and this as an exception view:

@view_config(

    context=sqlalchemy.exc,

    renderer=‘../templates/error.mako’,

    exception_only=True,

)

def reterror_sql(context, request):

    return {‘error’: ‘Temporary issue with our backend, please try again.’}

    

If the create_user call fails with a sqlalchemy error (which will cause the transaction to be aborted), then the exception view can still render the error page and the message as appropriate. However, an exception view is not allowed to raise yet another exception, that will cause that exception to be uncaught and it will be returned to the WSGI server, which will turn the response into a 500 error.

This allows you to then deal with the exception raised from sqlalchemy in a sane manner. If you are doing things within sqlalchemy outside of the view, those would also be rolled back, but that is the whole point of a transaction, it makes sure that when the request comes in, it doesn’t leave the state hanging in an unknown state.

If you absolutely have something that should always be committed after a request (success or not), then you should probably not be using the transaction manager.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duqv8dkUc-QpW0MkDSf5xAVjTrzUgPKQKFtMQS3H90FFdw%40mail.gmail.com.

[Jonathan Vanasco]

On Friday, March 3, 2017 at 2:30:09 AM UTC-5, Michael Merickel wrote:

TLDR - You may want to pin 'pyramid_tm < 2.0' until you have a chance to understand the changes.

two comments:

1. have you considered marking `pyramid_tm` as end-of-life, and pushing for the new changes as `pyramid_transaction` or some other uniquely different name?

2. it might make sense to have deployment advice recommend people install `pyramid_tm` etc as <2.0 and then <3.0 (or some other number) for future compatibility issues.

I've been bitten by large API changes like this before.  Not from Pyramid, but other projects where some 3rd party library updates another 3rd party library to an incompatible version.  It's a giant pain.  

[Mikko Ohtamaa]

two comments:

1. have you considered marking `pyramid_tm` as end-of-life, and pushing for the new changes as `pyramid_transaction` or some other uniquely different name?

Whilst this could make your life easier, it would make the life of maintainers a nightmare.

There are well known Python community best practices regarding version numbering and how library authors and application authors interact. One of them is using a pip freeze and requirements.txt to pin down your application library versions and not to rely on the future unreleased versions. Changing the library major version number is the socially accepted signal that things will break. Changing the package name would just clutter PyPi with pyramid_tm, pyramid_transaction, pyramid_yet_another transaction.

More information about maintaining known good set of libraries for your application

* https://www.kennethreitz.org/essays/a-better-pip-workflow

* https://ziade.org/2013/04/13/declaring-dependencies-in-python/

* http://nvie.com/posts/pin-your-packages/

> I've been bitten by large API changes like this before.  Not from Pyramid, but other projects where some 3rd party library updates another 3rd party library to an incompatible version.  It's a giant pain. 

Please see above for the best practices to maintain your application dependencies.

Best regards,

-Mikko

 

2. it might make sense to have deployment advice recommend people install `pyramid_tm` etc as <2.0 and then <3.0 (or some other number) for future compatibility issues.

 

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/9d2dba10-f67e-4ca4-a89c-808969fa4202%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Michael Merickel]

On Mon, Mar 6, 2017 at 10:56 AM, Jonathan Vanasco <jona...@findmeon.com> wrote:

two comments:

1. have you considered marking `pyramid_tm` as end-of-life, and pushing for the new changes as `pyramid_transaction` or some other uniquely different name?

Mikko already commented on this but I'll reiterate that if we renamed the library every time we wanted to change part of the design the maintenance nightmare would be unsustainable. Version numbers are the best way we have to progress the library and communicate the changes to the users. The changes being made here are almost all fixing *broken* features in pyramid_tm and are in the spirit of the library. If we were changing the entire purpose of the library then a rename would be in order. The main incompatibility is the position of the tween in the order... it is now over exception views instead of under. That means if you have hacks in your app to do database things in your exception views... I'm warning you that may break. The retry stuff was fundamentally broken before in anything except the simplest apps.

 

2. it might make sense to have deployment advice recommend people install `pyramid_tm` etc as <2.0 and then <3.0 (or some other number) for future compatibility issues.

This is basic advice that applies to every library ever written. The only thing that matters is to understand the library's versioning scheme which, admittedly, we may not do a great job explaining for the Pylons projects. For most of our projects we will update the major version number for a breaking change... Pyramid core is slightly different but it also puts a much larger emphasis on deprecation periods and maintaining older versions. We could always talk about improving this process in the future, but this isn't the place to do it.

 

I've been bitten by large API changes like this before.  Not from Pyramid, but other projects where some 3rd party library updates another 3rd party library to an incompatible version.  It's a giant pain.  

Yes, it is a giant pain. That's why I wanted to warn people explicitly in this case because I don't want anyone suffering dataloss because their transaction manager changed.

[Jonathan Vanasco]

Yes, it is a giant pain. That's why I wanted to warn people explicitly in this case because I don't want anyone suffering dataloss because their transaction manager changed.

That's why I brought this up - this is a bit more of a 'dangerous' API change.

People following best-practices won't be affected, but people who don't follow them will be -- and they're likely to flood here+github+stack-overflow with bug reports.  (at least that's what I've seen in the past).

To minimize that, i've seen projects switch names or implement some sort of affirmative consent by purposefully breaking the API.  

What Mikko said is true when everybody does the right thing -- unfortunately most people don't.

[Bert JW Regeer]

Then we’ll deal with the posts and questions when they come up… it’s not that big of a problem.