Relationship between requests and Zope transactions

[jens.t...@gmail.com]

Hi,

According to the documentation here, “Pyramid requests [to] join the active transaction as provided by the Python transaction package”. Looking at the transactions code, the default transaction manager used is a ThreadTransactionManager, i.e. one transaction per Python execution thread.

Now I did follow the Pyramid-SQLAlchemy-Cookiecutter recipe where DB sessions join the request’s transaction. Considering that a single incoming request is handled in Pyramid by a single thread (correct?) is it safe to say that the following is true?

import transaction

@view_config(…)
def some_view(request):
   # Request's transaction manager is the thread's transaction manager.
   request.tm == transaction.manager
   # Request's and thread's and tm's transaction is the same object.
   request.tm.get() == transaction.get()  # == transaction.manager.get()

Thanks!

Jens

[Michael Merickel]

Jens, by default your example is true, but it is not true in the cookiecutter configuration. The value of request.tm is defined by the tm.manager_hook setting and by default it is the threadlocal transaction.manager. The cookiecutter overrides the hook (and I suggest you do as well) to define a non-threadlocal manager configured in explicit=True mode which will help weed out bugs accessing transactions after they have been committed. I strongly suggest any code you write that needs the tm should use request.tm, not transaction.manager - as the former is configurable.

- Michael

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/79da9d5a-91e9-483e-b96f-b5e54cc3f1a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jens.t...@gmail.com]

Thanks Michael!

You are talking about these two lines of code in the cookiecutter’s model/__init__.py, correct?

settings = config.get_settings()

settings['tm.manager_hook'] = 'pyramid_tm.explicit_manager'

That manager hook allocates its own explicit transaction manager (see code) instead of using the transaction.manager default, as you’ve mentioned.

Using that manager hook I can then derive from an explicit manager and extend with custom functionality—and that would be the recommended way of doing so? That may help me solve a question I just asked at the transaction Github repo (see issue #62 there).

[jens.t...@gmail.com]

Maybe some background… the reason why I asked the initial question is because a few (and a growing number of) request handlers are complex; that is, they call multiple function levels deep.

Now request.dbsession, for example, is SQLA’s current DB session tied to the request’s transaction. To get access to it, functions keep handing that `dbsession` reference around. As we now begin to tie more data managers to the request’s transaction, handing their context around convolutes the parameters handed to functions.

So I thought it would be useful if much/all of a request’s execution context would be available through the current thread local memory. However, reading Why You Shouldn’t Abuse Thread Local (see the pyramid.threadlocal module) recommends against such an approach. Are there more recent and different recommendations?

[Michael Merickel]

You are talking about these two lines of code in the cookiecutter’s model/__init__.py, correct?

Right.

Using that manager hook I can then derive from an explicit manager and extend with custom functionality—and that would be the recommended way of doing so? That may help me solve a question I just asked at the transaction Github repo (see issue #62 there).

Yes, if you don't have a consistent location in your code where you can push/pop threadlocals then you're going to have a bad time. For something like the `tm` you'd still be wise to use your own threadlocal manager that is set to explicit mode versus using the transaction.manager shipped with the transaction project. You'd probably make a manager and register an ISynchronizer that pushed/popped the threadlocal so that it's only available when a transaction is open.

Would it make sense to use a custom request factory to implement such a thread local approach? Are there other recommendations, considering the warnings in pyramid.threadlocal?

So I thought it would be useful if much/all of a request’s execution context would be available through the current thread local memory. However, reading Why You Shouldn’t Abuse Thread Local (see the pyramid.threadlocal module) recommends against such an approach. Are there more recent and different recommendations? 

Sure you could do this with a custom request factory. The recommendations are not "out of date", they are about general programming paradigms that you should avoid if you want to write testable/reusable code. Pyramid tries to avoid *forcing* you to use threadlocals anywhere - but it's not going to try to stop you if you want to do it.

--

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5b86bb49-8fb5-4ebe-ad2b-c3a4da52bad3%40googlegroups.com.

[Mike Orr]

On Sun, Sep 2, 2018 at 7:31 AM <jens.t...@gmail.com> wrote:
> If requests during their life cycle would derive from threading.local and initialize such a storage area upon construction (e.g. after tween ingress) and tear it down (e.g. after tween egress) then deeply nested function could still access a request’s context without the need to pass that context around in parameters.

This is thinking about it the wrong way around. The best solution is
to add a 'request' argument to those functions, or the specific
request subobject they need. Sometimes you can't because you have to
pass through a function that wasn't written for Pyramid, but that's
what the threadlocal functions are for.

Pylons had a magic request object and a magic "template variables"
object that the subfunctions could import; they were threadlocal proxy
stacks behind the scenes. That was to replicate the popular behavior
in Myghty (definitely) and Rails (I assume). But they caused a lot of
problems and were more trouble than they were worth. So it was
considered an improvement when we migrated to Pyramid which passes an
explicit request object instead.

Why would you want to go back to threadlocals? Even if it sometimes
uses a threadlocal function to find the Zope registry; I consider that
a flaw rather than a feature.Semantically the subobjects are
REQUEST-local, not THREAD-local, so why not treat them that way?

[Jonathan Vanasco]

+1 on being explicit by passing around the request (or something similar). there are way too many edge cases when you try to compute the current request/transaction, and it's incredibly hard (a nightmare) to write tests that work correctly when you rely on the derived/computed methods.

[jens.t...@gmail.com]

Thank you Mike for the information, that all makes sense. I ended up with the same approach that SQLA takes: a “Session” object owned by a DataManager, joined to the Request’s TransactionManager. And then just pass a `dbsession` and `fnsession` and `jobsession` around to functions as needed.

Added yet another one or two parameters (sigh) to functions, but it keeps things functional and without outside (global, thread-local) state.

[Mike Orr]

functools.partal might help in some cases.

On Fri, Sep 7, 2018 at 1:44 AM <jens.t...@gmail.com> wrote:
>
> Thank you Mike for the information, that all makes sense. I ended up with the same approach that SQLA takes: a “Session” object owned by a DataManager, joined to the Request’s TransactionManager. And then just pass a `dbsession` and `fnsession` and `jobsession` around to functions as needed.
>
> Added yet another one or two parameters (sigh) to functions, but it keeps things functional and without outside (global, thread-local) state.
>

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
> To post to this group, send email to pylons-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/148c7bec-2d75-4525-9aa4-3902a4a2f2cb%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
Mike Orr <slugg...@gmail.com>

[Michael Merickel]

Added yet another one or two parameters (sigh) to functions, but it keeps things functional and without outside (global, thread-local) state.

 

This is where something like pyramid_services [1] really excels. It'll drastically cut down on concerns like that when you adopt some form of inversion-of-control pattern in your codebase.

[1] https://github.com/mmerickel/pyramid_services

On Fri, Sep 7, 2018 at 3:44 AM <jens.t...@gmail.com> wrote:

Thank you Mike for the information, that all makes sense. I ended up with the same approach that SQLA takes: a “Session” object owned by a DataManager, joined to the Request’s TransactionManager. And then just pass a `dbsession` and `fnsession` and `jobsession` around to functions as needed.

Added yet another one or two parameters (sigh) to functions, but it keeps things functional and without outside (global, thread-local) state.

--

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/148c7bec-2d75-4525-9aa4-3902a4a2f2cb%40googlegroups.com.