race condition mako template and scoped_session

[tuck]

hello,
we test meinheld a great wsgi server and try to stress a little bit our pyramid webapp.
We use pserve in production mode with cherrypy

bug meinheld blow us a big problem with the scoped_session. Some time
the scoped_session is closed (recycle ?) before the end of the template render.
I don't understand why meinheld is more sensitive than waitress.

Some templates loads lazy attributes and triggers queries. The usual exception
came from alchemy's objects that became detached from their scoped session before
the end of the render

The usage of render_to_response solve this, but miss the test.

this is the usual pyramid pattern
@view_config(route_name='my_ro

ute', renderer='/my_test.mako')
def some_view(request)
      return {'some_alchemy_proxy_objects':my_data_liste}

against old pylons way .
@view_config(route_name='my_route')
def some_view(request)
      return render_to_response('/my_test.mako', {'some_alchemy_proxy_objects':my_data_liste}, request=request)

Does anyone have and idea to keep scoped_session opened during template renderer ?

[Michael Merickel]

How are you managing the scoped session? As I'm sure you know, Pyramid does not deal with SQLAlchemy at all, so it should be clear to you where in your code the session is being closed. If you are using pyramid_tm with the ZopeTransactionExtension then sessions are closed upon a transaction.commit(), which normally only happens after all rendering is complete and the request is basically done. If you are calling commit manually, then this could be the case but it would be expected to break no matter what wsgi server you are using.

Anyway, we need to know how you are using SQLAlchemy in order to answer your question.

- Michael

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tuck]

thx michael, we are using pyramid_tm which use zope.sqlalchemy.

we decare scoped_session like this

my_session = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))

but ... we are dealing with lots of databases. For that we store the scoped_session inside a static dict
when pyramid start, we list all databases names and init all scoped session with get_session method.
We close the session the first time she's created where pyramid is launched.

sample of our code

MyDbManager(object)
     _dictToScopedSession = {}

    @classmethod
    def get_session(cls, db_name):
         if db_name and db_name not in _dictToScopedSession :
             try :
                local_sess = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
                configure_dbsession(local_sess, cls._settings, 'delivery.dbcnx.temps.', metadata=None, codmar=oldcodmar.upper())
                local_sess.close()
             except DatabaseError :
                log.error('sorry no database named %s'%db_name)
             else :
                cls._dictToScopedSession[db_name]=local_sess

         return cls._dictToScopedSession.get(db_name)

[Jonathan Vanasco]

the problems you're describing seem to be random , and not necessarily tied to render vs render_response.  

it looks like you're using sessions incorrectly , and probably seeing issues pop up because some pages are lazyloading attributes while others don't have an attribute to load.

scoped_session should be created and alive for the entirety of each request; it looks like your 'get_session' is immediately closing it.  it sounds like you're treating the scoped sessions like a "database connection string" or persistent connection ( which would often be one per request per app , unless you're connection pooling ).  scoped sessions are a unit of work for the database connection, and typically last for a single pageview on a web app.

i found the best way to deal with scoped sessions is to store them in a global request attribute, and then register a cleanup handler to close them out afterwards.

they have very little overhead on setting up ( if you have a lot and don't use them all ).  you can also just use lazy-loading / memoization techniques to create them only when needed.  

[tuck]

thank's jonathan for answer. We only close scoped session when pyramid webapp start.

After that, scoped session should be recycle as usual. Nothing different than pyramid's alchemy

scaffold. I don't see a difference between a scoped session coming from a static dict attached to 

a class and a scoped session attached to a module.

But this bug is very usual with meinheld. I put an issue to mopemope github. Meinheld seems

to use greenlet and scoped session. I'm still trying to build a sample webapp.