TG2.1, WebError and streaming (using iterators as return values) problems

[kgk]

TG installs WebError to give the nice debugging 

tracebacks in the browser.

For default  projects, development.ini has 

set debug = True 

which gives the traceback in the browser. Recently we have defaulted this

to false which then sends errors as emails and is more secure for 

productions sites.  Turns out this changes the behavior of streaming

applications When debug is true, streaming apps

are actually fully collected into a list before being returned[1].

Now that it is false, it is actually returning an iterator[2].

Depending on debug either errormiddleware or evalexception is

placed on the WSGI stack[3].

We noticed that  exposed methods that return iterators 

now lost access to  several variables 

notably. tg.request and tg.response and all the thread-local variables 

usually there. 

During the TG application setup a Registry middleware component is added

that pushes on request entry and pops on response these thread-local  

variables.  This component has a streaming option (that defaults 

to false)[4][5].   Without the streaming option, the request

object is popped as soon as the method returns even though

the iterator is still returned up for processing. 

Sooo.. if your application needs to return iterators:

1.  set debug = true in site.cfg 

2. or patch [4] to read 

          app = RegistryManager(app, True)

Can't seem to find a ticket system to file this..

[1] WebError-0.10.3-py2.6.egg/weberror/evalexception.py:438

[2] WebError-0.10.3-py2.6.egg/weberror/errormiddleware.py:163

[3] TurboGears2-2.1.4-py2.6.egg/tg/error.py:89

[4] TurboGears2-2.1.4-py2.6.egg/tg/configuration.py:970

[5] Paste-1.7.5.1bisque1-py2.6.egg/paste/registry:367

[Alessandro Molina]

The WebError behavior is known, if I'm not wrong is already documented
somewhere in tg doc that you must disable debugging to make streaming
work.
Anyway, streaming doc has to improve, I'll try to write some more for 2.2 doc.

If you want to use the request inside a streamed response you can do
something like:

@expose()
def stream(self):
def output_pause(req):
num = 0
while 1:
num += 1
yield '%s/%s<br/>' % (req.path_info, num)
time.sleep(1)
return output_pause(request._current_obj())

> --
> You received this message because you are subscribed to the Google Groups
> "TurboGears" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/turbogears/-/VKSZodbWJkUJ.
> To post to this group, send email to turbo...@googlegroups.com.
> To unsubscribe from this group, send email to
> turbogears+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/turbogears?hl=en.

[kgk]

Actually my point was that if you patch 

   TurboGears2-2.1.4-py2.6.egg/tg/configuration.py:970

Changing      

    app = RegistryManager(app)

 to 

    app = RegistryManager(app, streaming = True)

Then WebError works fine.. it's a bug a turbogears but I could find a place to 

report this.

Sorry if the message was lost in all the description.

[Alessandro Molina]

Sorry, lost it.
You can report the bugs into https://sourceforge.net/p/turbogears2/tickets/

Anyway, even if you enable streaming in RegistryManager you won't be able
to stream the response as the debug middlewares and extensions try to read it.

You must set debug at false in development.ini to make streaming work reliably.

> --
> You received this message because you are subscribed to the Google Groups
> "TurboGears" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/turbogears/-/UkJdx45qXL8J.

[kgk]

You can report the bugs into https://sourceforge.net/p/turbogears2/tickets/

Thanks I have reported it as I see it is still an issues 2.1.5

 

Anyway, even if you enable streaming in RegistryManager you won't be able
to stream the response as the debug middlewares and extensions try to read it.

You must set debug at false in development.ini to make streaming work reliably.

Even if it debug=false, you cannot access tg.request and by extension anything that needs

identity (tg.request.identity['repoze.who.userid' ) by the time time the stream return processing

has started, the variables have been popped.

Simply changing the registry manager in tg.configuration  to use

      app = RegistryManager(app, streaming = True) 

solves the issue, but there is no easy way to override this section of the turbogears configuration. 

 

[Alessandro Molina]

On Thu, Apr 19, 2012 at 1:14 AM, kgk <kkvil...@gmail.com> wrote:
> Even if it debug=false, you cannot access tg.request and by extension
> anything that needs
> identity (tg.request.identity['repoze.who.userid' ) by the time time the
> stream return processing
> has started, the variables have been popped.

Well, that is the reason why I passed request._current_obj() as
argument to the generator in my code sample.
It solves the issue of being able to access the request and identity.

Other issues are related to the fact that you have to remove, flush
and commit the DBSession yourself.

[kgk]

Well, that is the reason why I passed request._current_obj() as
argument to the generator in my code sample.
It solves the issue of being able to access the request and identity.

Ahh.. I didn't even notice that reference.. Unfortunately in my case,

I am not referencing the tg.request directly in the generator, but in some

library routines that have no idea they are part of it so passing a direct

reference to the request object would not work without extensive rewriting

(if even possible)

 

Other issues are related to the fact that you have to remove, flush
and commit the DBSession yourself.

Good to know.. so I guess each layer in WSGI stack really needs to take special

care of generator returns when they have post-request processing and handle those cases.