multiple requests within a pyramid commandline script

[Jonathan Vanasco]

I have a convenience class for my commandline scripts:

    from pyramid.paster import bootstrap

    class IntegratedApi():

        env = None

        bootstrapped = None

            

        def __init__(self):

            self.env = bootstrap('../../../web-pylons/app/production.ini')

    

        def finish(self):

            self.env['closer']()

is there a better way to generate new 'requests' under the commandline environment ?

each call to `bootstrap()` seems to generate an overhead of about 1.3s on my dev machine ( .9 on production ).   

i need to somehow make new 'requests' more on par with the normal overhead ( which is so small its virtually non-existant ).    

[Laurence Rowe]

My case is slightly different, but should be applicable for a command line app too. When running in development mode I create some test data at application startup time as a convenience. I wrap the app with WebTest, the equivalent for you would be something like:

    app = pyramid.paster.get_app('path/to/config.ini')

    from webtest import TestApp

    environ = {

        'HTTP_ACCEPT': 'application/json',

        'REMOTE_USER': 'TEST',

    }

    testapp = TestApp(app, environ)

(I have remote user authentication policy configured as part of pramid_multiauth for this purpose.)

You can then make requests like:

    item = {'foo': 'bar'}

    testapp.post_json(collection_url, item, status=201)

Laurence

[Jonathan Vanasco]

I need an opinion on the 'safety' of a potential approach...

Calls to `pyramid.paster.bootstrap` are definitely what is taking too long ( 2.5s on dev ! )

Looking at the path of execution  and relevant bits...

`pyramid.paster` `bootstrap`

    calls `paste.deploy` `get_app` ( which looks to be expensive )

    calls `pyramid.scripting` `prepare` ( which may or may not be expensive )

`pyramid.scripting` `prepare`

    does a lot of stuff with threadlocals / etc.  and that scares me.

my idea is to pull the app out of the env, then create a new env - injecting that app into it. 

the gist is below...

https://gist.github.com/jvanasco/5783417

i'm wondering if this is safe, or if much of it is needed :

- do i need to update the 'app' ?  is that used on anything ?

- should i pull the `registry` out of the env and pass that into `prepare_request` too ?

any ideas ?

[Michael Merickel]

I'm not entirely sure about what you're trying to do, but you can dissect bootstrap pretty easily.

`get_app` takes a while because it parses the ini file and loads your app. Just like pserve, you only have to do this once for the process.

After that, call `prepare` once per thread, or as many times as you want. If you're calling it a lot you'll want to invoke the closer to cleanup each time or all of those requests will hang around.

Anyway what I've said looks somewhat similar to your gist.

The registry argument to `prepare` is important if your INI file has multiple pyramid apps. For example, you're using a [composite:main] to dispatch between different apps, and you invoke `get_app('my.ini')` without specifying which pyramid app to load. It'll load all of them. At which point which one does `prepare` actually use?? It uses the last pyramid app loaded unless you specify which registry you want to use. This works most of the time because few people seems to use the composite applications in the same process. A workaround is to invoke `get_app('my.ini#myapp')` to only load the specific app you want (which could also reduce your startup time).

HTH,

Michael

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

Visit this group at http://groups.google.com/group/pylons-discuss.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Vanasco]

thanks a ton!  this helps a lot. it looks like i was on the right path!

i built some maintenance and test scripts using the pyramid commandline docs.  they're really great, but -- following the docs -- it's that call to boostrap() that takes a huge performance toll.  being able to avoid that with `prepare` is perfect ( and I already wrapped close() )

as for an example of what i'm trying to do :

we needed a script to send periodic customized email messages. 

instead of re-inventing the wheel, we wanted to use all the stuff we already built off of Pyramid:

- per-request caching and optimization of database calls

- use customization

- email transaction logging ( who what when where how why )

- email gateway support

because we have per-request objects and tracking , we needed to ensure a new , clean , request.  so we had multiple bootstrap() calls.  the overhead of bootstrap, drops our output by quite a bit -- 1200 emails/hr/server.  We're small now , so this isn't a problem.  We're growing, so it's a concern. Only calling bootstrap once will probably push us well above 12k emails/hr/server