Any examples

52 views
Skip to first unread message

Juju

unread,
Nov 22, 2011, 6:22:06 AM11/22/11
to PyF Framework Users
Dear list,

Has anyone ever used SQLAlchemy query as data producer ?
I'm looking for examples, can anybody share about it ?

Regards

Jonathan Schemoul

unread,
Nov 22, 2011, 6:34:10 AM11/22/11
to pyf-...@googlegroups.com
Are you meaning in a type code producer, or with a plugin ?

I've done some specific plugins already for some sqlalchemy models, so that can be done easily.
To prototype you can play from a code producer. Just yield your items directly from it.

To get objects you can inteact with without any call to the database you can either run them through a packet encoder (specifying the fields you want to keep), or use the SAObjectScrubber (it takes sqlalchemy objects and put them in an anonymous objects without relation to the database, for reporting for example).

Anyway, if you plan to keep your items, I recommend to preload any relations to avoid making subsequent requests to the DB. (eager_load SA option).

Jon

Juju

unread,
Nov 22, 2011, 9:24:24 AM11/22/11
to PyF Framework Users
C'est intéressant Jonathan. Tu pourrais poster des exemples de codes
correspondants ?
Mon cas précis est surtout de pouvoir traiter de gros jeu de données
sous forme d 'un flux, pour d'évidentes raisons d'économie mémoire.


On 22 nov, 12:34, Jonathan Schemoul <jonathan.schem...@gmail.com>
wrote:

Mathieu Bridon

unread,
Nov 22, 2011, 10:50:30 AM11/22/11
to pyf-...@googlegroups.com
On Tue, 2011-11-22 at 06:24 -0800, Juju wrote:
> C'est intéressant Jonathan. Tu pourrais poster des exemples de codes
> correspondants ?
> Mon cas précis est surtout de pouvoir traiter de gros jeu de données
> sous forme d 'un flux, pour d'évidentes raisons d'économie mémoire.

In a code producer:

----------
from myapp.model import session, MyModel

def produce():
query = session.query(MyModel)
for item in query:
yield item
----------

It really doesn't get any simpler than that, it's just a basic
SQLAlchemy request.

Items will be flowed to the next component in your tube.


--
Mathieu


Reply all
Reply to author
Forward
0 new messages