pylons & SQLAlchemy & memory

[diana]

So, I've created a hello world pylons app as a little test case
because I'm running into some memory issues with another pylons app.

(hello world app details below)

When I first launch the app (on my dev box), the memory profile looks
like this:

VIRT=138m
RES=20m

When I do a select * from a table with 90845 rows [Session.query
(PreLog).all()], the memory goes to the following and then holds
_forever_.

VIRT=348m
RES=182m

I had suspected that eventually garbage collection would kick in and
release the SQL result set (or whatever SQL metadata it's holding on
to), but that doesn't seem to be the case.

Any ideas? Surely, I'm missing some config option or something. The
code that deviates from the default (paster create & paster
controller) is as follows.

Thanks for your time,

--diana

I basically:

$ paster create -t pylons logs
$ paster controller pre_logs

And then modified:

---- MySQL ----

CREATE TABLE `pre_log` (
`modified` timestamp NOT NULL default CURRENT_TIMESTAMP,
`id` int(11) NOT NULL auto_increment,
`batch_id` int(11) NOT NULL,
`action` varchar(100) default NULL,
`profile_id` varchar(25) default NULL,
`start_datetime` datetime default NULL,
`end_datetime` datetime default NULL,
`status` varchar(25) default NULL,
`message` varchar(1000) default NULL,
PRIMARY KEY (`id`),
KEY `batch_id` (`batch_id`)
) ENGINE=MyISAM AUTO_INCREMENT=351808 DEFAULT CHARSET=utf8

---- routing.py ----

# CUSTOM ROUTES HERE
map.connect('pre_logs', '/pre_logs', controller='pre_logs',
action='index')

---- model/__init__.py ----

def init_model(engine):
global reflected_table
reflected_table = sa.Table("pre_log", meta.metadata, autoload=True,
autoload_with=engine)
orm.mapper(PreLog, reflected_table)
meta.Session.configure(bind=engine)
meta.engine = engine

class PreLog(object):
pass

---- meta.py ----

__all__ = ['Session', 'engine', 'metadata']
engine = None
Session = scoped_session(sessionmaker())
metadata = MetaData()

---- controllers/pre_logs.py ----

class PreLogsController(BaseController):

def index(self):
pre_log_query = Session.query(PreLog)
result = pre_log_query.all()
count = len(result)
return 'Hello World %s' % (count)

---- ini ----

[DEFAULT]
debug = false

[app:main]
sqlalchemy.url = mysql://foo:bar@localhost:3306/logs_db?charset=utf8
set debug = false

---- OUTPUT ----

Hello World 90845

[Waldecir Santos]

Maybe sqlalchemy cache ? btw this is not a wrong behavior, check the time of the second query, if it is faster than the first query, then it is the cache.

regards,

Waldecir

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
>
>

[Alexandre Conrad]

2010/1/11 Waldecir Santos <wald...@gmail.com>:

> Maybe sqlalchemy cache ? btw this is not a wrong behavior, check the time of the second query, if it is faster than  the first query, then it is the cache.

I don't think SQLAlchemy has caching. Maybe you should make sure your
session has been closed after the request returned. Otherwise all your
objects might keep living in the UOW of SA.

in lib/base.py make sure you have something like:

def __call__(self, environ, start_response):
try:
return WSGIController.__call__(self, environ, start_response)
finally:
meta.Session.remove()

(or "meta.Session.close()")

[Diana Clarke]

Yup, I've got meta.Session.remove() in the finally clause of the BaseController.

I going to try removing beaker from the middleware next, in case
beaker is the one doing the caching.

And then maybe try playing with Dozer again. If I figure out what's
going on, I'll post my findings.

Thanks for your time, Alexandre.

Have a good week,

--diana

[Waldecir Santos]

Nice to know, i never loked to my mem usage, maybe is a good thing open a bug in Beaker, btw let us know any thing new.

thnxk

Waldecir