Profiling Django (WAS Django database-api)

[Andre Terra]

While I know of the two methods mentioned by Anssi, I've often wondered how to profile my code from a project level.

I have some complex and database intensive asynchronous tasks running under celery which take a LONG time to complete and I'd just love to be able to keep track of the queries they generate in order to optimize and possibly remove the biggest bottlenecks.

"Real time" updates would be great, but I can settle for after-the-fact logs. I know of django-debug-toolbar [0] but since these aren't happening in a view, I'm not sure that app can help.

Any suggestions? Thanks in advance!

Cheers,
André Terra

[0] http://pypi.python.org/pypi/django-debug-toolbar

On Apr 3, 2012 6:13 AM, "KasunLak" <kasun.la...@gmail.com> wrote:

[Javier Guerra Giraldez]

On Tue, Apr 3, 2012 at 8:21 AM, Andre Terra <andre...@gmail.com> wrote:
> I have some complex and database intensive asynchronous tasks running under
> celery which take a LONG time to complete and I'd just love to be able to
> keep track of the queries they generate in order to optimize and possibly
> remove the biggest bottlenecks.

the easiest would be to write detailed logs, which _can_ be analysed
in real time, not only 'after the fact'.

my second idea would be to hack the log output so instead of writing
to a file, it would store messages (probably with some structure) to
some comfortable database. I'd use Redis, but i guess MongoDB or even
an SQL-based DB could work too. then you can easily filter and
aggregate times according to task type, when it happened, etc.

--
Javier

[Andre Terra]

Hey Javier,

Thanks for the reply. My problem with the logs in the past was that they tended to make the task even slower (due to recursion) but I guess that's probably because I didn't call the logging from the appropriate places in the code.

To make things a little more complicated, the task involves writing a large amount of data to a temp database, handling it and then saving some resulting queries to the permanent DB. This makes it a tad harder to analyze what goes on in the first part of the code.

I'll try logging again over the weekend and see how that works.. I just wish there were third party apps and tools for debugging this sort of problem.

Thanks again for your input.

Cheers,
AT

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

[Javier Guerra Giraldez]

On Tue, Apr 3, 2012 at 5:25 PM, Andre Terra <andre...@gmail.com> wrote:
> To make things a little more complicated, the task involves writing a large
> amount of data to a temp database, handling it and then saving some
> resulting queries to the permanent DB. This makes it a tad harder to analyze
> what goes on in the first part of the code.

i haven't had that kind of problem, but these are the things i would try:

- time-limiting logs. if the last call was too recent, just discard
the message (for some log levels, those used in the inner loops)

- send the logfiles to fast devices (maybe even in ram, like tmpfs in
Linux) and aggressively rotate them (so they don't accumulate
needlessly)

- log to a fast database (like Redis) and do automated analysis every
hour or maybe even every minute, discarding the raw log entries.

- log to a listening process that checks if each (time limited) entry
is out of the ordinary. if so, keep it for analysis. if not, discard
it. 'ordinary' could mean if some indicator is growing or reducing as
expected, or just changed from the last, or stable, or whatever you
could expect from your intended calculations.

--
Javier

[Tom Evans]

On Tue, Apr 3, 2012 at 11:25 PM, Andre Terra <andre...@gmail.com> wrote:
> Hey Javier,
>
> Thanks for the reply. My problem with the logs in the past was that they
> tended to make the task even slower (due to recursion) but I guess that's
> probably because I didn't call the logging from the appropriate places in
> the code.
>
> To make things a little more complicated, the task involves writing a large
> amount of data to a temp database, handling it and then saving some
> resulting queries to the permanent DB. This makes it a tad harder to analyze
> what goes on in the first part of the code.
>
> I'll try logging again over the weekend and see how that works.. I just wish
> there were third party apps and tools for debugging this sort of problem.
>
> Thanks again for your input.
>
> Cheers,
> AT

Hi Andre

One pretty cool method I've used for live debugging in the past is to
'log' to rabbitmq (Note - not celery - raw amqp), and send messages to
a logging exchange. You can use a topic key so that different
processes can be distinguished.

Then, if you want to see the running logs of what is happening, you
simply attach a dumb listener that binds a queue to that exchange,
receives the message and prints it out. If you're no longer
interested, detach the listener, and rabbitmq will just discard the
messages with little overhead.

If you want to do stuff with AMQP, I'd recommend pika [1], which is
just brilliant and easy to understand (other amqp libraries, less
so!).

Cheers

Tom

[1] http://pika.github.com/

[Javier Guerra Giraldez]

On Wed, Apr 4, 2012 at 4:07 AM, Tom Evans <teva...@googlemail.com> wrote:
> One pretty cool method I've used for live debugging in the past is to
> 'log' to rabbitmq (Note - not celery - raw amqp), and send messages to
> a logging exchange. You can use a topic key so that different
> processes can be distinguished.

a similar thing can be done with ZeroMQ, with the advantage of not
needing a mq daemon. but i guess Andre already uses rabbitmq, so
yours would be the easiest way.

--
Javier