Expected performance of the django development server?

[Richard Brockie]

Hello again,

I'm to the point in my django development that I am beginning to use realistic amounts of test data. I'm using postgresql as the database server, with PyCharm as my IDE, everything in a virtualenv on an SSD on an Ivy Bridge Core i7 processor with 16 GB of RAM running Windows 7 64-bit.

One disappointing thing I have just encountered is the lack of speed when working with my real-world data. I'm comparing this with the same data in a non-framework-based php/MySQL webapp running on XAMPP (which means apache) in parallel on the same system.

With DjDT installed (Django development tools), I'm getting the following stats:

• Quick to load page:SQL: 5 queries in 4 ms, Time: 76 ms
• Slow to load page: SQL: 1189 queries in 463 ms, Time: 9754 ms
  

From task manager, the slow to load page is making "python.exe *32" use 1 whole logical processor during the page load.

Are these expected response times? I'm hoping this is the result of a non-optimized development server, and not the expected performance of Django itself.

Any comments or advice?

Thanks,
R.

[Andrew Farrell]

The django development server is slow by intentional neglect; It isn't supposed to be used in production because the django team does not want to divide its focus by supporting a full-fledged web server that is performance-optimized and security-audited. They want to include a development server that can be used by someone who wants to go from 0-to-understanding-and-building with django without first having to install a production server like gunicorn.

A few useful tools for improving your performance are select-related, prefetch-related, and Django-debug-toolbar.

They can help you reduce the number of queries you are making.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9c8f3393-78b1-4d9f-a78d-954b957ebdfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Farrell]

To clarify: I'm not a django core dev and by "neglect" I don't mean that anyone is being neglectful, just focusing efforts elsewhere.

[Andrew Farrell]

Another tool that is less robust but easier than django-debug-toolbar to simply drop into your code is

import django.db.connection as conn

followed by putting

print(conn.queries)

or

print(len(conn.queries))

before and after where you suspect large numbers of repeated queries are being executed.

(See also: http://stackoverflow.com/questions/2133627/using-django-db-connection-queries)

[Jani Tiainen]

On Sun, 4 Jan 2015 20:41:58 -0800 (PST)
Richard Brockie <richard...@gmail.com> wrote:

> Hello again,
>
> I'm to the point in my django development that I am beginning to use
> realistic amounts of test data. I'm using postgresql as the database
> server, with PyCharm as my IDE, everything in a virtualenv on an SSD on an
> Ivy Bridge Core i7 processor with 16 GB of RAM running Windows 7 64-bit.
>
> One disappointing thing I have just encountered is the lack of speed when
> working with my real-world data. I'm comparing this with the same data in a
> non-framework-based php/MySQL webapp running on XAMPP (which means apache)
> in parallel on the same system.
>
> With DjDT installed (Django development tools), I'm getting the following
> stats:
>

> - Quick to load page:SQL: 5 queries in 4 ms, Time: 76 ms
> - Slow to load page: SQL: 1189 queries in 463 ms, Time: 9754 ms

>
> From task manager, the slow to load page is making "python.exe *32" use 1
> whole logical processor during the page load.
>
> Are these expected response times? I'm hoping this is the result of a
> non-optimized development server, and not the expected performance of
> Django itself.
>
> Any comments or advice?

By judging amount of queries of your "slow" page you probably have model(s) with foreign keys that you lazy load - which means that each fk is loaded individually from the database with separate query.

For example if you have a model that contains 4 foreign keys and you query 100 instances (rows) you would actually invoke 1 + 4 * n queries, which in example case would be 401.

Without actually knowning your code one of the first options you usually do to optimize queries is to use select_related and prefetch_related queryset methods to cut down number of queries needed.

--

Jani Tiainen

[Vijay Khemlani]

1189 queries is quite a large amount, are all of those really needed? I think some profiling is in order.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20150105080927.4cc08eac%40jns42-l.w2k.keypro.fi.

[Richard Brockie]

Hi Jani,

That's a correct diagnosis. I knew Django was doing lazy loading, but didn't fully understand the implications.

In my case I have a filtered and ordered queryset resulting in ~400 records. Upon processing in the template, each was prompting a database visit 3 times when outputting information: hence the ~1200 trips to the database.

I've added prefetch_related specifying the related models I'm working with in the template.

My 'slow' case has now become: 46 queries in 39 ms, Time: 506 ms: a factor of 20 speed improvement.

Thanks for the suggestion!
R.

[Collin Anderson]

Hi Richard,

Like you've seen, the _number_ of queries often has a large effect on speed.

Also, try comparing your page load time with and without debug toolbar enabled. I found that DDT is slow itself :).

Collin