Django 3.0 -> 3.1 performance degradation

[Jonatan Heyman]

Hi!

I've recently upgraded an old Django app from Django 1.11 all the way up to Django 4.0. Even though the app is pretty large, the migration went smooth (or so I thought). However, when all tests passed and I deployed the app to production, I saw that the response time of the app took a significant hit (the response time for the 95th percentile went from about 300 ms to 900 ms, and the median from about 100 ms to 250 ms). I also saw that the server CPU load had increased significantly.

I wrote some load tests and started downgrading Django. Now I've narrowed down the problem to appear when I upgrade from Django 3.0.x to Django 3.1.x, with no other significant changes to my app code.

Is this something someone else has experienced, or does anyone know what might be going on?

Are there any known changes that might increase CPU usage? In that case, I could throw more hardware at the problem.

Any ideas, pointers or insights would be greatly appreciated!

Best,
Jonatan

[Antonis Christofides]

Hi! This is a very interesting problem, and I'm afraid I'm not qualified to answer it. What I'd like to point out is that, if I had done all the work that you seem to have done, I would be tempted to start bisecting in between versions 3 and 3.1 in order to find the exact commit that causes the problem.

(However, in order to avoid that, I might also take a second look at the release notes, in case there was some information in there. And I might also ask in the developers mailing list—many of those who might know the answer don't read django-users.)

It's possible it's a Django issue, but it's also possible it isn't. For example, it could be some error in your caches or templates or other configuration that happens to only manifest itself in Django>=3.1.

In any case, I'm eager to learn how this plays out.

Regards,

Antonis

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5f05c06c-bd22-417b-9688-20aca9632bcbn%40googlegroups.com.

[Jonatan Heyman]

Hi! And thanks for the reply!

I've read the release notes but couldn't see anything that popped out as the possible cause. But I'm going to give them another read. Currently, the app is running fine on 3.0, and I haven't started debugging what might be the cause. I started with a message to this list, hoping that this would sound familiar to someone else :).

Bisecting between 3.0 and 3.1 is a possibility, though it's time-consuming since I have to rebuild the Docker image, deploy it to the staging server and run the load test for a little while between each test.

I'll post an update when I make any progress!

Best,

Jonatan

[Jason]

also would be helpful if you could give some examples.  All that is stated is "there's a significant slowdown", but nothing where or if there's been any tracing in place.  

Is this happening with db queries?  

serializing? 

rendering? 

are you using old packages which might have open/closed issues about performance?