Urgent request: help with Python/GAE endpoint performance issue

[Dave Furfero]

One of our endpoints is running slowly due to its query. We would ideally like to have it running in time for a Wednesday press release.

Volunteers with Python and GAE experience, please take a look at the related GitHub issue.

Contact me by email or on Slack with any questions or suggestions. 

Thanks!

furf

[Robert Jensen]

Hi Dave,

Haven't had time to play with this, but at a glance I would say L767 is the problem:
https://github.com/Lessig2016/pledgeservice/blob/master/backend/handlers.py#L767

The call to `model.User.all()` should probably take place outside the loop so that it only hits the db once.

Best,

Rob

--
You received this message because you are subscribed to the Google Groups "teamlessigtech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to teamlessigtec...@googlegroups.com.
To post to this group, send email to teamles...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/teamlessigtech/0b5b629c-46d3-40e2-b8de-c960d1a534f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Furfero]

I agree. It is inefficient (n+1) to call the query inside of the loop. It is also inefficient to load the entire data set each time when only one record is needed. 

It would be more efficient to make one query for all pledges and another for all users and zip the queries together. Or, just make a single joined query.

[Dominic Tancredi]

Also, why calculate every time? Just a suggestion:

How about calculate once, store total in a field, then update the total on pledge creation? 1 write vs 1k reads that require (n+1) calculations. 

--
You received this message because you are subscribed to the Google Groups "teamlessigtech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to teamlessigtec...@googlegroups.com.
To post to this group, send email to teamles...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/teamlessigtech/6fb7b4b7-b8ca-4e12-a6bd-464786ef7bbd%40googlegroups.com.

[Barry Rafkind]

Here's a pull request that implements Rob's suggestion.

To view this discussion on the web visit https://groups.google.com/d/msgid/teamlessigtech/CAL5_rPw%3D3AB8POxEDK14B-haa-2eSh%3D72LsnXgU6V5-JqbUxSw%40mail.gmail.com.

[Max Metral]

It's probably fine for now, but I agree that this sort of denormalization/caching is crucial for the types of spikes you can see in campaigns. Might be worth starting by trying to maintain the total, and then having a switch to use the cache or not just in case you need it in a pinch.