How to fetch 10000 above record within second in django?

[Prashanth Patelc]

Hi all,

How to fetch the data from db (mysql) above 10000 records within second, 

I written query it's taking more time like 38 seconds, how to to fetch the lac of records within seconds any example in django using orm or MySQL query ?

[Clive Bruton]

On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:

> I written query it's taking more time like 38 seconds, how to to
> fetch the lac of records within seconds any example in django using
> orm or MySQL query ?

Are you saying that you have a raw SQL query, or you are using a
Django query?

My experience is that the biggest lag in database retrieval is the
amount of data retrieved, so if you can reduce the number of fields
returned, that would be a good start.

If you are using a Django query, and it has related elements, then
the biggest boost you can give is using "select_related" and
"prefetch_related".


-- Clive

[Prashanth Patelc]

I'm using orm query fetching all data and showing into html templates , it's taking more time for get the data , 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518%40indx.co.uk.

[Clive Bruton]

The easiest way to debug this is to install django_debug_toolbar and
look at the sql queries to see where the hold-up is.

As previously, restrict the number of fields returned using "values"
and/or use "select_related" or "prefetch_related" to get related data
in one or two queries, rather than having Django getting that data on
(potentially) multiple queries per record.


-- Clive

> d/msgid/django-users/CAMCU6CrAKQL6qA6hWHPDFbfpCT1R9v5%2BY_Qi-HOkPf%
> 3DOXYt2yA%40mail.gmail.com.

[Ayanfeoluwa Koyinsola]

You might now have to display all the data at once, see if you can use paginisation to reduce the numbers of queries

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/52330EF7-BBD9-44E6-94E3-C0A733F4EF36%40indx.co.uk.

[Prashanth Patelc]

I'm already using pagination but when I hiting particular url it is taking more than 1minute for loading the data and displaying in html .

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK-XCvfn84Xy%3D8kRpt%2BfJ_DmwUMxkeZ9iTnaGuUV5M3SNN-Twg%40mail.gmail.com.

[Clive Bruton]

On 17 Jan 2023, at 12:06, Ayanfeoluwa Koyinsola wrote:

> You might now have to display all the data at once, see if you can
> use paginisation to reduce the numbers of queries

I think this will only help if there are related lookups.


-- Clive

[Jason]

https://dev.mysql.com/doc/refman/8.0/en/using-explain.html is a good guide to showing how to use EXPLAIN to see what your query is running.

https://use-the-index-luke.com/ is a great introductory source for indexes

https://django-debug-toolbar.readthedocs.io/en/latest/ added to your project can show where your bottlenecks are.