Need to Store All Model data in RAM

[Vaibhav Mishra]

Hi All,

I have a table of over 100,000 rows which I need to compare with another list to find if the entry matching a rows is in there or not 

The following does not work

getdata = data.objects.all()


mydata= getdata.filter(mycriteria)

Process information and comparision from 'mydata'

.The code .should fetch all data and should no longer go to Database. , but it still seem to hit database

.

Whenever I am running code, I see that for each comparision , the Code is Hitting the database resulting in over thousands of queries and lot of delay in finishing Processing data. 

How can I force All Data to be stored in RAM so that its no longer needed to access database. ?

[Andréas Kühne]

Hi,

You can't. The way that you are working - it would always go back to the database - because you are using the filter methods.

You would need to create a list from the items, then do the comparison in python - which probably will be slower (but not necessarily).

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a120e643-936e-4f6a-9662-7811808fa8cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chetan Ganji]

Hi Vaibhav, 

Andréas is right. You need to evaluate the queryset first as the orm queries are lazy. 

So the solution would be something like below example. 

another_list = []

getdata = data.objects.all() # this is a queryset

# apply filters if you need to

getdata = list(getdata) # this would be a list in python

for item in getdata: # won't hit database again

if item in another_list:

# do something

pass

Regards,

Chetan Ganji

+91-900-483-4183

ganji....@gmail.com

http://ryucoder.in

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCevkAVGFf%3DMrS8VVsv2BgPuDkRUqW_tSFwk1dfrzb7pvg%40mail.gmail.com.

[Vaibhav Mishra]

Hi Chetan,

Thanks for your help, I have a confusion , the code uses another_list as a list on line 1 ,and is being queried in the loop below , but where did it get populated ? Sorry if I sound stupid.  

[Vaibhav Mishra]

Thanks Andreas, Will try out the list Method.

[Chetan Ganji]

Bhai, read your original question 

"I have a table of over 100,000 rows which I need to compare with another list to find if the entry matching a rows is in there or not "

Which another list were you talking about?

so the another_list could be a pre-populated list of python or another list made of queryset. 

e.g. 

another_list = [1, 2, 3, 4] OR

another_list = list(Modelname.objects.all()) 

I hope this clears your confusion. 

Regards,

Chetan Ganji

+91-900-483-4183

ganji....@gmail.com

http://ryucoder.in

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5b0499d9-073f-4d3e-8f38-4160e4b50f65%40googlegroups.com.

[Vaibhav Mishra]

Hi Chetan,

Thanks ,Ok , Now I get it  :)   Will try this and share my experience.