Help with a huge database query.

29 views
Skip to first unread message

odrzen

unread,
May 27, 2021, 6:10:58 PM5/27/21
to django...@googlegroups.com
Hello Django community o/

I want to create a API call in order to get all data from a specific model ( table ) - but I want to exclude-remove only specific fields from this.
So, I try to find a right way to execute this difficult and "heavy" query/request with a efficient way.
This is what I have achieved so far:

```
#from django.http import HttpResponse, HttpResponseRedirect, JsonResponse
#from django.core import serializers
from apples.models import Apple
from oranges.models import Orange

published_fruits =  Apple.objects.select_related('Orange').filter(state='published').defer(
                               # Fields I don't want:
                               'field1',
                               'field2',
                               'field3'
                               )

So, I have the following results:
[
  {
    "model": "apples.apple",
    "pk": "5326t236-8415-48f4-89e5-1789vc9of442",
    "fields": {
      "id": "apple-type1",
      "name": "Brazil",
      ".....": "....",
      "exports ": []
    }
  },
  { ... }
  {
    "model": "apples.apple",
    "pk": "6435673472-fret2-523t-523t-d41159t23432213",
    "fields": {
      "id": "apple-type2",
      "name": "India",
            ".....": "....",
      "exports ": []
    }
  }
]
```


My proble is :
1. The fields who I don't want (exclude), finally I got it.
2. I want to directly pass all this response to a JSON http output for the users and I have it with the following way:
```
json_response = serializers.serialize('json', published_fruits)
return HttpResponse(json_response, content_type='application/json')
```
I get all the respone in JSON, but ( as you can see above ) with the following stracture:
```
  {
    "model": "apples.apple",
    "pk": "6435673472-fret2-523t-523t-d41159t23432213",
    "fields": {
      "id": "apple-type2",
      "name": "India",
            ".....": "....",
      "exports ": []
    }
  }
```

Please, could you help me to avoid the following part from each record  :
```
    "model": "apples.apple",
    "pk": "6435673472-fret2-523t-523t-d41159t23432213",
    "fields": {
```
from the JSON ?

Thank you very much in advance!

David Nugent

unread,
May 27, 2021, 7:14:52 PM5/27/21
to django...@googlegroups.com
I think a better approach to using QuerySet.defer() is to use .values() instead, explicitly specifying which fields you need to export.

Just FYI rest_framework handles this out of the box and provides an API to specify not only which fields to export but filters as well.

Regards
/d


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/MvvGE0gJ6x5v1Geo7zgnHdB8TLr79HKJwubqhQzrOKUSB3Q4oHPeWCctdFmOAJ74PbqQ9gHkg44hKVrPqp9BFiZ7Bn7E-LzM30aNrbhcf7s%3D%40protonmail.com.

odrzen

unread,
May 31, 2021, 5:04:45 PM5/31/21
to django...@googlegroups.com
If I use the  values()  or  values_list()  the request in database has a long waiting time and some time I also got a  MemoryError (because 4GB RAM is not enough).
My examples:

Apple.objects.select_related('Orange').filter(state='fresh').values_list(
                               # Fields I don't want:
                               'field1',
                               'field2',
                               'field3'
                               # ..... ,
                               'field37'
                               )

( yes I want 37 different values ).

I also try with  .only()  with very nice response time, but in the end I will get all the fields. For example:
temp = Apple.objects.select_related('Orange').filter(state='fresh').only(
                               'field1',
                               'field2',
                               'field3' )

temp_json = serializers.serialize('json', temp)
return HttpResponse(temp_json, content_type='application/json')

it returns me all the fields! Not only the 'field2', 'field2' and 'field3'.
Could you explain me why ?


Finally, I also tried to run a query with  raw()  but the response time is disappointing.


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Reply all
Reply to author
Forward
0 new messages