How many read operations is the get() method causing?

[Emil Sandin]

Hi, I am running a Django app on appengine, but I am constantly
hitting the roof with read operations.
I have not found an easy way to find what's causing this, so I am
trying to find it out myself.
To me, it seems that even if I use all().filter(..) or get(), the
database is queried for all instances.

I wrote a custom manager like this:

class LoggingManager(models.Manager):
def get_query_set(self):
query_set = super(SignalsManager, self).get_query_set()
logging.debug("Got the following queryset from the backend:")
logging.debug(query_set)
if query_set.count():
key = "num_of_reads_for_" + str(type(query_set[0]))
num_of_calls_so_far = memcache.get(key) or 0
num_of_calls_so_far += query_set.count()
memcache.delete(key)
memcache.add(key, num_of_calls_so_far )
logging.debug("++++++++++++++++++++++++++++++++++")
logging.debug("total number of reads for " + key)
logging.debug(num_of_calls_so_far)
logging.debug("----------------------------------")
return query_set

I then created a class:

class Foo(models.Model):
bar = models.CharField(max_length=10)
objects = LoggingManager()

When I tried this in the shell I got:

>>> Foo(bar="Hi").save()
>>> Foo(bar="there").save()
>>> Foo.objects.get(bar="Hi")
DEBUG:root:Got the following queryset from the backend:
DEBUG:root:[<Foo: Foo object>, <Foo: Foo object>]
DEBUG:root:++++++++++++++++++++++++++++++++++
DEBUG:root:total number of reads for num_of_reads_for_<class
'models.Foo'>
DEBUG:root:2
DEBUG:root:----------------------------------
<Foo: Foo object>
>>> Foo.objects.all().filter(bar="Hi")
DEBUG:root:Got the following queryset from the backend:
DEBUG:root:[<Foo: Foo object>, <Foo: Foo object>]
DEBUG:root:++++++++++++++++++++++++++++++++++
DEBUG:root:total number of reads for num_of_reads_for_<class
'models.Foo'>
DEBUG:root:4
DEBUG:root:----------------------------------
[<Foo: Foo object>]
>>> Foo.objects.all().filter(bar="NotExisting")
DEBUG:root:Got the following queryset from the backend:
DEBUG:root:[<Foo: Foo object>, <Foo: Foo object>]
DEBUG:root:++++++++++++++++++++++++++++++++++
DEBUG:root:total number of reads for num_of_reads_for_<class
'models.Foo'>
DEBUG:root:6
DEBUG:root:----------------------------------
[]

According to the Django documentation, 'the act of creating a QuerySet
doesn't involve any database activity'.
But since the queryset contains two instances of Foo object when I use
the get() method or all().filter(), isn't the database queried for all
Foo instances? And then the queryset is filtered?
Am I doing something wrong here? Is there a better way to do this?
And does anybody know how this would affect App Engines Quote Details,
since they have a limit of number of operations?

I am using:
Python 2.5.1,
Django (1, 3, 0, 'alpha', 1)

[Daniel Roseman]

This message is probably off-topic for this list, as Django running on AppEngine is not really the same as Django standalone - it depends on django-nonrel to talk to the nonrelational datastore, so database behaviour is very different. You would probably be better off asking on a specific django-nonrel mailing list.

In ordinary Django, each call to LoggingManager would result in a single database hit, which would be executed when it reaches the first logging call, since in order to output the string representation of the QuerySet Django would need to actually evaluate it. The call to type(query_set[0]) would not cause another hit, because Django has already stored the values inside the QuerySet's internal cache.

Clearly django-nonrel is doing something different here, but I don't have enough experience with it to tell you what. I can say that in plain AppEngine, without Django, you can call `queryset.kind()` to get the queryset's model without any database access, but I don't know if that will work with django-nonrel.

--

DR.