Queryset cloning is very expensive

[myx]

Almost every method of QuerySet clones it. Sometimes it is very
ineficcient, for example when querysets are constructed in loops. I
have a function, which makes about 20 queries, and its execution time
is about 100ms. After profiling the function, I saw that clone() takes
most of the time: 70ms. It is called about 60 times, 1ms per call. The
queries themselves are very fast (about 15ms for all queries). So
constructing the query takes much longer time than executing it.
What cloning is needed for? Can I prevent queryset from cloning, or do
something else to reduce cpu consumption?

[kurvenschubser]

Hi myx,

cloning is useful for chaining of filters, e.g.
User.objects.filter(name="Harry").exclude(lastname="Potter").filter(somethingelse="bla")

But most of the time, I found I can construct querysets using a
dictionary for collecting the query conditions:

d = {name:"Harry", lastname:"Potter", somethingelse:"bla"}
User.objects.filter(**d)

For more complex queries, e.g.

if not d:
return


q0 = Q(**dict(d.popitem()))

for i, (k, v) in enumerate(d.items()):
qx = Q(**{k:v})
if i % 2:
q0 | qx
else:
q0 & qx

User.objects.filter(q0)

[myx]

Thank you for the reply. But I meant a slightly different case:

Item.objects.filter(...).order_by(...).values_list('id', flat=True)[:
10]

As you can see, there are four cloning operations. Wouldn't be
chaining possible without cloning the queryset? The method could just
return the original queryset. Seems like I'll have to write raw
queries in such cases :(

On Feb 11, 1:18 pm, kurvenschubser <malte.engelha...@gmx.net> wrote:
> Hi myx,
>
> cloning is useful for chaining of filters, e.g.

> User.objects.filter(name="Harry").exclude(lastname="Potter").filter(somethi ngelse="bla")

[myx]

I found a solution which is suitable for me.
As it turns out, all those methods just clone queryset, and then call
appropriate methods of Query object.
So, for example, to set ordering and to get a slice without cloning
queryset, I can do the following:

qs.query.add_ordering('-created')
qs.query.set_limits(high=10)

[Karen Tracey]

Note you are now using a Django-internal API that is not guaranteed to be stable (see http://docs.djangoproject.com/en/1.2/misc/api-stability/). Your code could very well break in some future version of Django.

Karen

--
http://tracey.org/kmt/

[Alexander Schepanovski]

On 14 фев, 15:21, myx <i.virab...@gmail.com> wrote:
> Thank you for the reply. But I meant a slightly different case:
>
> Item.objects.filter(...).order_by(...).values_list('id', flat=True)[:
> 10]
>
> As you can see, there are four cloning operations. Wouldn't be
> chaining possible without cloning the queryset? The method could just
> return the original queryset. Seems like I'll have to write raw
> queries in such cases :(

My own hacky solutions is:

class QuerySetMixin(object):
def __init__(self, *args, **kwargs):
self._no_monkey.__init__(self, *args, **kwargs)
self._inplace = False

def inplace(self):
self._inplace = True
return self

def _clone(self, klass=None, setup=False, **kwargs):
if self._inplace and klass is None:
self.__dict__.update(kwargs)
return self
else:
clone = self._no_monkey._clone(self, klass, setup,
**kwargs)
clone._inplace = self._inplace
return clone

QuerySet._no_monkey.__init__ = QuerySet.__init__
QuerySet._no_monkey._clone = QuerySet._clone
QuerySet.__init__ = QuerySetMixin.__init__
QuerySet._clone = QuerySetMixin._clone
QuerySet.inplace = QuerySetMixin.inplace

Can be used as:
Item.objects.all().inplace().filter(...).order_by(...).values_list('id',
flat=True)[:
10]

[Malte Engelhardt]

Sorry for the late response.

Maybe it's sufficient to override the QuerySet clone method to just return self, not really clone it.

2011/2/14 myx <i.vir...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.