Performance optimisation documents, ticket 20877

[Daniele Procida]

<https://code.djangoproject.com/ticket/20877>

Work in progress; comments appreciated: <https://github.com/django/django/pull/1463>

Daniele

[Daniele Procida]

I noticed while looking for material for <https://code.djangoproject.com/ticket/20877> that <https://docs.djangoproject.com/en/dev/models/instances.html#updating-attributes-based-on-existing-fields/> mentions that:

product.number_sold = F('number_sold') + 1

is faster than:

product.number_sold += 1

though this doesn't seem to be mentioned in the database optimisation page.

That's easy enough to address, and <https://docs.djangoproject.com/en/dev/topics/db/optimization.html#do-database-work-in-the-database-rather-than-in-python/> seems like a sensible place for it.

However the mentions of F() that I did find raised a number of questions.

The F() class seems to be a handy general-purpose way to refer to the value of a model field..


Firstly, it's not explained how, in expressions like:

product.number_sold = F('number_sold') + 1

(from <https://docs.djangoproject.com/en/dev/models/instances.html#updating-attributes-based-on-existing-fields/>) Django knows that F('number_sold') is refers to the product model.

Does it know because product.number_sold is the field that this expression refers to? What would happen if we did:

product.number_in_stock = F('number_in_stock') - F('number_sold)

(i.e. can we make such calculations multiple other fields in one go?), or:

product.number_to_reorder = F('number_sold)

for example? What are the rules of the usage syntax of F()?

Secondly, the main documentation for F() <https://docs.djangoproject.com/en/dev/topics/db/queries.html#query-expressions/> doesn't mention this kind of use at all: it only suggests that it might be useful in queries.

Since this use seems to be just one of multiple uses for F(), shouldn't a more general description of F() belong somewhere else (where) instead?


Finally, are there any other useful ways to use F() not covered by these two examples?


Daniele

[Marc Tamlyn]

(Disclaimer: I didn't write any of this code, I'm just working from my own possibly flawed understanding)

The main point of using F is not generally for performance per se, rather for database consistency. In the first example, the performance increase is negligable, but you get two significant benefits. The first is that you minimise the risk of race conditions by effectively issuing "increment" rather than "set" to the database. The second is that you can increase a number of rows the same way - Product.objects.update(number_sold=F('number_sold') + 1). These add performance in that you end up doing fewer queries, but that's actually what you're gaining.

It is worth noting that the *only* reason to set an attribute to be an F expression is to then update the value in the database. In fact, you cannot access the value after you've done this, you get a "django.db.models.expressions.ExpressionNode" instance. In that sense, doing  product.number_sold = F('number_sold') + 1 is really a long hand version of the update() query.

As for what expressions are valid, I believe the things you suggested do work, but I imagine exactly what works will depend on what operations the database (and backend) support. Django itself isn't doing anything clever here, it's just providing some architecture to allow you to delegate functionality to the database.

I agree that the main documentation for F() should reference the update case (IMO as an update() rather than the longhand version).

Marc

Daniele

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

[Some Developer]

I've always found the documentation on the usage of F() to be somewhat
substandard compared to the rest of the documentation to the point were
I rarely if ever use it.

If the documentation could be beefed up I'm sure more people would be
happy to use it in the general case rather than just small side cases.

[Curtis Maloney]

If, at the same time, perhaps ExpressionNode got a mention too, that could really open up some opportunities :)

--
You received this message because you are subscribed to the Google Groups "Django developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

[Andre Terra]

On Sat, Aug 17, 2013 at 10:22 PM, Curtis Maloney <cur...@acommoncreative.com> wrote:

If, at the same time, perhaps ExpressionNode got a mention too, that could really open up some opportunities :)

Speaking of opportunities, here's a worthwhile ticket if we want to go further than docs for improving F():

https://code.djangoproject.com/ticket/14030

Cheers,

AT

[Curtis Maloney]

Just a simple case of what I was talking about earlier, here's something we used to avoid having to pass datetime.datetime.now() to a lot of timestamp filtering queries:

class Now(ExpressionNode):

    def __init__(self):

        super(Now, self).__init__(None, None, False)

    def evaluate(self, evaluator, qn, connection):

        return 'CURRENT_TIMESTAMP', []

A little research shows this should work [almost] identically on all the majors, and even Oracle and MS SQL.  The "almost" is become some will shift the timestamp during a transaction, and others won't.

Now, how many other handy "SQL Standard" functions/expressions are there that people would love to use?

--

Curtis

--

You received this message because you are subscribed to the Google Groups "Django developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

[Daniele Procida]

Any further comments would be welcomed. There's some disagreement about the appropriateness of the last section, <https://github.com/django/django/pull/1463/files#L5R318> so it would be particularly usefukl to have some further opinion on that.

Daniele

[Florian Apolloner]

Hi Daniele,

On Friday, September 13, 2013 4:18:05 PM UTC+2, Daniele Procida wrote:

Any further comments would be welcomed. There's some disagreement about the appropriateness of the last section, <https://github.com/django/django/pull/1463/files#L5R318> so it would be particularly usefukl to have some further opinion on that.

Personally I'd remove it completely for the following reasons:
 * Especially newcomers tend to think that they need the most performant system and will probably try to switch out whatever they can, even if not necessary; and switching to PyPy might not be a good idea in most of those cases ;)
 * This dual-import of libraries where you try to select the most "efficient" versions quite often results in issues for those versions which didn't get used by the developer. As an example simplejson and json are not 100% compatible and I think had issues with the c-variant of Elementtree too. Especially when it comes to unicode/str handling the c-variants tend to have some ugly edge cases (at least in my experience -- this is not a problem though, but you have to test all variants…). Imo it's most of the times easier to just develop against one, if you need the speed and verified that the c-variant actually offers an improvement use that and __require__ it.

Florian

[Daniele Procida]

On Fri, Sep 13, 2013, Florian Apolloner <f.apo...@gmail.com> wrote:

>On Friday, September 13, 2013 4:18:05 PM UTC+2, Daniele Procida wrote:
>>
>> Any further comments would be welcomed. There's some disagreement about
>> the appropriateness of the last section, <
>> https://github.com/django/django/pull/1463/files#L5R318> so it would be
>> particularly usefukl to have some further opinion on that.
>>
>
>Personally I'd remove it completely for the following reasons:
> * Especially newcomers tend to think that they need the most performant
>system

)
> * This dual-import of libraries where you try to select the most
>"efficient" versions quite often results in issues for those versions which
>didn't get used by the developer

I've re-written that section now to emphasise the caveats and the need to verify improvements, and to de-emphasise the prospect of exciting perfromance gains.

Daniele

[Tim Graham]

I feel like we open a can of worms if we go down this road. Where do we draw the line between what we do and don't mention regarding *Python* optimization? The Django docs don't need to layout the current state of the Python ecosystem IMO. I don't doubt the info is helpful, but I don't think Django's docs is the right place for it. There are many Python web frameworks out there, should every framework mention this type of stuff in their docs? It seems like a duplication of effort.

(summary of what I said in IRC)

[Marc Tamlyn]

+1 to what Tim said. Most Django sites performance problems are not at the level of the python code running, rather inefficient databases/caching/templates etc. If your core logic is comparatively slow you're already past a basic optimisation guide. Keep it simple, and BENCHMARK.