Is file based cache is safe for concurrent process?

[Ivan]

Hi all.
Can anyone tell, does django.cache locks file for writing by concurrent process?

[Russell Keith-Magee]

You've already asked this on django-users. Django-developers is for discussing the development of Django itself -- it shouldn't be used as "second tier" support if you don't get the answer you want on django-users.

Yours,

Russ Magee %-)

On Wed, Mar 27, 2013 at 8:16 PM, Ivan <ivan...@gmail.com> wrote:

Hi all.
Can anyone tell, does django.cache locks file for writing by concurrent process?

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ivan]

Yes, you are right. But I asked for another reason.
For example, I see, that werkzeug's cache write cache trough temporary file https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
But Django cache do not use neither, tmp file or file locking.
So, multiple processes can write a file simultaneously. Also, other processes can obtain incomplete data.

I think it's a bug, so I asked a question in this group (django-developers).



четверг, 28 марта 2013 г., 0:55:20 UTC+2 пользователь Russell Keith-Magee написал:

[Russell Keith-Magee]

On Thu, Mar 28, 2013 at 7:13 AM, Ivan <ivan...@gmail.com> wrote:

Yes, you are right. But I asked for another reason.
For example, I see, that werkzeug's cache write cache trough temporary file https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
But Django cache do not use neither, tmp file or file locking.
So, multiple processes can write a file simultaneously. Also, other processes can obtain incomplete data.

I think it's a bug, so I asked a question in this group (django-developers).

Ok - *that* is a different matter. For future reference, if you're suggesting that something might be a bug, it's helpful to make that clear in your question. 

As I said in my response on django-users, the file based cache is primarily there as a proof of concept that the caching API can be targeted at multiple backends. However, if someone were to propose a patch to add file locking or temp file write-throughs, I don't imagine it would be turned down.

So - feel free to open a ticket for this; and if you want to try your hand at a patch, this should be a relatively simple patch to create (although tests could be a little bit complex). As a helper, the django.core.files.locks module already contains some OS-abstracted code for file locking.

Yours,

Russ Magee %-)

[Ivan]

> As I said in my response on django-users, the file based...

-- Oh, I didn't see it yet. Thanks.

> if you want to try your hand at a patch, this should be a relatively simple patch to create (although tests could be a little bit complex).

-- yes, it's a good idea, but i need first to study the Contributing guide https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/

Thanks for answer.



четверг, 28 марта 2013 г., 2:23:52 UTC+2 пользователь Russell Keith-Magee написал:

[Ivan Smirnoff]

I was also created the backend, but did not have time to make a fork.
See https://bitbucket.org/evotech/cache-tagging/src/d26adf9f66581923f03ca2d89261568b521eb82e/cache_tagging/django_cache_tagging/backends.py?at=default

Also, I think, it will be better to move directory clearing to Thread for async, and run it not every time (see link above)

By the way, file locking is better than tmp file for resource consumption. During cache body creation, concurrent processes can also make useless cache bodies in case usage tmp file (not locking).

2013/5/31 <ja...@eight.nl>

I've created a ticket with a script that demonstrates the issue and provides an implementation that uses werkzeug's tempfile approach https://code.djangoproject.com/ticket/20536

Before I start forking on github I'd like some feedback on the ticket. Thanks!

Op donderdag 28 maart 2013 01:23:52 UTC+1 schreef Russell Keith-Magee het volgende:

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/PCgYCInrL0g/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

[Ivan Smirnoff]

> During cache body creation, concurrent processes can also make useless cache bodies in case usage tmp file (not locking).

-- sorry, during cache writing to tmp file...

2013/5/31 Ivan Smirnoff <ivan...@gmail.com>