Skip to first unread message
Kiran Subbaraman
Oct 15, 2015, 10:20:55 AM10/15/15
to web2py-users
Hello,
I wrote code where I was caching various data, and it so happened that to construct some of this data, the caches were recursively called. The caches are a combination of cache.ram with cache.disk.
I didn't realize this was an issue, till I tried to run it in the web2py environment.
Basically the request to the server to fetch this data was never completed - I suspect this is because of locks at the cache.disk level.
I have attached a sample application that helps replicate the issue.
I have a need to compute various data, prior to serving requests. These are computed, and cached (combination of cache.ram with cache.disk)
The results of some of the requests are cached too.
This definitely leads to a situation where caches are called recursively.
Please let me know if I need to elaborate on something here ... will gladly do so.
Kiran
I wrote code where I was caching various data, and it so happened that to construct some of this data, the caches were recursively called. The caches are a combination of cache.ram with cache.disk.
I didn't realize this was an issue, till I tried to run it in the web2py environment.
Basically the request to the server to fetch this data was never completed - I suspect this is because of locks at the cache.disk level.
I have attached a sample application that helps replicate the issue.
I have a need to compute various data, prior to serving requests. These are computed, and cached (combination of cache.ram with cache.disk)
The results of some of the requests are cached too.
This definitely leads to a situation where caches are called recursively.
- Is there a best practice that I can follow for a situation like this? If I shouldn't recursively cache, then what else can I do?
- If the cache.disk locks up (am suspecting this, though am not able to produce evidence towards this), or there is an issue at this level, I was expecting to see some details in the web2py log. I had enabled the DEBUG log level.
- I tried to debug this using the interactive browser based debugger, but then that debugger steps out of the debug stack, once it encounters this 'lock' issue. I had to resort to using Eclipse Pydev to run web2py within it, and run the debugger from there. It was inconclusive too, but atleast the debug view did not disappear on me.
Any suggestions in this space, will really help me. In the meantime, I will have to rework my application to see if I can step away from these recursive caches for the time being. Or atleast use cache.ram and cache.disk, such that I do not end up with cache.disk being invoked recursively.
Please let me know if I need to elaborate on something here ... will gladly do so.
Kiran
Leonel Câmara
Oct 15, 2015, 10:29:23 AM10/15/15
to web2py-users
Yep, the cache locks the key it's writing so this is not possible. If the keys are not the same it still gets locked because of statistics gathering. The only way to prevent this would be to have a disable statistics option in cache disk, which I am inclined to develop as it would also allow truly concurrent access to cache disk.
Kiran Subbaraman
Oct 15, 2015, 10:41:04 AM10/15/15
to web2py-users
Thanks for confirming this issue.
The other thing I noticed is that if I just go ahead and create an instance of CacheInRam and CacheOnDisk, and combine them as before, and go ahead and use them - this would mean that a cache ram & disk instance is created for practically every request, wasteful - but then I do not see this locking happening. So, this locking is an issue with the same instance being used to access the disk cache ... and doesn't arise if different cache disk instances are trying to work recursively?
Also, if there is any help I can extend to help with this issue, please let me know. Am in the middle of putting up my building and deploying my application ... but will set aside time to help with this issue.
The other thing I noticed is that if I just go ahead and create an instance of CacheInRam and CacheOnDisk, and combine them as before, and go ahead and use them - this would mean that a cache ram & disk instance is created for practically every request, wasteful - but then I do not see this locking happening. So, this locking is an issue with the same instance being used to access the disk cache ... and doesn't arise if different cache disk instances are trying to work recursively?
Also, if there is any help I can extend to help with this issue, please let me know. Am in the middle of putting up my building and deploying my application ... but will set aside time to help with this issue.
Leonel Câmara
Oct 15, 2015, 11:07:12 AM10/15/15
to web2py-users
Ohh wait a minute I thought only cache disk was involved. If you create new instances of CacheRam instead of using the one already provided you are not using the cache at all as the new instances will have an empty cache.
Kiran Subbaraman
Oct 15, 2015, 11:19:17 AM10/15/15
to web...@googlegroups.com
Ok, two things:
Firstly - As demonstrated in the sample application that I attached with the original post, I was using the combination of cache.ram with cache.disk, as my default cache: cache.ram(key, lambda: cache.disk(...), ttl)
Secondly, in the scenario where I create instance of CacheInRam, and CacheOnDisk, and use them like so:
cache_ram = CacheInRam()
cache_disk = CacheOnDisk(ONE_CONSTANT_FOLDER)
cache_ram(key, lambda: cache_disk(key, lambda: ..., ttl), ttl)
In the second case, yes the instances are new ones per request ( but this is a scenario I tried when testing my application independent of web2py). Note that it goes to the disk everytime, and all the cache disk instances are referring to the same cache folder. So, in this case, whether it refers to the same cache folder or not, it basically is not sharing the cache data ... is that what you meant? This is the reason why I did not see any contention.
Whereas, in this second case, if I were to retain a singleton of cache_disk, then I should be seeing the issue with cache.disk?
Firstly - As demonstrated in the sample application that I attached with the original post, I was using the combination of cache.ram with cache.disk, as my default cache: cache.ram(key, lambda: cache.disk(...), ttl)
Secondly, in the scenario where I create instance of CacheInRam, and CacheOnDisk, and use them like so:
cache_ram = CacheInRam()
cache_disk = CacheOnDisk(ONE_CONSTANT_FOLDER)
cache_ram(key, lambda: cache_disk(key, lambda: ..., ttl), ttl)
In the second case, yes the instances are new ones per request ( but this is a scenario I tried when testing my application independent of web2py). Note that it goes to the disk everytime, and all the cache disk instances are referring to the same cache folder. So, in this case, whether it refers to the same cache folder or not, it basically is not sharing the cache data ... is that what you meant? This is the reason why I did not see any contention.
Whereas, in this second case, if I were to retain a singleton of cache_disk, then I should be seeing the issue with cache.disk?
________________________________________ Kiran Subbaraman http://subbaraman.wordpress.com/about/
On Thu, 15-10-2015 8:37 PM, Leonel
Câmara wrote:
Ohh wait a minute I thought only cache disk was involved. If you create new instances of CacheRam instead of using the one already provided you are not using the cache at all as the new instances will have an empty cache.
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Leonel Câmara
Oct 15, 2015, 11:59:18 AM10/15/15
to web2py-users
No, cache disk does share data as long as it's the same folder and it locks the files themselves.
Anyway using the caches as in your last scenario should not cause any problems.
Kiran Subbaraman
Nov 3, 2015, 12:50:00 AM11/3/15
to web2py-users
Just checking - should I open an issue for this... so that it is tracked?
On Thursday, October 15, 2015 at 7:59:23 PM UTC+5:30, Leonel Câmara wrote:
Reply all
Reply to author
Forward
0 new messages