caching variable data sample

[Pierre]

purpose is to keep a single renewable pseudo-random data sample live 

the w2p books describes a situation of a constant query :
http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer?search=caching+selects#Caching-selects
def cache_db_select(): 
    logs = db().select(db.log.ALL, cache=(cache.ram, 60)) 
    return dict(logs=logs)

now suppose every call a different query like here :

def cache_this():
    nget = somevalue
    kount = db(db.atable.id > 0).count() 
    offset = randint(0, kount - nget)
    limitby = (offset, offset + nget)
    rows = db(db.atable.id > 0).select(limitby=limitby, cache=(cache.redis, 60))
    return dict(rows=rows)



this accumulates cached 'material' and returns a different rows on every call
how do I make cache_this behave like the book example ?

I could cache kount and offset variables as well :

kount = db(db.atable.id > 0).count(cache=(cache.redis, 60)) 
offset = cache.redis('offset', lambda: randint(0, kount-nget), time_expire=60) 


but i suppose this would lead to a synchronization problem and thus possible duplicate cached select

[Pierre]

the cache select mechanism is full of mystery:

given book example code:

def cache_db_select(): 
    logs = db().select(db.log.ALL, cache=(cache.ram, 60)) 
    return dict(logs=logs)

what happens to next cache_db_select call after the 60 seconds has elapsed ?
Does it overwrite the previously cached rows ?

One need to know a key in order to clear a specific cached value associated with that key. When performing a cache select no key is being passed so how to clear a specific cached select item (for instance logs) ?

this gives me nerve crises........It should be simple to do a simple thing however one week digging this and I didn't move 1 inch ahead

[Anthony]

Looks like you want to cache the results of the entire function -- so why not do that:

@cache(some_key, time_expire=60, cache_model=cache.ram)
def cache_this():
    ...
    rows = db(db.atable.id > 0).select(..., cacheable=True)
    return dict(rows=rows)

Just be sure to set cacheable=True, which will make the entire Rows object cacheable.

Anthony

[Anthony]

On Thursday, September 14, 2017 at 9:33:42 AM UTC-4, Pierre wrote:

the cache select mechanism is full of mystery:

given book example code:

def cache_db_select(): 
    logs = db().select(db.log.ALL, cache=(cache.ram, 60)) 
    return dict(logs=logs)

what happens to next cache_db_select call after the 60 seconds has elapsed ?
Does it overwrite the previously cached rows ?

It is just using the web2py caching mechanism, which is described here: http://web2py.com/books/default/chapter/29/04/the-core#cache. So, yes, if the time has expired, the old cached value will be replaced with a fresh value.

 

One need to know a key in order to clear a specific cached value associated with that key. When performing a cache select no key is being passed so how to clear a specific cached select item (for instance logs) ?

If you need to manually clear cached selects, you should probably not use the above mechanism, as the key used is part of the internal implementation (it is the md5 hash of the database URI plus the raw SQL query). Instead, you should set cacheable=True in the .select() call, and then just cache the resulting Rows object using cache.ram() with whatever key you like.

 

this gives me nerve crises........It should be simple to do a simple thing however one week digging this and I didn't move 1 inch ahead

 

Maybe you should have asked a week ago. ;-)

Anthony

[Pierre]

actually the real function is a little more complex since it returns rows plus other datas that don't require caching. I tried to move the cache_this function to a module (don't know if this is orthodox) but then i get a decorator error:

@current.cache('data_sample', time_expire=60, cache_model=current.cache.redis)
attributeError cache object has not attribute redis

[Pierre]

very True: productivity rules and I am far below modern standards........(more or less the 'Bangladesh level')    :)

[Pierre]

the more digging the deeper one gets :

https://groups.google.com/forum/?fromgroups#!searchin/web2py/cache$20in$20module/web2py/AZa5Boj3y3E/_BPMTdXwSaMJ

and here is the cache_this  module version :

from gluon.cache import lazy_cache

@lazy_cache('data_sample', time_expire=60, cache_model='redis')
def cache_this():
    ...
    rows = db(db.atable.id > 0).select(limitby=limitby, cacheable=True)
    return rows

This seems to do the trick.  Is it 'kosher' ?

[Anthony]

On Friday, September 15, 2017 at 6:15:44 AM UTC-4, Pierre wrote:

actually the real function is a little more complex since it returns rows plus other datas that don't require caching. I tried to move the cache_this function to a module (don't know if this is orthodox) but then i get a decorator error:

@current.cache('data_sample', time_expire=60, cache_model=current.cache.redis)
attributeError cache object has not attribute redis

You don't necessarily have to put the function in a module -- it can go in a model or even in the controller (if the function takes no arguments, start its name with a double underscore to prevent it from being exposed as an action). You can even define the function inside the controller function that calls it if that's the only place it is needed.

Anthony

[Anthony]

Forgot about that. It's documented here: http://web2py.com/books/default/chapter/29/04/the-core?search=lazy_cache#Warning--Do-not-use-the-current-object-in-global-scope-in-a-module. Yes, that would be the way to go if you want to move the function to a module.

Anthony

[Pierre]

I'd prefer the module option but I need to pass time_expire as a variable (db.atable.__variableName), I don't think I can do that in a module can I ?

[Anthony]

On Monday, September 18, 2017 at 10:09:51 AM UTC-4, Pierre wrote:

I'd prefer the module option but I need to pass time_expire as a variable (db.atable.__variableName), I don't think I can do that in a module can I ?

You are not limited to using the decorator to cache the results of a function. Instead, you can just do something like this:

from mymodule import cache_this

myrows = cache.redis('myrows', cache_this, time_expire=myvariable)

Anthony

[Pierre]

thanks Anthony...

looks like the cache select puzzle is now complete..... :  )


however we still have a 'serious productivity lag here in Bangladesh'.......