Query Caching

205 views
Skip to first unread message

Lyle Karstensen

unread,
Jun 28, 2016, 12:30:32 AM6/28/16
to Lucee
In looking at the CF 2016 version I noticed that cfquery has the ability to specify a cacheid in there query tag. This allows you to pass an array of id's for clearing that cache by those id's. I created my own udf for lucee using mongodb as my query caching with this functionality but would prefer to no longer support my own udf and use the built in functionality. Is there a plan to bring lucee cfquery up to compatibility with CF? I believe this cacheid has been in the code for a year or two.





Lyle Karstensen

unread,
Jul 9, 2016, 10:57:27 AM7/9/16
to Lucee
ANYONE?????? Is anyone out there?

Nando Breiter

unread,
Jul 9, 2016, 11:39:52 AM7/9/16
to lu...@googlegroups.com
Lyle,

For this sort of thing, best to submit an enhancement request. 

That said, a few days ago I was experimenting with using query caching in ACF 11 and found it much too buggy to rely on. I was trying to set the cacheId to be able to programmatically clear the cache, but simply couldn't get it to work.  I wound up implementing it by using cachePut() and cacheGet(), which worked perfectly from the get go. Here's some code to explain:


var qOpenWorkPeriods = cacheGet("sal-findOpenPayableWorkPeriodHoursInDateRange");
if ( isNull( qOpenWorkPeriods ) ) {
var qOpenWorkPeriods = findOpenPayableWorkPeriodHoursInDateRange( arguments.companyId, arguments.startDate, arguments.endDate );
cachePut("sal-findOpenPayableWorkPeriodHoursInDateRange", qOpenWorkPeriods, createTimespan(0,0,0,30));
}

To me, it is much more straightforward to use the cache functions for caching, and they work. You'll find these cache methods also implemented in Lucee, and I'd suggest doing it this way. In both ACF and Lucee, CacheRemove() will take either a list or array of id's. 

In ACF, the cacheId's for queries are buried in an automatic region or something, and no matter what I tried, setting a custom region, using dot notation to try and drill down to the cacheId, using the cacheId without the region, hunting for them using getAllCacheIds, cursing at the screen ... nothing worked. I may simply be an idiot missing something obvious, but I will never try to use a cacheId on a query again in ACF.




Aria Media Sagl
+41 (0)76 303 4477 cell
skype: ariamedia

--
Win a ticket to dev.objective from Lucee via Twitter, see http://bit.ly/1UbTMWj for details, good luck and see you there...
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/936e77c3-6860-4faf-98f0-455716c8773a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nando Breiter

unread,
Jul 9, 2016, 12:35:27 PM7/9/16
to lu...@googlegroups.com
by the way ... the main reason I don't like the cacheId approach on a query is that there are other data types that may need to be cached. Taking this approach, it seems we supposedly need the same cacheID attribute for lists, arrays, structs and components. Is there a cfarray tag? Nope. Where to put this cacheID attribute for other datatypes? This line of reasoning seems to imply we would then also need tags for lists, arrays and structs, <cfarray ... > <cflist ... > <cfstruct ... > so we had a place to hang this cacheID attribute, and that gets horribly awkward. 

Cache functions for each datatype that would take a cacheId argument is another possibility, ArrayCachePut(), ArrayCacheGet(), but unnecessary. A single set of cache functions for all data types makes much more sense. 

So yes, someone is out here, and he thinks using CachePut(), CacheGet() and CacheRemove() is the right way to handle caching in CFML. 





Aria Media Sagl
+41 (0)76 303 4477 cell
skype: ariamedia

Lyle Karstensen

unread,
Jul 9, 2016, 1:56:37 PM7/9/16
to Lucee
Thanks for the reply, that is good to know. I guess I will stick with what I already built.

FYI

I have implemented a Built In Tag that has been doing my query caching for years (since moving to railo). I have a query tag called cfiqquery that is a direct replacement for cfquery. I added 3 parameters (action, tags and compress). I also built my own cfiqqueryparam tag since you cannot use cfqeryparam outside the cfquery tag. All of this code implements query and queryparam with a layer of caching built on mongodb. 

If you want to clear cache that you have tagged you simply call <cfiqquery datasource="blah" action="cleartags" tags="comma,sep,list,of,tags">

My query caching is so fast that a single mongo db replica set can easily handle 2000 requests per second. We actually recently had an issue where the data coming back from cache was too big and was maxing out our 1g network. Therefore I implemented gzip on the json representation of the query to stop that problem. Currently our caching layer responds in roughly 2ms and went from a constant 100MBps to about 12MBps.

Andrew Dixon

unread,
Jul 10, 2016, 2:58:49 PM7/10/16
to lu...@googlegroups.com
Hi Lyle,

If this feature is missing from Lucee please check the Lucee JIRA (https://luceeserver.atlassian.net/) and see if there is a ticket for it, if not please add one and then it will at least be there to be looked at at some point in the future.

Kind regards,

Andrew

--
Win a ticket to dev.objective from Lucee via Twitter, see http://bit.ly/1UbTMWj for details, good luck and see you there...
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Lyle Karstensen

unread,
Jul 10, 2016, 8:30:44 PM7/10/16
to Lucee
I think you may misunderstand how the caching works. The cache storage is the same for each type and has attributes stored in some sort of structure. If you look at the cache and how it is stored it is something like

Array
{
id:somecacheid,
data:['blah1','blah2'],
expires:somedate,
hits:(number of times matched)
cacheid:'blahblabh'
}

String
{
id:hash(),
data:'some string data',
expires:somedate,
hits:(number of times matched)
cacheid:'blahblabh'
}

Struct

{
id:hash(),
data:{key:'value',key:'value'},
expires:somedate,
hits:(number of times matched)
cacheid:'blahblabh'
}

When you do a cached query it creates a json base64 representation of the query then reverses that.

Nando Breiter

unread,
Jul 11, 2016, 10:25:46 AM7/11/16
to lu...@googlegroups.com
My point regarding caching is that the language interface should be kept simple, rather than going down the route of having separate ways to cache different data types. There is a tendency from the ACF devs to keep throwing "features" in the language without much if any thought to design - I think someone called it PHP-ification, where you wind up with a bunch of different ways to do something. It becomes a mess, more difficult to learn than necessary, that isn't well maintained, and difficult to develop and maintain programming tools for. The way this cacheID attribute has been implemented in ACF is a case in point. To me, the solution is to center on one, well designed and well developed way to cache stuff, resisting this tendency to keep throwing multiple ways to do things into CFML, as quickly as possible, releasing them full of bugs, and then moving on to the next release cycle to do the same.



Aria Media Sagl
+41 (0)76 303 4477 cell
skype: ariamedia

Reply all
Reply to author
Forward
0 new messages