Concurrent CacheLoader implementation

[Tim Peierls]

I posted a new blog entry that includes a concurrent CacheLoader implementation, i.e., it implements loadAll to load the values concurrently, using an ExecutorService, rather than sequentially.

http://tembrel.blogspot.com/2012/03/executorservice-returned-by.html

The main point of the posting is actually about using Hazelcast to implement a distributed CacheLoader, so there is Hazelcast-specific code, but the non-Hazelcast case portion is easily extractable and seems useful on its own. I'd hate to be wheel-reinventing, but while you'd think there would be dozens of things like this out there, I can't find any. Maybe that's just a sign that Guava Cache/LoadingCache is still a new thing.

--tim

[Benjamin Manes]

I think its because most of the time the client library to a storage tier includes batched reads to avoid slow sequential calls or query storms. So the concurrency, timeout handling, etc. are mostly abstracted from the caller. For instance when I first implemented a bulk memoizing cache ~5 years ago it was on top of memcached. Are you sure that Hazelcast doesn't have something like this as well? Not knowing that library, naively MapLoader looks promising.

--tim

--
guava-...@googlegroups.com
Project site: http://guava-libraries.googlecode.com
This group: http://groups.google.com/group/guava-discuss
 
This list is for general discussion.
To report an issue: http://code.google.com/p/guava-libraries/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "guava")

[Tim Peierls]

The limitation with MapLoader is that you have to specify it as part of the configuration supplied when creating the Hazelcast instance -- which normally happens during start-up of a long-running server.

I did a lot of work with MapLoader and MapStore a while back, badgering the Hazelcast team into adding a MapStoreFactory to the configuration so I could inject my MapLoader/MapStore instances using Guice. I even found a way to convey per-IMap loader/store parameters to a MapStoreFactory by encoding them into the name of the IMap instance, using a template: map-{path}-{keyType}-{valueType}-{eagerKeyLoading}. It's perverted, but it works.

But there's no way to convey an arbitrary Function using this scheme, so an IMap-based loadAll is limited to a fixed set of loader functions that you specify in advance of creating the Hazelcast instance. The ConcurrentCacheLoader approach only requires that the Function passed to the ConcurrentCacheLoader Builder (and the key and value types) be Serializable.

However, I do think it would be worth writing an IMap-based Cache and LoadingCache, i.e., using an IMap (without a MapLoader or MapStore) to cache the values so that they are available to Caches with the same name on other nodes of the cluster. That could be a big win if the cost of an optimized call within the cluster to serve a cached value is often significantly less than the cost to load the value from scratch. (Note to Adrian Cole: This might apply in the "distributed jclouds context" setting that you mentioned in a different thread.)

--tim

[Charles Fry]

Hi Tim,

This is very interesting, and I'm glad to see it written up so clearly! I could easily get behind adding something like com.google.common.cache.CacheLoaders.loadAllInParallel(CacheLoader<K, V>, Executor) which turned the sequential calls into parallel calls. Would that be enough for you?

It ends up that we already have an internal CacheLoaders.asyncReload(CacheLoader<K, V>, Executor), and this seems like a good fit to go right along side it.

I don't see either of these being core enough for deeper integration, but I'm in favor of supporting common but simple CacheLoader extensions in a CacheLoaders class.

Charles

--tim

--

[Tim Peierls]

On Mon, Mar 12, 2012 at 11:14 AM, Charles Fry <f...@google.com> wrote:

This is very interesting, and I'm glad to see it written up so clearly! I could easily get behind adding something like com.google.common.cache.CacheLoaders.loadAllInParallel(CacheLoader<K, V>, Executor) which turned the sequential calls into parallel calls. Would that be enough for you?

(I like the idea of wrapping an existing CacheLoader to add the loadAll implemention!)

It would be great to have, but it wouldn't address my primary motivation, which was to distribute the Function evaluations across a Hazelcast cluster. That's OK -- Guava doesn't have to do everything for me! -- but even more flexible for my purposes would be CacheLoaders.loadAllInParallel(ConcurrentFunction<K, V>), where ConcurrentFunction could be defined along these lines:

https://pastebin.com/Y9GV7Fkw

except ConcurrentFunction should be an interface (with an accompanying AbstractConcurrentFunction), and you might not want to include the timed version of apply. (For my purposes, I would instead embed the time constraint into my custom ConcurrentFunction instance.)

But adding ConcurrentFunction just for this one use isn't right. (I can think of at least one other possible use: parallel Maps.transformEntries.)

 

It ends up that we already have an internal CacheLoaders.asyncReload(CacheLoader<K, V>, Executor), and this seems like a good fit to go right along side it.

Yup.

Implementing reload in ConcurrentCacheLoader (and DistributedCacheLoader) is on my list.

 

I don't see either of these being core enough for deeper integration, but I'm in favor of supporting common but simple CacheLoader extensions in a CacheLoaders class.

Sounds good to me. 

I'm still planning to create Cache and LoadingCache implementations that use Hazelcast-distributed maps to hold the cached values, which only makes sense if loading is very often much slower than access across the cluster. There's a lot of design space here, including whether to have a local near cache (using a standard Guava cache) with distributed notification of invalidation -- speeds up common accesses but increases network traffic. (Ideally I can support the whole space via a builder.)

In addition, I'll use your idea of wrapping existing CacheLoaders to add distributing loading so that the user won't have to supply a distributed CacheLoader to an already distributed cache.

--tim

[Louis Wasserman]

We already do have AsyncFunction.  Might that be appropriate here?

Louis Wasserman
wasserm...@gmail.com
http://profiles.google.com/wasserman.louis

--

[Tim Peierls]

Nice! Yes, I think that would work. In which case, no need for CacheLoaders.loadAllInParall, just add static CacheLoader.from(AsyncFunction<K, V>) I'll recast what I've done in those terms. For my Hazelcast case, I've just now written an AsyncFunction implementation and a static factory method that converts a Function and a (Hazelcast) Executor to an instance of that  AsyncFunction impl.

That keeps the Hazelcast-specific stuff neatly stowed away. 

Is there a static method that takes an AsyncFunction and an Iterable of keys and produces a Map of keys to values? I'll want to write one if not.

--tim

[Louis Wasserman]

No, but it'd be straightforward enough to whip up using Futures.allAsList or successfulAsList, depending on your preferences.

Louis Wasserman
wasserm...@gmail.com
http://profiles.google.com/wasserman.louis

[Tim Peierls]

Oh, right. duh

[Charles Fry]

Ah, this is getting interesting.

Actually I've been planning to create an AsyncCacheLoader and AsyncLoadingCache, though I haven't worked through all the details in the design yet. It sounds like we might be able to at least create AsyncCacheLoader now. Then we could have AsyncCacheLoader.from(AsyncFunction<K, V>).

Note that the advantage of going the AsyncCacheLoader route instead of just adding CacheLoader.from(AsyncFunction<K, V>) is that it facilitates the selective over-riding of reload or loadAll.

How does that sound?

Charles

[Louis Wasserman]

Ooooh.  I like that. Asynchronous loading and caching isn't exactly addressed by LoadingCache, even now, is it...

Louis Wasserman
wasserm...@gmail.com
http://profiles.google.com/wasserman.louis

[Charles Fry]

Right. Having a Cache<K, Future<V>> has all kinds of unexpected gotchas too.

You'll notice that I've already prepped the internals of LocalCache to speak in terms of Futures, so the main remaining task now is to agree on the API.

Here are my specific thoughts so far; I welcome feedback:

AsyncCacheLoader

ListenableFuture<V> loadFuture(K key) throws Exception;

ListenableFuture<V> refresh(K key) throws Exception;

Map<K, ListenableFuture<V>> loadAllFutures(Iterable<? extends K> keys) throws Exception;

AsyncLoadingCache

The LoadingCache methods we need are:

Future<V> getFuture(K key);

void refresh(K key);

ImmutableMap<K, Future<V>> getAllFutures(Iterable<? extends K> keys);

The Cache methods are a little more tricky. On the one hand, most of them are less obviously asynchronous. On the other hand, other Cache implementations might need to accomodate asynchronous errors even from void methods. The simplest methods are these ones:

void cleanUp();

ListenableFuture<V> getFuture(K key, Callable<? extends V> valueLoader);

ImmutableMap<K, V> getAllPresent(Iterable<? extends K> keys);

V getIfPresent(K key);

long size();

CacheStats stats();

That still leaves all of the invalidate and the put methods. Do we want to make those asynchronous, exposing some error handler or some Future<Void> to return errors?

Open Questions

• What should the asMap() view do? Expose V or Future<V>?
• Do we want to make it avoid name collisions between CacheLoader and AsyncCacheLoader, and between LoadingCache and AsyncLoadingCache?

[Tim Peierls]

I've gone ahead and written CacheLoaders.fromAsyncFunction(AsyncFunction<K, V>) -- thank you, Louis, for suggesting this -- and it addresses my needs nicely. (Also wrote a HazelcastAsyncFunction that hides all the Hazelcast details from the rest of the machinery.)

But I don't see what AsyncCacheLoader and AsyncLoadingCache add to the picture.

--tim

[Tim Peierls]

On Mon, Mar 12, 2012 at 9:25 PM, Tim Peierls <t...@peierls.net> wrote:

But I don't see what AsyncCacheLoader and AsyncLoadingCache add to the picture.

Ah, I missed Charles' note:

Note that the advantage of going the AsyncCacheLoader route instead of just adding CacheLoader.from(AsyncFunction<K, V>) is that it facilitates the selective over-riding of reload or loadAll.

But it's not hard to imagine additional static factory methods taking extra Function or AsyncFunction arguments to specify reload and/or loadAll behavior, without having to add the full AsyncCacheLoader machinery, which feels like a lot of work for no compelling uses that I've seen so far.

--tim

[Tim Peierls]

Summarized some of today's developments here:

http://tembrel.blogspot.com/2012/03/in-discussion-on-guava-mailing-list.html

I left the AsyncCacheLoader stuff out, because I'm missing the use cases.

--tim

[Louis Wasserman]

I'm not convinced of the correctness of this implementation -- in particular, I don't believe that the map is guaranteed to be filled when you return it from loadAll; just adding the callback could cause the entries to get added to the map at any time.

Instead, I might consider going ahead and paying the cost of sequentially filling an ImmutableMap with the entries before returning; that should still be cheap compared to the computations you're farming out.  Not sure if there's a better way.

Louis Wasserman
wasserm...@gmail.com
http://profiles.google.com/wasserman.louis

[Tim Peierls]

Responded briefly to your comment on my blog before seeing this. 

But my original thinking was that the Future returned by successfulAsList only returns its List when all the (Settable)Futures have completed, through a call to either set or setException (leaving aside the possibility of IE). And Futures.addCallback with two arguments runs the callback in the same thread, right? So returning the results map directly should be OK.

I'm not thrilled about the unused List<V> returned from successfulAsList, but it was sooo easy to write... :-)

--tim

[Tim Peierls]

Thanks (again) to Louis showing me a race that I obstinately didn't want to see (to the point of misreading the Javadocs!), I've fixed the CacheLoaders.fromAsyncFunction implementation. It's much simpler now.

Blog entry and code snippet updated.

--tim

[Charles Fry]

I left the AsyncCacheLoader stuff out, because I'm missing the use cases.

Here's what it comes down to: By focusing on AsyncFunction we allow users an easy way to specify an asynchronous load, and then we bind the refresh and loadAll implementations. So as soon as you want asynchronous load and a custom loadAll, you have to re-implement the asynchronous load yourself.

By introducing AsyncCacheLoader it is no longer necessary for users to implement AsyncFunction, as they could with the same amount of work simply extend AsyncCacheLoader. (Of course we still might want a convenience method in case they already had an AsyncFunction.) By extending AsyncCacheLoader instead of AsyncFunction, it would then become trivial to later decide to override loadAll, and to be presented with an asynchronous API for doing so.

Does that sound right?

Charles

[Kevin Bourrillion]

Yeah; the way I see it, if the use cases are considered important, then AsyncCacheLoader is a more robust approach to addressing them.

--

guava-...@googlegroups.com
Project site: http://guava-libraries.googlecode.com
This group: http://groups.google.com/group/guava-discuss
 
This list is for general discussion.
To report an issue: http://code.google.com/p/guava-libraries/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "guava")

--
Kevin Bourrillion @ Google

Java Core Libraries Team
http://guava-libraries.googlecode.com

[Louis Wasserman]

Isn't it the caes that with Cache alone, you can't support ListenableFuture<V> Cache.getAsync(K key, Callable<? extends V> load), or the equivalent -- at least not without a lot of very tricky logic (I think?).  And I can totally see that being a thing that deserves support!

Louis Wasserman
wasserm...@gmail.com
http://profiles.google.com/wasserman.louis

[Charles Fry]

Tim, I spoke with Chris and we agreed that the short term thing we want to add is indeed CacheLoader.from(AsyncFunction<K, V>).

Would you be interested in putting together a patch for us based on what you've already done, and sending it to Louis for review?

Charles

[Tim Peierls]

Sure, will send to Louis directly.

[Charles Fry]

FYI, we generally use http://codereview.appspot.com/ for external code reviews. Louis could tell you more about the process than I can...

[Adrian Cole]

jclouds would love to beta test this: all of our rest clients return
ListenableFuture<T>, leading to extremely straightforward cache
management once this is in.

-A

[Charles Fry]

We'll definitely update this thread once the review is complete, and the code is submitted.

[Tim Peierls]

Resurrecting nearly-year-old conversation: Whatever happened to CacheLoader.from(AsyncFunction)? 

--tim