Fastest way to incrBy and expire on multiple keys

[sh...@digitalsanctum.com]

I'm looking for guidance/suggestions on the best (fastest) way to perform an INCRBY and EXPIRE call on multiple keys. I'm doing this from an AWS Lambda function and operating on ElastiCache (AWS managed Redis version 3.2.4). For the Lambda function, I have the option of using at most two threads and the number of "aggregates" will vary but usually under 1000. Since the Lambda operates with a hard stop timeout, I could use async but need to be assured the time to perform these operations are within the timeout. 

The code I currently have looks like this:

    List<Aggregate> updatedAggregates = null;

    try (StatefulRedisConnection<String, String> connection = this.redisClient.connect()) {

      RedisCommands<String, String> syncCommands = connection.sync();

      log.info("Connected to Redis in {}", stopwatch.stop().toString());

      stopwatch.start();

      updatedAggregates = aggregates.stream()

          .map(aggregate -> {

            String key = aggregate.getId();

            Long requestCount = aggregate.getRequestCount();

            Long newCount = syncCommands.incrby(key, requestCount);

            syncCommands.expire(key, this.aggregateTtl);

            return new DefaultAggregate(key, aggregate.getBlockedValue(),

                aggregate.getTimeBucket(), newCount, aggregate.getExpirationTime());

          })

          .collect(toList());

    } catch (Exception e) {

      log.error("Error updating aggregates", e);

    }

    ...
    // do more work with updatedAggregates...

While this works, it's much slower than I expect it to be and I'm looking to double performance.

Thanks in advance!

[Mark Paluch]

First of all: Please do not cross-post.

I think you could improve by using the async API without actually going async.

What happens in your code is that you issue commands and await command completion before you send the next command. The wait contains the time in which the command is sent over the transport, received, processed and responded including all network-generated latency.

You could do two things in which the second one is just an optimization:

1. Use the async API to issue INCRBY and EXPIRE commands. Since you're in a synchronous flow, it makes no sense to deal with callbacks. Rather collect all futures and make sure all futures are completed/synchronize with a timeout (get(timeout, TimeUnit)). This way you're not required to wait for command completion and you get rid of some latency here. Each command is sent individually.

2. Your code looks quite isolated, you could use command batching which reduces the number of TCP frames sent to your Redis server and the network packages are more compact. Check out the documentation [0] about flushing/pipelining.

Cheers, 

Mark

[0] https://github.com/lettuce-io/lettuce-core/wiki/Pipelining-and-command-flushing#command-flushing