vert.x exactly-once processing

[Girish M]

We would like to use vert.x for BigData event processing. Events are received from a source as pre-defined JSON and stored in Hadoop for further analysis. Normally, Kafka or something similar is used, but we think vert.x fits in neatly here.

Requirements:

* Support high-volume of events (~1000 events/sec)

* Exactly-once delivery

Looking at the vert.x architecture, exactly-once delivery is not supported inherently and the application has to build this capability with a persistent datastore. Is there a better way to achieve this natively in vert.x?

[jordan.h...@gmail.com]

Vert.x does not provide exactly once semantics because exactly-once is very difficult to abstract in such an unopinionated framework and what "exactly once" means is dependent upon the application.

In reality, there's no such thing as exactly-once processing, there is only idempotence and through it exactly-once semantics. Systems that provide exactly once *semantics* do so through idempotence, and you can do the same. For instance, Storm's exactly once semantics are in reality achieved by processing and acking tuples in batches by storing resulting state in a persistent store. In the event of a replay of tuples, tuples in old batches will be ignored. But what's critical about how Storm provides exactly-once semantics is that it stores the metadata about batches with the data that results from those batches. That is, it stores the last batch processed with the resulting aggregate.

Similarly, Kafka's high level consumer batches messages and stores the highest offset processed by each consumer for each partition. But this does not achieve exactly-once semantics. If a consumer fails half way through processing a batch, the already processed messages will ultimately be replayed to some other consumer. If that consumer is non-deterministic (e.g. stores state in some external database) that can have detrimental effects. In order to achieve exactly-once processing, state has to be stored *with the last offset processed* for each processed message or batch of messages, otherwise there's some potential that the consumer can fail between storing the last processed offset and storing the output of the last processed offset.

So, that is to say that true exactly-once processing is a fantasy, and idempotence in the interest of exactly-once semantics is far too opinionated for Vert.x. But you can certainly create exactly-once semantics via idempotence on the Vert.x event bus, though it will require some coordination. Ultimately, the specific implementation is dependent on your application. At what point do exactly once semantics apply? Once the message is sent by a producer, or once it's received by a consumer? The basic pattern is this: publish messages from a producer with a timeout and tag messages with sequential identifiers. When a message is processed, store the last processed message with the result of that message in a fault-tolerant persistent store. If a message with a sequential ID lower than the last processed message is received, discard it. When a message is processed, ack it by replying to the producer. If a producer's send times out, resend the message with the same sequential ID. The idempotence of the consumer will ensure exactly-once semantics.

Even without batching messages, you should very easily be able to handle exactly once semantics for ~1000 messages/sec.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/6ddc527e-11c0-4849-9c06-fb0bb8d6895a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

Exactly once delivery is very hard to implement properly (actually it's impossible) and it's not an area I would like Vert.x to go into - it's a huge can of worms.

In my view, in most cases it's unnecessary. See http://www.infoq.com/articles/no-reliable-messaging

[Tim Fox]

+1

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/C4851103-9A6B-424D-ACE6-186CDF9F914A%40gmail.com.