Hi Mark
The version number is used to represent the last consumed event on the
node level. Sensei Gateway consumes events for the whole node and is
responsible for assigning the version. After that the pair of the
actual data event and version is routed to the appropriate partition
or discarded if it doesn't belong to the partition located on the
current node. Versions don't have to be incremental and Kafka doesn't
ensure the ordering of the incoming messages. But we use that ordering
to determine what is the biggest version persisted in Zoie, so that
if the node is restarted the gateway would be aware of it
In general it makes sense to use the Kafka offset as the version only
if we are using Kafka simple consumer, meaning that we've connected to
a single Kafka broker and are pulling messages manually from some
partitions
https://github.com/senseidb/sensei/blob/master/sensei-gateways/src/main/java/com/senseidb/gateway/kafka/SimpleKafkaStreamDataProvider.java.
High level Kafka consumer hides this from the client, and we rely
either on the Kafka autocommit interval or commitCurrentOffsets
method. That's why at
https://github.com/senseidb/sensei/blob/master/sensei-gateways/src/main/java/com/senseidb/gateway/kafka/KafkaStreamDataProvider.java
the version is just the local timestamp. The problem with the second
approach is that Kafka commit is not sync'ed with the Zoie persist
event and we might lose some data if the node crashes.
As far as I know for Kafka 0.7 and earlier message offsets might be
different on different brokers for the same partition/topic. This has
been fixed in Kafka 0.8, so we can implement the new Kafka gateway
that keeps track of offsets per partition and prevent possible data
loss
Hope this answers your question :) Just to reiterate: keeping version
as Kafka offset doesn't give us anything if we use Kafka high level
consumer api
Thanks,
Volodymyr
2013/6/24 Mark Williams <
zaido74...@yahoo.com>:
> --
> You received this message because you are subscribed to the Google Groups
> "Sensei" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to
sensei-searc...@googlegroups.com.
> For more options, visit
https://groups.google.com/groups/opt_out.
>
>