Producing and consuming dynamic kafka topics

[Can çobanoğlu]

Hi all,

I am trying to build "topic per device" structure in Lagom. What is the proper way to do this. ?  In the examples, it is easy to setup in descriptor like below

.withTopics(
        topic("device-events", this::deviceEvents)
                // Kafka partitions messages, messages within the same partition will
                // be delivered in order, to ensure that all messages for the same user
                // go to the same partition (and hence are delivered in order with respect
                // to that user), we configure a partition key strategy that extracts the
                // name as the partition key.
                .withProperty(KafkaProperties.partitionKeyStrategy(), DeviceEvent::getDeviceId)
)

What if i create topics with this style : "device-events-{deviceId}".  And what is the correct way to design subscribers to consume incoming data from these topics. 

Is this approach suitable or not ? I would like to hear your insights.

Thanks. 

[Tim Moore]

Hi Can,

Apologies for not responding sooner. Dynamic topics aren't really supported by Lagom, and it isn't a very common or recommended pattern for Kafka as far as I know. Here's some information from Jay Kreps (the creator of Kafka) that is related: https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka/answer/Jay-Kreps

I'm assuming that what you're trying to do is create a way for a device to read its events back without seeing events for other devices, is that right?

Jay's response says: "I believe what these people are really looking for is key-value store with range-scans like Cassandra." This is what we recommend as well. You can have a consumer of the consolidated device-events topic in your service that writes the events (perhaps in some transformed form) into Cassandra using Lagom's CassandraSession. Then, you can define a service call that queries Cassandra for events by device ID, and use that as the endpoint for your devices.

Because Kafka is an at-least-once system in this scenario, you'll need to be sure you can de-duplicate the events in the stream when writing to Cassandra. Some kind of unique event ID as part of the primary key is a good idea.

Cheers,

Tim

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/49acd449-9665-47d9-a00d-7c3dbfa1c128%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tim Moore

Senior Engineer, Lagom, Lightbend, Inc.

tim....@lightbend.com
+61 420 981 589
Skype: timothy.m.moore

[Can çobanoğlu]

Hi Tim,

Thanks for your response. Firstly, after i read the quora answer that you shared with me has helped me and now i think, creating topic per device is not a good idea. 

I'm assuming that what you're trying to do is create a way for a device to read its events back without seeing events for other devices, is that right?

 

 Actually, let's assume that there are lots of different types of devices and all devices sensor data should come into the cloud stream processing point via kafka cluster (frankly, first station is a gateway and it is responsible for transferring data from Mqtt to Kafka) with the proper data form. The first thing stream processing point (a service that is responsible for ingesting incoming raw data) has to do is resolving which device is sending data and redirecting this incoming sensor data to related virtual device actor (which is here a lagom's persistent entity of a device)

Long story short, it is not my first concern to separate device data while consuming.

26 Temmuz 2017 Çarşamba 07:19:53 UTC+3 tarihinde Tim Moore yazdı:

Hi Can,

Apologies for not responding sooner. Dynamic topics aren't really supported by Lagom, and it isn't a very common or recommended pattern for Kafka as far as I know. Here's some information from Jay Kreps (the creator of Kafka) that is related: https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka/answer/Jay-Kreps

I'm assuming that what you're trying to do is create a way for a device to read its events back without seeing events for other devices, is that right?

Jay's response says: "I believe what these people are really looking for is key-value store with range-scans like Cassandra." This is what we recommend as well. You can have a consumer of the consolidated device-events topic in your service that writes the events (perhaps in some transformed form) into Cassandra using Lagom's CassandraSession. Then, you can define a service call that queries Cassandra for events by device ID, and use that as the endpoint for your devices.

Because Kafka is an at-least-once system in this scenario, you'll need to be sure you can de-duplicate the events in the stream when writing to Cassandra. Some kind of unique event ID as part of the primary key is a good idea.

Cheers,

Tim

On Mon, Jul 17, 2017 at 9:55 PM, Can çobanoğlu <cancob...@gmail.com> wrote:

Hi all,

I am trying to build "topic per device" structure in Lagom. What is the proper way to do this. ?  In the examples, it is easy to setup in descriptor like below

.withTopics(
        topic("device-events", this::deviceEvents)
                // Kafka partitions messages, messages within the same partition will
                // be delivered in order, to ensure that all messages for the same user
                // go to the same partition (and hence are delivered in order with respect
                // to that user), we configure a partition key strategy that extracts the
                // name as the partition key.
                .withProperty(KafkaProperties.partitionKeyStrategy(), DeviceEvent::getDeviceId)
)

What if i create topics with this style : "device-events-{deviceId}".  And what is the correct way to design subscribers to consume incoming data from these topics. 

Is this approach suitable or not ? I would like to hear your insights.

Thanks. 

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/49acd449-9665-47d9-a00d-7c3dbfa1c128%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.