How to use a state store outside of a stream processing thread?!

250 views
Skip to first unread message

Ryan Worsley

unread,
Sep 20, 2016, 3:06:17 AM9/20/16
to Confluent Platform
Hi!

I want to access a state store outside of a stream processing thread, but it throws a NullReferenceException because its internal 'metrics' property is null.  I can see that this is set by a call to the 'init' method which takes a ProcessorContext and a root StateStore.

My question is, is it possible to use a state store outside of a stream processing thread?  If not why not!  Is there some canonical way that I'm missing?

My application is intended to hold state that can be queried by external calls...  Surely this is an intended design?

BONUS question: Are the state stores thread-safe?

Cheers,
Ryan.

Damian Guy

unread,
Sep 20, 2016, 7:28:31 AM9/20/16
to Confluent Platform
Hi Ryan,

On Apache Kafka trunk and in the upcoming 0.10.1 release you will be able to get read-only access to the state stores via the new Interactive Queries feature (aka Queryable State), KIP-67 

On trunk the state stores are thread-safe.

Thanks,
Damian 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/b359d15e-ac44-49bd-8c8b-ec15ed962504%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ryan Worsley

unread,
Sep 20, 2016, 2:33:23 PM9/20/16
to Confluent Platform
Hi Damian,

Really grateful for your reply.  What's my best course of action?  Is there a way I can get a hold of 0.10.1 right now? (I'm in a PoC phase, so happy to test it out).  Or should I grab and build trunk?  Alternatively I think I could probably work around it by implementing a state store supplier?

Any steer you can give would be very welcome :)

Cheers,
Ryan.


On Tuesday, 20 September 2016 12:28:31 UTC+1, Damian Guy wrote:
Hi Ryan,

On Apache Kafka trunk and in the upcoming 0.10.1 release you will be able to get read-only access to the state stores via the new Interactive Queries feature (aka Queryable State), KIP-67 

On trunk the state stores are thread-safe.

Thanks,
Damian 

On Tue, 20 Sep 2016 at 08:06 Ryan Worsley <ryanth...@gmail.com> wrote:
Hi!

I want to access a state store outside of a stream processing thread, but it throws a NullReferenceException because its internal 'metrics' property is null.  I can see that this is set by a call to the 'init' method which takes a ProcessorContext and a root StateStore.

My question is, is it possible to use a state store outside of a stream processing thread?  If not why not!  Is there some canonical way that I'm missing?

My application is intended to hold state that can be queried by external calls...  Surely this is an intended design?

BONUS question: Are the state stores thread-safe?

Cheers,
Ryan.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Damian Guy

unread,
Sep 20, 2016, 2:50:22 PM9/20/16
to Confluent Platform
Hi Ryan,

You can checkout the 0.10.1 branch - it was only cut yesterday, so it is pretty close to trunk.  

Cheers,
Damian

On Tue, 20 Sep 2016 at 19:33 Ryan Worsley <ryanth...@gmail.com> wrote:
Hi Damian,

Really grateful for your reply.  What's my best course of action?  Is there a way I can get a hold of 0.10.1 right now? (I'm in a PoC phase, so happy to test it out).  Or should I grab and build trunk?  Alternatively I think I could probably work around it by implementing a state store supplier?

Any steer you can give would be very welcome :)

Cheers,
Ryan.


On Tuesday, 20 September 2016 12:28:31 UTC+1, Damian Guy wrote:
Hi Ryan,

On Apache Kafka trunk and in the upcoming 0.10.1 release you will be able to get read-only access to the state stores via the new Interactive Queries feature (aka Queryable State), KIP-67 

On trunk the state stores are thread-safe.

Thanks,
Damian 

On Tue, 20 Sep 2016 at 08:06 Ryan Worsley <ryanth...@gmail.com> wrote:
Hi!

I want to access a state store outside of a stream processing thread, but it throws a NullReferenceException because its internal 'metrics' property is null.  I can see that this is set by a call to the 'init' method which takes a ProcessorContext and a root StateStore.

My question is, is it possible to use a state store outside of a stream processing thread?  If not why not!  Is there some canonical way that I'm missing?

My application is intended to hold state that can be queried by external calls...  Surely this is an intended design?

BONUS question: Are the state stores thread-safe?

Cheers,
Ryan.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5ccda578-6d82-45e0-805e-cf11ec481e1c%40googlegroups.com.

Ryan Worsley

unread,
Sep 20, 2016, 2:52:41 PM9/20/16
to Confluent Platform
Hi Damian,

Thanks again for your quick response. I'll check it out shortly - where should I look for hints on how to use this new API?

Cheers,
Ryan.

Eno Thereska

unread,
Sep 20, 2016, 4:15:10 PM9/20/16
to Confluent Platform
Hi Ryan,

There is an example in QueryableStateIntegrationTest.java in the "shouldBeAbleToQueryState" method. In addition, there is an end-to-end example at https://github.com/confluentinc/examples/blob/master/kafka-streams/src/main/java/io/confluent/examples/streams/queryablestate/QueryableStateExample.java. 

We are working to add documentation for this feature and that should be done soon. Meanwhile, feel free to have a look at KIP-67 which has the main design and APIs, at a high level.

Thanks
Eno

Ryan Worsley

unread,
Sep 21, 2016, 2:09:48 AM9/21/16
to Confluent Platform
Thanks a lot!  Really appreciate your efforts - the timing is impeccable :)

Ryan Worsley

unread,
Sep 21, 2016, 12:56:17 PM9/21/16
to Confluent Platform
Hi Eno,

Is that QueryableStateExample checked in?  I'm getting a 404 on clicking the link and can't find it in the source?!

Thanks,
Ryan.


On Tuesday, 20 September 2016 21:15:10 UTC+1, Eno Thereska wrote:

Ryan Worsley

unread,
Sep 21, 2016, 1:00:02 PM9/21/16
to Confluent Platform

Ryan Worsley

unread,
Sep 22, 2016, 1:50:37 AM9/22/16
to Confluent Platform
Hi Damian,

I'm using the 0.10.1 branch (built locally) and I'm getting a spurious exception thrown by the schema registry (that I wasn't getting before).

Exception in thread "main" java.lang.NoClassDefFoundError: kafka/common/TopicExistsException
 at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:141)


Any ideas?


Regards,
Ryan.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Damian Guy

unread,
Sep 22, 2016, 4:45:15 AM9/22/16
to Confluent Platform
Hi Ryan,

Yes, it is because that class has moved. It is now: org.apache.kafka.common.errors.TopicsExistException. I guess you are using the 3.0.x version of schema registry? You might need to try the 3.1.0-SNAPSHOT version, as we are using here: https://github.com/confluentinc/examples/blob/master/kafka-streams/pom.xml

Thanks,
Damian


To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/60721dd5-f45c-472f-b7a2-85b488eb8b3f%40googlegroups.com.

Ryan Worsley

unread,
Sep 22, 2016, 4:48:59 AM9/22/16
to Confluent Platform
Hi Damian,

You guys are awesome - thanks!

Regards,
Ryan.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages