Checking partition of message in Kafka Streams

[Tianxiang Xiong]

In our Kafka Streams app (call it MyApp), we want to make sure that producers put messages on the right partitions. That is, given a message [K, V], we want to check that the partition of [K, V] is the same as that calculated with our Partitioner (let's say it's DefaultPartitioner), and that the producer isn't using a partitioning strategy different from what MyApp expects.

Is there a way to validate this in the Kafka Streams app after sourcing a topic as a KStream? From what I can see, when applying a filter to the KStream, we get access to K and V, but not the partition that [K, V] was on.

Some example code would be very helpful.

[Michael Noll]

The DSL does not expose partition information for an incoming record, but the Processor API does via the `ProcessorContext`, which is being updated for each incoming record.

That is, within `Processor#process()`, you have access to a `ProcessorContext` instance that you can use to retrieve information such as the topic, partition, and offset of the current record (e.g. `ProcessorContext#partition()` and `ProcessorContext#offset()`).

Hope this helps!

Michael

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

[Michael Noll]

Some pointers:

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/Processor.html#process-K-V-

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/ProcessorContext.html

On Thu, Feb 16, 2017 at 4:03 PM, Michael Noll <mic...@confluent.io> wrote:

The DSL does not expose partition information for an incoming record, but the Processor API does via the `ProcessorContext`, which is being updated for each incoming record.

That is, within `Processor#process()`, you have access to a `ProcessorContext` instance that you can use to retrieve information such as the topic, partition, and offset of the current record (e.g. `ProcessorContext#partition()` and `ProcessorContext#offset()`).

Hope this helps!

Michael

On Thu, Feb 16, 2017 at 2:11 PM, 'Tianxiang Xiong' via Confluent Platform <confluent-platform@googlegroups.com> wrote:

In our Kafka Streams app (call it MyApp), we want to make sure that producers put messages on the right partitions. That is, given a message [K, V], we want to check that the partition of [K, V] is the same as that calculated with our Partitioner (let's say it's DefaultPartitioner), and that the producer isn't using a partitioning strategy different from what MyApp expects.

Is there a way to validate this in the Kafka Streams app after sourcing a topic as a KStream? From what I can see, when applying a filter to the KStream, we get access to K and V, but not the partition that [K, V] was on.

Some example code would be very helpful.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

[Michael Noll]

And a few code pointers:

https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/MixAndMatchLambdaIntegrationTest.java (which shows how to implement a `Processor`, and also how to integrate with the DSL)

[Tianxiang Xiong]

That's interesting, thanks.

A related issue: how is the Cluster for the partition method of DefaultPartitioner exposed?

On Thursday, 16 February 2017 16:09:51 UTC-8, Michael Noll wrote:

And a few code pointers:

https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/MixAndMatchLambdaIntegrationTest.java (which shows how to implement a `Processor`, and also how to integrate with the DSL)

On Thu, Feb 16, 2017 at 4:05 PM, Michael Noll <mic...@confluent.io> wrote:

Some pointers:

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/Processor.html#process-K-V-

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/ProcessorContext.html

On Thu, Feb 16, 2017 at 4:03 PM, Michael Noll <mic...@confluent.io> wrote:

The DSL does not expose partition information for an incoming record, but the Processor API does via the `ProcessorContext`, which is being updated for each incoming record.

That is, within `Processor#process()`, you have access to a `ProcessorContext` instance that you can use to retrieve information such as the topic, partition, and offset of the current record (e.g. `ProcessorContext#partition()` and `ProcessorContext#offset()`).

Hope this helps!

Michael

On Thu, Feb 16, 2017 at 2:11 PM, 'Tianxiang Xiong' via Confluent Platform <confluent...@googlegroups.com> wrote:

In our Kafka Streams app (call it MyApp), we want to make sure that producers put messages on the right partitions. That is, given a message [K, V], we want to check that the partition of [K, V] is the same as that calculated with our Partitioner (let's say it's DefaultPartitioner), and that the producer isn't using a partitioning strategy different from what MyApp expects.

Is there a way to validate this in the Kafka Streams app after sourcing a topic as a KStream? From what I can see, when applying a filter to the KStream, we get access to K and V, but not the partition that [K, V] was on.

Some example code would be very helpful.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

[Michael Noll]

AFAIK the `Cluster` instance (one of the input params of `DefaultPartitioner#partition()`) isn't exposed.

Would you need this information, and if so for which use cases?

-Michael

On Thu, Feb 16, 2017 at 5:24 PM, 'Tianxiang Xiong' via Confluent Platform <confluent...@googlegroups.com> wrote:

That's interesting, thanks.

A related issue: how is the Cluster for the partition method of DefaultPartitioner exposed?

On Thursday, 16 February 2017 16:09:51 UTC-8, Michael Noll wrote:

And a few code pointers:

https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/MixAndMatchLambdaIntegrationTest.java (which shows how to implement a `Processor`, and also how to integrate with the DSL)

On Thu, Feb 16, 2017 at 4:05 PM, Michael Noll <mic...@confluent.io> wrote:

Some pointers:

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/Processor.html#process-K-V-

http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/ProcessorContext.html

On Thu, Feb 16, 2017 at 4:03 PM, Michael Noll <mic...@confluent.io> wrote:

The DSL does not expose partition information for an incoming record, but the Processor API does via the `ProcessorContext`, which is being updated for each incoming record.

That is, within `Processor#process()`, you have access to a `ProcessorContext` instance that you can use to retrieve information such as the topic, partition, and offset of the current record (e.g. `ProcessorContext#partition()` and `ProcessorContext#offset()`).

Hope this helps!

Michael

On Thu, Feb 16, 2017 at 2:11 PM, 'Tianxiang Xiong' via Confluent Platform <confluent...@googlegroups.com> wrote:

In our Kafka Streams app (call it MyApp), we want to make sure that producers put messages on the right partitions. That is, given a message [K, V], we want to check that the partition of [K, V] is the same as that calculated with our Partitioner (let's say it's DefaultPartitioner), and that the producer isn't using a partitioning strategy different from what MyApp expects.

Is there a way to validate this in the Kafka Streams app after sourcing a topic as a KStream? From what I can see, when applying a filter to the KStream, we get access to K and V, but not the partition that [K, V] was on.

Some example code would be very helpful.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tianxiang Xiong]

I want to check that messages being fed into MyApp have been sent to the right partition. 

My understanding is that producers can send messages to whichever partition they like, so if a producer P uses a different Partitioner from the one MyApp expects, we need to know about it and tell the people who wrote P to fix it.

The approach I'm taking is:

- Create a transform node to serve as the first node of MyApp's topology

- Get the partition of each incoming message using the ProcessorContext

- Calculate the expected partition for the message using the Partitioner for MyApp

- Compare the two partitions and filter out messages that aren't on the right partition

That's interesting, thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Matthias J. Sax]

I guess this should work.

However, I am not sure if this is the best approach. A Streams app (from
my point of view) should be not depend on the partitioning strategy of
external producers. Maybe you want to reconsider the design of your app.

The only assumption Streams follows is, that data of input topics is
partitions by key.

Usually, if producers apply a custom partitioning strategy, there is a
good reason for this -- and it seems not to be a good idea to dictate
the partitioning strategy. Furthermore, the might be other consumers for
the same input topics. You might impact those applications, too.

For your specific case, ie, you really cannot redesign you app and need
a specific partitioning strategy, I am wondering if it might be a better
design, if you pay the cost of an extra topic that repartitions the data
as you want it to be partitioned.

Ie, instead of your transform node that does all those checks, you can
simply do a #through() with your own partitioner for each input topic
before doing the actual Streams computation.


-Matthias

> <http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/Processor.html#process-K-V->
> http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/ProcessorContext.html

> <http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/processor/ProcessorContext.html>
>
> On Thu, Feb 16, 2017 at 4:03 PM, Michael Noll
> <mic...@confluent.io> wrote:
>
> The DSL does not expose partition information for an
> incoming record, but the Processor API does via the
> `ProcessorContext`, which is being updated for each
> incoming record.
>
> That is, within `Processor#process()`, you have
> access to a `ProcessorContext` instance that you can
> use to retrieve information such as the topic,
> partition, and offset of the current record (e.g.
> `ProcessorContext#partition()` and
> `ProcessorContext#offset()`).
>
> Hope this helps!
> Michael
>
>
>
>
> On Thu, Feb 16, 2017 at 2:11 PM, 'Tianxiang Xiong'
> via Confluent Platform
> <confluent...@googlegroups.com> wrote:
>
> In our Kafka Streams app (call it MyApp), we
> want to make sure that producers put messages on
> the right partitions. That is, given a message
> [K, V], we want to check that the partition of
> [K, V] is the same as that calculated with

> our /Partitioner/ (let's say
> it's /DefaultPartitioner/), and that the

> producer isn't using a partitioning strategy
> different from what MyApp expects.
>
> Is there a way to validate this in the Kafka
> Streams app after sourcing a topic as a

> /KStream/? From what I can see, when applying a
> /filter/ to the /KStream/, we get access to K

> and V, but not the partition that [K, V] was on.
>
> Some example code would be very helpful.
>
> --
> You received this message because you are
> subscribed to the Google Groups "Confluent
> Platform" group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to

> confluent-platf...@googlegroups.com.

> To post to this group, send email to
> confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com

> <https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> Follow us: Twitter
> <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>
>
>
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> Follow us: Twitter
> <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>
>
>
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>

>
> --
> You received this message because you are subscribed to the
> Google Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to

> confluent-platf...@googlegroups.com <javascript:>.

> To post to this group, send email to

> confluent...@googlegroups.com <javascript:>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com

> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tianxiang Xiong]

Messages of input topics being partitioned by key is exactly what we want.

However, as I understand it, producers can send any message to any partition; there's no enforcement of producers sending messages with the same keys to the same partition. E.g. producer P1 can send message <K1, V1> to partition PT1, while producer P2 can send message <K1, V2> to partition PT2.

Suppose we then start two instances of MyApp; the first instance consumes PT1, the second PT2. Then neither will get all messages with key K1, which is a problem.

>                         confluent-platform+unsub...@googlegroups.com.

>         confluent-platform+unsub...@googlegroups.com <javascript:>.

>         To post to this group, send email to
>         confluent...@googlegroups.com <javascript:>.
>         To view this discussion on the web visit
>         https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com
>         <https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>         For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>.
>
>
>
>
>     --
>     *Michael G. Noll*
>     Product Manager | Confluent
>     +1 650 453 5860 | @miguno <https://twitter.com/miguno>
>     Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
>     <http://www.confluent.io/blog>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Ian Wrigley]

Producers will always send messages with the same key to the same partition, unless you

a) Override the standard Partitioner (which uses the hashcode of the key to determine which Partition to write to, or

b) Change the number of Partitions in the topic.

Or, I guess, c) explicitly write to a specific Partition in the producer.send() call.

If you do none of these things, then you can absolutely reply on all the messages with the same key being in the same Partition.

Ian.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5b42f0f0-702a-4e92-9765-8e1eb1aef9b4%40googlegroups.com.

[Tianxiang Xiong]

Right, that makes sense; however, your app would have to trust that all producers use the same partitioner that the app expects (which is probably an OK assumption).

If your app wants to ensure that messages with the same key are on the same partition, however, it seems that Matthias's suggestion would work--use a through with a StreamPartitioner. If we simply want the default partitioning strategy, this seems even easier--just create a StreamPartitioner that always returns null.

Ian.

[Matthias J. Sax]

I see. Thus, your application does not have any special requirements
(what is good!). You just worry about the general requirement that
records are partitions correctly (what is a Streams requirement that
holds for all Streams applications).

Overall I agree that the co-location requirement is somewhat strong.
However, so far nobody reported any issue with this assumption. I guess
the reason is that even if input topics are partitioned differently,
multiple things need to occur at the same time to break a Streams
application.

1. a random partitioning is used, what is rarely the case (even for
custom partitioner, usually a key-base partitioning strategy is chosen
-- just a different once than the default)
1(b). if multiple producer write to the same topic using different
custom partitioners (but if this is the case, you overall system setup
is broken most likely anyway -- for streams this is basically the same
case as if a random partitioning is done)

2. multiple topics must be consumed by the same KStream (even if this
is possible, in most cases, a single topic is consumed)

3(a). if multiple topics are consumed, the number of partitions is the
same over all topics (it's very likely that different topics do have a
different number of partitions, and this will trigger an internal
repartitioning before a key-based operation anyway)

or
3(b). if multiple topics are consumer with the same number of
partitions, but different partitioning strategies are use for different
topics (cf. 1(b))

4. you apply an key-based operation on a KStream (like agg or join)
using the original topic key -- in most application the key is set
within your Streams app anyway, and thus data repartitioning will happen
before the key-based operator


Thus, overall it's quite unlikely that this happens. Last but not least,
if you write your application, you should know in the first place what
topics (and their properties) you consume. And it seems rather unlikely,
that those properties change later on (why should they?)



-Matthias

On 2/19/17 2:48 PM, 'Tianxiang Xiong' via Confluent Platform wrote:
> Right, that makes sense; however, your app would have to trust that all
> producers use the same partitioner that the app expects (which is
> probably an OK assumption).
>

> If your app wants to /ensure/ that messages with the same key are on the

> same partition, however, it seems that Matthias's suggestion would

> work--use a /through/ with a /StreamPartitioner/. If we simply want the

> default partitioning strategy, this seems even easier--just create a

> /StreamPartitioner/ that always returns null.

>
> On Sunday, 19 February 2017 14:12:47 UTC-8, Ian Wrigley wrote:
>
> Producers will always send messages with the same key to the same
> partition, unless you
> a) Override the standard Partitioner (which uses the hashcode of the
> key to determine which Partition to write to, or
> b) Change the number of Partitions in the topic.
>
> Or, I guess, c) explicitly write to a specific Partition in the
> producer.send() call.
>
> If you do none of these things, then you can absolutely reply on all
> the messages with the same key being in the same Partition.
>
> Ian.
>
>> On Feb 19, 2017, at 4:09 PM, Tianxiang Xiong <tianxia...@gmail.com

>> confluent-platf...@googlegroups.com.

>> > To post to this group, send email to
>> > confluent...@googlegroups.com.
>> > To view this discussion on the web
>> visit
>> >
>> https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com
>> <https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com>
>>
>> >

>> <https://groups.google.com/d/msgid/confluent-platform/9e4b8f98-15f4-49da-88b0-d938d10e7849%40googlegroups.com?utm_medium=email&utm_source=footer

>> <https://twitter.com/ConfluentInc>> | Blog
>> > <http://www.confluent.io/blog>
>> >
>> > --
>> > You received this message because you are subscribed
>> to the
>> > Google Groups "Confluent Platform" group.
>> > To unsubscribe from this group and stop receiving
>> emails from
>> > it, send an email to

>> > confluent-platf...@googlegroups.com

>> <javascript:>.
>> > To post to this group, send email to
>> > confluent...@googlegroups.com <javascript:>.
>> > To view this discussion on the web visit
>> >
>> https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com
>> <https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com>
>>
>> >

>> <https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com?utm_medium=email&utm_source=footer

>> <https://groups.google.com/d/msgid/confluent-platform/07e846ab-d563-4759-864e-40855db4c460%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>>
>> >
>> > For more options, visit
>> https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>

>> > <https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>>.
>> >
>> >
>> >
>> >
>> > --
>> > *Michael G. Noll*
>> > Product Manager | Confluent
>> > +1 650 453 5860 | @miguno <https://twitter.com/miguno>
>> > Follow us: Twitter <https://twitter.com/ConfluentInc

>> <https://twitter.com/ConfluentInc>> | Blog
>> > <http://www.confluent.io/blog>
>> >
>> > --
>> > You received this message because you are subscribed to the
>> Google
>> > Groups "Confluent Platform" group.
>> > To unsubscribe from this group and stop receiving emails
>> from it, send

>> > an email to confluent-platf...@googlegroups.com
>> > <mailto:confluent-platf...@googlegroups.com>.

>> > To post to this group, send email to
>> confluent...@googlegroups.com
>> > <mailto:confluent...@googlegroups.com>.
>> > To view this discussion on the web visit
>> >
>> https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com
>> <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com>
>>
>> >

>> <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com?utm_medium=email&utm_source=footer
>> <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>>

>> > For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>>
>>
>> --

>> You received this message because you are subscribed to the Google
>> Groups "Confluent Platform" group.
>> To unsubscribe from this group and stop receiving emails from it,

>> send an email to confluent-platf...@googlegroups.com

>> <javascript:>.
>> To post to this group, send email to confluent...@googlegroups.com
>> <javascript:>.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/confluent-platform/5b42f0f0-702a-4e92-9765-8e1eb1aef9b4%40googlegroups.com
>> <https://groups.google.com/d/msgid/confluent-platform/5b42f0f0-702a-4e92-9765-8e1eb1aef9b4%40googlegroups.com?utm_medium=email&utm_source=footer>.

>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
> --

> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/27aa713c-58e3-41e4-b649-d3afa058ac7b%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/27aa713c-58e3-41e4-b649-d3afa058ac7b%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tianxiang Xiong]

You just worry about the general requirement that records are partitions correctly (what is a Streams requirement that holds for all Streams applications).

That's right, but it seems a bad idea for MyApp to trust (and not verify) that this is the case when ingesting an input topic, since apps don't control how input topics are partitioned.

It's interesting that TopologyBuilder#addSink has an option to specify a `StreamPartitioner`, but `addSource` doesn't. Maybe taking a `StreamPartitioner` in `addSource` would be a solution to this problem?

2. multiple topics must be consumed by the same KStream (even if this 
is possible, in most cases, a single topic is consumed) 

Why only multiple topics? A single topic that does not use a key-based partitioning strategy would have the same problem, right?

3(a). if multiple topics are consumed, the number of partitions is the 
same over all topics (it's very likely that different topics do have a 
different number of partitions, and this will trigger an internal 
repartitioning before a key-based operation anyway) 

Don't input topics need to have the same number of partitions? I recall something about that in a discussion with Damien. If you have topics T1 and T2, where T1 has M partitions and T2 has N, M != N, can they both be input topics to MyApp?

>>         confluent-platform+unsub...@googlegroups.com.

>>         >         confluent-platform+unsub...@googlegroups.com

>>         > an email to confluent-platform+unsub...@googlegroups.com
>>         > <mailto:confluent-platform+unsub...@googlegroups.com>.

>>         > To post to this group, send email to
>>         confluent...@googlegroups.com
>>         > <mailto:confluent...@googlegroups.com>.
>>         > To view this discussion on the web visit
>>         >
>>         https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com
>>         <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com>
>>
>>         >
>>         <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com?utm_medium=email&utm_source=footer
>>         <https://groups.google.com/d/msgid/confluent-platform/d5aee37d-73dd-4774-8ed9-6e97933a1f50%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>>
>>         > For more options, visit https://groups.google.com/d/optout
>>         <https://groups.google.com/d/optout>.
>>
>>
>>     --
>>     You received this message because you are subscribed to the Google
>>     Groups "Confluent Platform" group.
>>     To unsubscribe from this group and stop receiving emails from it,

>>     send an email to confluent-platform+unsub...@googlegroups.com

>>     <javascript:>.
>>     To post to this group, send email to confluent...@googlegroups.com
>>     <javascript:>.
>>     To view this discussion on the web visit
>>     https://groups.google.com/d/msgid/confluent-platform/5b42f0f0-702a-4e92-9765-8e1eb1aef9b4%40googlegroups.com
>>     <https://groups.google.com/d/msgid/confluent-platform/5b42f0f0-702a-4e92-9765-8e1eb1aef9b4%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>     For more options, visit https://groups.google.com/d/optout
>>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.