shared subscription and push/pull delivery

[Andrey Stelmashenko]

How to make "shared subscription" message processing type?

My current understanding about knative eventing:

- Knative uses push mechanism to deliver messages, I specify function name OR uri in the subscriber section for trigger.

- Channels/brokers respect message ordering

E.g.

Publisher(s) --push msg A, B, C--> Broker -> trigger --C, B, A--> consumer

1) Publisher pushes messages A, B, C

2) trigger sends message A to consumer

3) consumer processes A, sends ACK

4) trigger sends message B to consumer

5) consumer processes B, sends ACK

6) trigger sends message C to consumer

7) consumer processes C, sends ACK

so it is one-by-one in-order processing, correct?

Let's say I need a queue of tasks which do no require ordered processing. Publishers publish messages to be processed in parallel (e.g. round robin), and we want fire and forget scenario (submit message to durable channel/broker).

In terms of knative I see it like:

Publishers (*) --Push msgs--> KN-Broker --Push msgs--> Trigger Filter --Push msgs --> (KN-ingress --Push msg-->) KN-Function

KN serving scales functions to process messages.

How to achieve that?

What if functions scaling has reached upper limit and still messages are coming, is processing timeout and retry happens?

One more thing, kn-eventing uses push mechanism to deliver messages, is it possible to have 'pull' mechanism, to be able to apply reactive-stream processing and rely on back pressure mechanism?

Please let me know if I need to clarify something, and let's discuss this topics. 

Thank you

[Scott Nichols]

Answers inline,

On Thu, Apr 29, 2021 at 7:50 AM Andrey Stelmashenko <astelm...@viax.io> wrote:

How to make "shared subscription" message processing type?

My current understanding about knative eventing:

- Knative uses push mechanism to deliver messages, I specify function name OR uri in the subscriber section for trigger.

- Channels/brokers respect message ordering

We do not require ordered event processing. This is a trait of the underlying persistence layer you configure.

 

E.g.

Publisher(s) --push msg A, B, C--> Broker -> trigger --C, B, A--> consumer

1) Publisher pushes messages A, B, C

2) trigger sends message A to consumer

3) consumer processes A, sends ACK

4) trigger sends message B to consumer

5) consumer processes B, sends ACK

6) trigger sends message C to consumer

7) consumer processes C, sends ACK

so it is one-by-one in-order processing, correct?

This is true for some implementations of channel and broker, but I would not depend on it, what likely happens is A, B, and C are delivered around the same time on different threads depending on again, the implementation.

 

Let's say I need a queue of tasks which do no require ordered processing. Publishers publish messages to be processed in parallel (e.g. round robin), and we want fire and forget scenario (submit message to durable channel/broker).

In terms of knative I see it like:

Publishers (*) --Push msgs--> KN-Broker --Push msgs--> Trigger Filter --Push msgs --> (KN-ingress --Push msg-->) KN-Function

KN serving scales functions to process messages.

Use a Knative Serving Service as the runtime for your function, and it will scale based on load.

 

How to achieve that?

What if functions scaling has reached upper limit and still messages are coming, is processing timeout and retry happens?

I think you are asking: "What happens when we hit the max QPS of the Broker/Trigger implementation I am using?" and in that case, the queue will grow, perhaps until you run out of memory or disk to store those messages. One thing you might want to do is hold the request open between the broker and the function and only ACK when you are sure that processing will be successful. This gives you a chance to NACK and then that event will be retried. 

 

One more thing, kn-eventing uses push mechanism to deliver messages, is it possible to have 'pull' mechanism, to be able to apply reactive-stream processing and rely on back pressure mechanism?

We would like to explore pull models in the future, we are focused on getting to a 1.0 of Knative, and for eventing that means Push only, Pull will happen post-1.0 as it is another common use case.

 

Please let me know if I need to clarify something, and let's discuss this topics. 

Thank you

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/741acb87-1a7a-46c9-999a-b041e58e4d4dn%40googlegroups.com.

[Andrey Stelmashenko]

Thanks for the answers,

One point is not fully clear to me:

Let's say I need a queue of tasks which do no require ordered processing. Publishers publish messages to be processed in parallel (e.g. round robin), and we want fire and forget scenario (submit message to durable channel/broker).

In terms of knative I see it like:

Publishers (*) --Push msgs--> KN-Broker --Push msgs--> Trigger Filter --Push msgs --> (KN-ingress --Push msg-->) KN-Function

KN serving scales functions to process messages.

Use a Knative Serving Service as the runtime for your function, and it will scale based on load.

 I understand how kn-serving helps with scaling, but what about durability?

So I need two things here:

1. durable delivery, it means message is preserved until it is processed OR ttl is over
2. scaling - which is covered by kn-serving

My understanding that I need to combine kn-eventing (e.g. kafka-channel/broker) to address delivery guarantees and kn-serving to address scaling.

Is there broker/channel configuration to setup shared subscription (I use the term from pulsar docs: shared subsciption ) ?

[Scott Nichols]

On Fri, Apr 30, 2021 at 2:07 AM Andrey Stelmashenko <astelm...@viax.io> wrote:

Thanks for the answers,

One point is not fully clear to me:

Let's say I need a queue of tasks which do no require ordered processing. Publishers publish messages to be processed in parallel (e.g. round robin), and we want fire and forget scenario (submit message to durable channel/broker).

In terms of knative I see it like:

Publishers (*) --Push msgs--> KN-Broker --Push msgs--> Trigger Filter --Push msgs --> (KN-ingress --Push msg-->) KN-Function

KN serving scales functions to process messages.

Use a Knative Serving Service as the runtime for your function, and it will scale based on load.

 I understand how kn-serving helps with scaling, but what about durability?

Serving does not help, that is why we also have the Eventing side of the house (Broker/Triggers).

 

So I need two things here:

1. durable delivery, it means message is preserved until it is processed OR ttl is over

you get this on the delivery side of the broker.

 

1. scaling - which is covered by kn-serving

ack, you can use serving to scale real big.

 

My understanding that I need to combine kn-eventing (e.g. kafka-channel/broker) to address delivery guarantees and kn-serving to address scaling.

Is there broker/channel configuration to setup shared subscription (I use the term from pulsar docs: shared subsciption ) ?

Ah, interesting. No, I do not know of an implementation that would do this directly for sure, but in this case you might be more interested in the Channel/Subscription model in Eventing. There you can control how things are persisted and delivered more finely. Broker/Trigger is for eventing mesh usage, channels would be more for what you are trying to do with a pubsub like work queue. I am not 100% sure, but perhaps someone on the Kafka or AMQP side ( or even the Google Pub/Sub side) could chime in with an example work queue you could take a look at. 

Which persistence layer (messaging broker) are you targeting?

 

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/6e534f5d-df24-490d-92fd-415e06eff7cfn%40googlegroups.com.

[Andrey Stelmashenko]

> Which persistence layer (messaging broker) are you targeting?

Right now we are using pulsar (not a heavily depend on it yet). But we do not like pulsar functions, and looking at knative. Knative eventing does not support pulsar, and now we have a dilemma, either invest and add pulsar channels to knative or migrate to supported by knatvie broker, like nats or kafka.