Knative eventing multi-tenancy

[Andrey Stelmashenko]

Hi, Community! Guys, I have a question related knative-eventing multi-tenancy. It is said that eventing component supports multi-tenancy, but it's not easy to understand how to proper use it in multi-tenant env.
I've searched through slack and found few discussions and this document https://github.com/lionelvillard/eventing/blob/0620b6f0e55d722c31c2663d260a520bdbca8f5d/docs/multitenancy.md
I'm looking at kn-eventing: kn broker/channelr+kafka. Kafka does not supprt multi-tenancy, and they achieve multi-tenancy using multi-tenant clusters of kafka.

We are looking at kafka as to message bus with delivery guarantees only (not interested in kafka as storage and stream processing framework).

Goal: have multi-tenant eventing with delivery guarantees

Multi-tenancy definition in this context: functions/services of different tenants can exchange messages, physically can not exchange messages with other tenants functions.

My current understanding is to have setup like this:
knative-eventing k8s NS (shared):

• kn components (serving, eventing)
• kn kafka-components (kafka controller, broker-dispatcher, broker-receiver)

kafka k8s NS (shared):

• kafka cluster

tenant-X k8s NS (per-tenant):

• brokers
• kafka-channels
• functions

Questions:

1) Is it possible to have single kafka cluster for all tenants? 
     My concerns here is how channels are mapped to kafka, are there possibility of name conflicts etc?
2) I'm wondering if it is possible to have shared kafka broker and achieve multi-tenancy on cloudevents level (source and type properties)?

From antoineco and slinkydeveloper answers:

> broker maps to kafka topic

So this means that broker is tenant specific and we should create many brokers in a specific k8s NS

> You can use the extension partitionkey to set up the key of the kafka record, doing that you'll enable parallelism and ordering of your messages

this means that it is possible to use shared (between tenants) brokers and use different triggers for routing, right?

k8s NS gives us separation of different tenants entities.

Please point me to discusstions/examples/docs if there are any existing or let's discuss the topic here.
Thank you.

[Francesco Guardiani]

> Is it possible to have single kafka cluster for all tenants?

It's possible and topic names are assigned using the channel name, hence they'll be unique (at least in theory)

> You can use the extension partitionkey to set up the key of the kafka record, doing that you'll enable parallelism and ordering of your messages

this means that it is possible to use shared (between tenants) brokers and use different triggers for routing, right?

I'm not sure I get the question here. When you create a broker in knative eventing, using mtbroker + kafka channel combination, you get 2 topics. These topics will live somewhere in your kafka cluster, but it's not specified where. You can configure the number of partitions to use, but the assignment of such partitions is transparent and not k8s aware at all. In other words, if you have 4 kafka channel dispatcher pods, they'll just get partitions assigned in a "random way" (it's not really random, but to you it seems random).

The partitionkey feature makes sure that you can get ordering within the same partition. To understand it, think about kafka channel: when you use a kafka channel "alone" one topic is created, with the number of partitions chosen by you. When send a record to the kafka channel, using the partitionkey as record key, the kafka producer will set the partition. If you always use the same partitionkey "abc" for all the records, and you send events sequentially, all the events will be pushed in order to the same partition, hence you achieve ordering.

When it comes down to routing messages in knative eventing, you need to use the trigger filter feature and not reason in terms of kafka.

> can you share more details about the broker using 2 channels?

the mtbroker creates, for each broker, a channel for the ingress and a channel for the filter:

ingress channel -> broker ingress -> filter channel -> broker filter

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/1c790b4b-7822-4974-85e7-285648f45897n%40googlegroups.com.

--

Francesco Guardiani

Software Engineer

Red Hat Srl

fgua...@redhat.com