Accessing kafka from outside kubernetes

[george...@gmail.com]

Hello everybody,

I've setup a 3 node kafka cluster running on kubernetes using stateful sets and headless services.
The problem is now I want read/write data from/to the cluster from outside the cluster.
I have found a couple of suggestions here https://stackoverflow.com/questions/41868161/kafka-in-kubernetes-cluster-how-to-publish-consume-messages-from-outside-of-kub.

Essentially, I've found 2 options:
1. Use hostNetwork: true when starting the kafka container and use as ADVERTISED_HOST in kafka the kubernetes worker node - this has
2. Use Services type NodePort to expose the kafka broker port to the outside world - drawback here is that it will open a port on all worker nodes and second the request will go through the kube proxy

On top of this I still need to sync my external dns system with the dns of the kafka worker nodes.

Could anyone share their experience on how they achieved this?

Thanks,
George

[Brandon Philips]

Find a Kubernetes Pod by Label Selector and Port-forward Locally

Given a namespace "your-namespace" and a label query that identifies the pods you are interested in connect to a particular pod instance. If the label selector doesn't find a unique pod it will connect to the first pod by name. Ensure you replace 8080 with your pod's port.

kubectl -n your-namespace get pods -n your-namespace -l run=hello-world -o jsonpath='{.items[1].metadata.name}' | xargs -I{} kubectl -n your-namespace port-forward {} 8080

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[george...@gmail.com]

Hi Brandon,

Thanks for the response.
Kafka is publishing a message to a topic in two-step process:
a. Connect to a broker which is a pod running on a kubernetes worker and get metadata about the topic i.e. who is the leader of the topics partitions (info stored in zookeeper)
b. Connect to that leader and start sending data to it

Say I would use port forwarding to solve phase a. However, for phase b I would need to have some port opened on the worker node where the kafka leader container is running. Since I want my publisher to connect directly to the worker node running the kafka leader.

On top of that I see that the portforward.go mentions something about a "upgraded HTTP request" for the communication. I want to be able to connect on the TCP level to the kafka node.

One approach from my side was to use nodePort to open a port on the nodes where kafka is running. However, this seems not to work properly see https://github.com/kubernetes/kubernetes/issues/23920.

Thanks,
George

[Brandon Philips]

HostPort should be fixed in CNI and should be fixed-fixed in Kubernetes 1.7: https://github.com/kubernetes/kubernetes/issues/23920#issuecomment-299535316

NodePort should work though, are you experiencing issues with that?

Brandon

[george...@gmail.com]

Hi Brandon,

Thanks for the link! Good to hear that it will be fixed in version 1.7. I will try out the alpha version.

NodePort is working fine, the only drawback of it for my use case is that it opens a port on all kubernetes nodes.

Thanks for the help!

[Brandon Philips]

Try the source ip preservation beta feature: https://github.com/kubernetes/kubernetes/issues/35758

I have no idea where the official docs for this is though. SIG Networking?

[Bowei Du]

The docs for the GA feature should be available with the 1.7 release (feature went GA this release).

It looks like there is a section discussing the feature via annotations here:

https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer

Bowei

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.

[Brandon Philips]

Thanks Bowei!

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.

For more options, visit https://groups.google.com/d/optout.

--

[lucian...@gmail.com]

hello,
how did you solved it, i`m having the same issue
thanks