Having a issue with kafka service in lagom

830 views

Skip to first unread message

jayaram

unread,

Jan 11, 2018, 6:31:55 AM1/11/18

to Lagom Framework Users

Hi Team,

we want to deploy lagom services with cassandra and kafka,But while deploying into kubernetes we are getting the below error message for kafka

Error message:
akkaSource=akka.tcp://orders...@orderservice-0.orderservice...svc.cluster.local:2551/system/sharding/kafkaProducer-order-events/singleton/singleton/producer, sourceActorSystem=orderservice] - Unable to locate Kafka service named [kafka_native]

This is the service name we are trying to connect kafka
lagom.services {
kafka_native = "tcp://kf1.mydomain.com:9092"
}

Regards
Jayaram

Tim Moore

unread,

Jan 11, 2018, 11:51:19 PM1/11/18

to jayaram, Lagom Framework Users

Hi Jayaram,

You mentioned that you're deploying to Kubernetes, but the configuration "lagom.services" is only read if you are using the ConfigurationServiceLocator implementation as your service locator. With Kubernetes, it's more typical to use the DnsServiceLocator to take advantage of Kubernetes's built-in DNS server.

Can you confirm which service locator implementation you're using?

Is your Kafka cluster also deployed with Kubenetes?

If so, the best solution is to customize the service name used to lookup the Kafka brokers to match the name of the Kafka service deployed in Kubernetes, so that you can rely on the DNS service locator to find your Kafka cluster:

lagom.broker.kafka.service-name = kafka # or whatever you have defined as the service name in k8s

You can also pass this in as an environment variable named KAFKA_SERVICE_NAME. It can be convenient to set this in your Kubernetes configuration rather than in application.conf.

Otherwise, you can configure Lagom to bypass the service locator and put the list of broker endpoints directly in your config files:

lagom.broker.kafka.service-name = '' # empty string disables service locator lookup

lagom.broker.kafka.brokers = "kf1.mydomain.com:9092" # this can be a comma-separated string if you have >1

You can find details on both of these approaches at https://www.lagomframework.com/documentation/1.4.x/java/KafkaClient.html#Configuration

Best,

Tim

On Thu, Jan 11, 2018 at 10:01 PM, jayaram <openworld...@gmail.com> wrote:

Hi Team,

we want to deploy lagom services with cassandra and kafka,But while deploying into kubernetes we are getting the below error message for kafka

Error message:

akkaSource=akka.tcp://orderservice@orderservice-0.orderservice...svc.cluster.local:2551/system/sharding/kafkaProducer-order-events/singleton/singleton/producer, sourceActorSystem=orderservice] - Unable to locate Kafka service named [kafka_native]

This is the service name we are trying to connect kafka
lagom.services {
kafka_native = "tcp://kf1.mydomain.com:9092"
}

Regards
Jayaram

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/bcca0421-8eee-4197-a380-ad208f64e172%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim Moore

Lagom Tech Lead, Lightbend, Inc.

tim....@lightbend.com
+61 420 981 589
Skype: timothy.m.moore

jayaram

unread,

Jan 12, 2018, 1:36:41 AM1/12/18

to Lagom Framework Users

Hi Tim,

Thanks for your reply we will to try configure Lagom to bypass the service locator.

By using below configuration file in application conf

lagom.broker.kafka.service-name = '' # empty string disables service locator lookup

lagom.broker.kafka.brokers = "kf1.mydomain.com:9092" # this can be a comma-separated string if you have >1

Regards,
jayaram

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/bcca0421-8eee-4197-a380-ad208f64e172%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dustin Shimono

unread,

Feb 17, 2018, 10:43:46 AM2/17/18

to Lagom Framework Users

Hi Tim,

I ran into the same issue as Jayaram and simply wanted to provide feedback about your options which worked for me and may help others.

First, this is what I had in the k8s service definition for Kafka to set the name to kafka.

{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
    "name": "kafka",
    "labels": {
      "app": "kafka",
      "component": "kafka-broker"
    }
  },

Personally I like the suggestion to set KAFKA_SERVICE_NAME in each Lagom service that uses Kafka. This is what worked for me:

{
  "name": "KAFKA_SERVICE_NAME",
  "value": "_native._tcp.kafka.default.svc.cluster.local"
},

I also tried specifying kafka service name via application.conf as lagom.broker.kafka.service-name = kafka, but this did not work for me. I continued to get the error "Unable to locate Kafka service named [kafka]"

Setting the broker property did work though, but prefer the KAFKA_SERVICE_NAME option.

lagom.broker.kafka.service-name = "" # empty string disables service locator lookup (note in java had to use double quotes, not single quotes)

lagom.broker.kafka.brokers = "kafka.mydomain.com:9092" # this can be a comma-separated string if you have >1

Appreciate the suggestions as I was able to get a Lagom system running using a Kafka service running inside k8s.

Regards,

Dustin

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/bcca0421-8eee-4197-a380-ad208f64e172%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can çobanoğlu

unread,

Mar 11, 2018, 8:22:55 AM3/11/18

to Lagom Framework Users

Hi Dustin,

I have few questions that i think you can help me.

I set up my cluster (minikube) so far i managed to complete all prior stuff but i got stuck here.

My questions as below;

- which k8s manifests are you using for Kafka setup ?

- I did not put lagom.broker.kafka.service-name and lagom.broker.kafka.brokers in my .conf file but i could see error below in my pod logs

2018-03-11T11:46:14.827Z [error] com.lightbend.lagom.internal.broker.kafka.Producer$TaggedOffsetProducerActor [sourceThread=rule-svc-akka.actor.default-dispatcher-3, akkaTimestamp=11:46:14.821UTC, akkaSource=akka.tcp://rule...@rule-svc-0.rule-svc.default.svc.cluster.local:2551/system/sharding/kafkaProducer-service.rule/io.iven.platform.rule.impl.evt.RulesEvt3/io.iven.platform.rule.impl.evt.RulesEvt3/producer, sourceActorSystem=rule-svc] - Unable to locate Kafka service named [_native._tcp.kafka.default.svc.cluster.local]

My kafka-service.yml

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    app: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    targetPort: 9092
    protocol: TCP
  selector:
    app: kafka
  type: LoadBalancer

kafka-cluster.yml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka
  template:
    metadata:
      labels:
        name: kafka
        app: kafka
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181,zoo2:2181,zoo3:2181
        - name: KAFKA_CREATE_TOPICS
          value: mytopic:2:1

zookeeper-cluster.yml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-1
      name: zookeeper-1
  template:
    metadata:
      labels:
        app: zookeeper-1
        name: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-2
      name: zookeeper-2
  template:
    metadata:
      labels:
        app: zookeeper-2
        name: zookeeper-2
    spec:
      containers:
      - name: zoo2
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "2"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-3
      name: zookeeper-3
  template:
    metadata:
      labels:
        app: zookeeper-3
        name: zookeeper-3
    spec:
      containers:
      - name: zoo3
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "3"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3

zookeeper-service.yml

apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo2
  labels:
    app: zookeeper-2
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-2
---
apiVersion: v1
kind: Service
metadata:
  name: zoo3
  labels:
    app: zookeeper-3
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-3

one of my lagom services manifest

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rule-svc
  name: rule-svc
spec:
  clusterIP: None
  ports:
  - name: http-lagom-api
    port: 9000
    protocol: TCP
    targetPort: 9000
  selector:
    app: rule-svc

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rule-svc
  name: rule-svc-akka-remoting
spec:
  ports:
  - port: 2551
    protocol: TCP
    targetPort: 2551
  selector:
    app: rule-svc


---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: rule-svc
spec:
  serviceName: rule-svc
  replicas: 1
  template:
    metadata:
      labels:
        app: rule-svc
    spec:
      containers:
      - name: rule-svc
        image: localhost:5000/platform/rule-svc
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: 512Mi
        ports:
        - containerPort: 9000
          name: http-lagom-api
        - containerPort: 2551
          name: akka-remote
        env:
        - name: APPLICATION_SECRET
          value: "ATTENTION_BE_SURE_TO_CHANGE_ME!!!!"
        - name: RULE_BIND_PORT
          value: '9000'
        - name: RULE_BIND_IP
          value: "0.0.0.0"
        - name : KAFKA_SERVICE_NAME
          value: _native._tcp.kafka.default.svc.cluster.local
        - name: CASSANDRA_SERVICE_NAME
          value: _native._tcp.cassandra.default.svc.cluster.local
        - name: AKKA_REMOTING_PORT
          value: '2551'
        - name: AKKA_REMOTING_BIND_PORT
          value: '2551'
        - name: AKKA_REMOTING_HOST
          value: "$HOSTNAME.rule-svc.default.svc.cluster.local"
        - name: AKKA_REMOTING_BIND_HOST
          value: "$HOSTNAME.rule-svc.default.svc.cluster.local"
        - name: AKKA_SEED_NODES
          value: rule-svc-0.rule-svc.default.svc.cluster.local:2551,rule-svc-1.rule-svc.default.svc.cluster.local:2551,rule-svc-2.rule-svc.default.svc.cluster.local:2551
        readinessProbe:
          httpGet:
            path: "/_status/circuit-breaker/current"
            port: 9000
          initialDelaySeconds: 5
          periodSeconds: 5