Cannot get Kafka indexing service running with Secured Kafka with Kerberos (SASL_PLAINTEXT)

[Hamit Mermerkaya]

Hi all,

I have been trying to get Kafka indexing service in imply.io running with with Secured Kafka with Kerberos (SASL_PLAINTEXT) for some time, but with no success.

I have running Kerberos secured Kafka broker, producer and consumer, they are all tested .

After setting  javaopts parameters relavant to security given in red below in conf-quickstart/druid/_common/common.runtime.properties

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager  -Djava.security.auth.login.config=/etc/kafka/kafka_client_kerberos_jaas.conf -Djava.security.krb5.conf=/etc/krb.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.kdc=server.myhostname.com -Djava.security.krb5.realm=MYREALM.COM

, I follow the steps given in tutorial kafka indexing service in imply.io and add the consumer properties ("security.protocol": "SASL_PLAINTEXT", "sasl.kerberos.service.name": "kafka" ) related to security in json file, but I  am just getting stuck in the step "curl -XPOST -H'Content-Type: application/json' -d 

@/home/kafka/kafka-task.json http://localhost:8090/druid/indexer/v1/supervisor", and receiving the following error message

<p>Problem accessing /druid/indexer/v1/supervisor. Reason: 

<pre>    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>

In overlord.log , that error  turns out to be caused by: "org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode",  although I pass this argument  in the way given above.

Could you please help me in getting  Kafka indexing service running with with Secured Kafka ?

overlord.log and kafka_task.json attached. 

yours,

Hamit

[David Lim]

Hey Hamit,

The supervisor, which lives on the overlord, also has a Kafka consumer that it uses to determine the task configuration and therefore the overlord also needs to have the Kerberos configuration. Try setting

-Djava.security.auth.login.config=/etc/kafka/kafka_client_kerberos_jaas.conf

in your overlord/jvm.config file as well.

[Hamit Mermerkaya]

Hi David,

It worked. 

Thanks for the help.

yours,

Hamit

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/W2SiPnNsy0U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d2a54758-dba5-473a-8cf5-76b54be8674f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Hamit Mermerkaya]

Hi David,

I have another question for you  regarding this issue.

I said  before it worked for the case all the users are super users.

In my case, I have two users  with the names "kafka" and  "kafka-client".

User "kafka"  is running zookeeper and kafka server in secured mode with kerberos  and user "kafka-client"  is running  only druid. 

In kafka server properties file, if they are set as super users, the druid works as I said before.

But when kafka-client is removed from the super.users with and "allow.everyone.if.no.acl.found" is set to false, and then trying to give the permission  to it with the following kafka console command

"bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=kafka1.myhostname.com:2181 --add --allow-principal User:kafka-client --operation Read  --topic test-topic  --group test-group"

after I add  "group.id": "test-group" line to json file,  I can not get it running.

Error mesage I got in the overlord.log is as follows

ERROR [KafkaSupervisor-igwcdrv1-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[igwcdrv1] failed to handle notice:
{class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class org.apache.kafka.common.errors.GroupAuthorizationException, exceptionMessage=Not aut
horized to access group: kafka-supervisor-ebjnpajj, noticeClass=RunNotice}
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: kafka-supervisor-ebjnpajj
2016-12-08T10:04:45,033 INFO [KafkaSupervisor-igwcdrv1-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-12-08T10:04:45.032Z","
service":"druid/overlord","host":"hamits:8090","severity":"component-failure","description":"KafkaSupervisor[igwcdrv1] failed to handle notice","data":{"class":"io
.druid.indexing.kafka.supervisor.KafkaSupervisor","exceptionType":"org.apache.kafka.common.errors.GroupAuthorizationException","exceptionMessage":"Not authorized to access group: kafka-supervisor-ebjnpajj","exceptionStackTrace":"org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: kafka-supervisor-ebjnpajj\n","noticeClass":"RunNotice"}}]

 

It turns out that group name I enter in json is not passed to the system. It is "kafka-supervisor-ebjnpajj" that druid itself puts 

I would like to stress out  that I don't have any problem when running kafka console consumer instead of druid after typing the command given above for read permission 

Do you have an idea about that?

Thanks in advance.

yours,

Hamit.

[David Lim]

Hey Hamit,

Unfortunately it looks like the supervisor overrides the group ID with its own as you can see here: https://github.com/druid-io/druid/blob/master/extensions-core/kafka-indexing-service/src/main/java/io/druid/indexing/kafka/supervisor/KafkaSupervisor.java#L601

You could either build the extension from source and move that line before 'props.putAll(ioConfig.getConsumerProperties());', or the alternative option would be to use a wildcard (i.e. --group *) when setting the Kafka ACLs.

I'll check in a patch for this for next release.

Hi David,

Hamit

To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

[David Lim]

See: https://github.com/druid-io/druid/pull/3765

[Hamit Mermerkaya]

Hi David,

I went with the building of extension after making changes you proposed and it worked. 

Thank again.

yours,

Hamit.

To unsubscribe from this group and all its topics, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/fee0ec66-a754-4e08-8d64-29ba0d6a77c5%40googlegroups.com.