Kafka ACL's with SSL Protocol is not working

[Raghu B]

Hi All,

I am trying to enable ACL's in my Kafka cluster with along with SSL Protocol.

I tried with each and every parameters but no luck, so I need help to enable the SSL(without Kerberos) and I am attaching all the configuration details in this.

Kindly Help me.

I tested SSL without ACL, it worked fine (listeners=SSL://10.247.195.122:9093)

This is my Kafka server properties file:

############################# ACL SETTINGS #############################

auto.create.topics.enable=true

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

security.inter.broker.protocol=SSL

#allow.everyone.if.no.acl.found=true

#principal.builder.class=CustomizedPrincipalBuilderClass

#super.users=User:"CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown"

#super.users=User:Raghu;User:Admin

#offsets.storage=kafka

#dual.commit.enabled=true

listeners=SSL://10.247.195.122:9093

#listeners=PLAINTEXT://10.247.195.122:9092

#listeners=PLAINTEXT://10.247.195.122:9092,SSL://10.247.195.122:9093

#advertised.listeners=PLAINTEXT://10.247.195.122:9092

        ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks

        ssl.keystore.password=123456

        ssl.key.password=123456

        ssl.truststore.location=/home/raghu/kafka/security/server.truststore.jks

        ssl.truststore.password=123456

Set the ACL from Authorizer CLI:

> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=10.247.195.122:2181 --list --topic ssltopic

Current ACLs for resource `Topic:ssltopic`: 

  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown has Allow permission for operations: Write from hosts: * 

XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh --broker-list 10.247.195.122:9093 --topic ssltopic --producer.config client-ssl.properties

[2016-12-13 14:53:45,839] WARN Error while fetching metadata with correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

[2016-12-13 14:53:45,984] WARN Error while fetching metadata with correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties

#group.id=sslgroup

security.protocol=SSL

ssl.truststore.location=/Users/rbaddam/Desktop/Dev/kafka_2.11-0.10.1.0/ssl/client.truststore.jks

ssl.truststore.password=123456

#Configure Below if you use Client Auth

ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.11-0.10.1.0/ssl/client.keystore.jks

ssl.keystore.password=123456

ssl.key.password=123456

XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh --bootstrap-server 10.247.195.122:9093 --new-consumer --consumer.config client-ssl.properties --topic ssltopic --from-beginning

[2016-12-13 14:53:28,817] WARN Error while fetching metadata with correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$)

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-52826

Thanks in advance,

Raghu

[Jay]

 bin/kafka-acls.sh --authorizer-properties zookeeper.connect=10.247.195.122:2181 --list --topic ssltopic

Current ACLs for resource `Topic:ssltopic`: 

  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown has Allow permission for operations: Write from hosts: * 

It maybe cause by Read and Describe permission from the topic. You only have Write permission. Consumers need to have Read and Describe permission from topic.

[Awadhesh Kumar]

Hi Raghu,

Were you able to solve ACL issue in Kafka SSL setup? I have set both Describe & Write ACl to the topic but getting the same error.

What was the steps made to make it work?

Thanks
Awadhesh

[singh.sh...@gmail.com]

I believe you can set ACLs for Kafka topics only if you have a Kerberized kafka cluster

Kafka ACLs donot get enabled in non-kerberized Kafka cluster.

[Awadhesh Kumar]

Thanks for the update. It is nowhere mentioned that it will not work. Can you please give any reference document that says about this. I don't want to enable Kerberos authentication between broker & zookeeper thats why I am not using any kerberos authentication and keytab file setup. I want to simply use Certificate based SSL setup to enable both Authorization adn Authentication and Authentication working fine with this setup.

When I post the messages through producer app after adding ACL for producer, I am getting following error in producer

phosts >  ./kafka-console-producer.sh --broker-list phosts.awadhesh.net:11111 --topic acl.test  --producer.config /home/awadhesh/certificates/ssl/producer_ssl_acl.properties

hello

[2017-10-10 09:49:17,575] WARN Error while fetching metadata with correlation id 1 : {acl.test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

[2017-10-10 09:49:17,698] WARN Error while fetching metadata with correlation id 2 : {acl.test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

and if I see the kafka-authorizer log, it show the access allowed log as below

[2017-10-10 08:48:44,681] DEBUG operation = Describe on resource = Topic:acl.test from host = 172.99.128.41 is Allow based on acl = User:CN=phosts.awadhesh.net,OU=CLEAR,O=CLEAR,L=CLEAR,ST=CLEAR,C=US has Allow permission for operations: Describe from hosts: * (kafka.authorizer.logger)

[2017-10-10 08:48:44,681] DEBUG Principal = User:CN=phosts.awadhesh.net,OU=CLEAR,O=CLEAR,L=CLEAR,ST=CLEAR,C=US is Allowed Operation = Describe from host = 172.99.128.41 on resource = Topic:acl.test (kafka.authorizer.logger)

If I see the ACL using Kafka-acl command, I see the following response -

Current ACLs for resource `Topic:acl.test`:

User:CN=phosts.awadhesh.net,OU=CLEAR,O=CLEAR,L=CLEAR,ST=CLEAR,C=US has Allow permission for operations: Describe from hosts: *

User:CN=phosts.awadhesh.net,OU=CLEAR,O=CLEAR,L=CLEAR,ST=CLEAR,C=US has Allow permission for operations: Write from hosts: *

I want to know if I am missing any thing here or I need to setup Keytab file for Zookeeper/Broker communication setup

Thanks

[singh.sh...@gmail.com]

i'm not sure if it is mentioned explicitly ..

However, there was a post on hortonworks community that mentioned that, 

will try to find that & forward.

meanwhile, pls try to kerberize and then use ACls... it should work.

[Nitish Michael]

Hi All,

   I am trying to setup security on top of my my Topic, my question is, Is  Kerberos mandatory for Kafka authentication or only ACL is sufficient for authentication?

or we can do authentication by only and only setting up ACL. (Not adding anything for SSL authentication)

Thanks & Regards

Nitish 

[akshay sinha]

Hi Nitesh, - Did you find an answer to your question. I am also seeing a similar issue while setting up a SSL authentication and Authorization without going kerberos. Authentication works fine but authorization not... 

Thanks - Akshay

[indu...@gmail.com]

Hi ALL,
            I am setting up Kafka security  using SSL then the authentication is working fine but the authorizations is not i.,e i gave only write permission for producer on the  topic but it accept  read permission  for consumer also