Kafka Connect ACL exception

[Binoy Thomas]

I'm using Kafka Connect in standalone mode using a JDBC source connector

the SSL Principal is CN=localhost.connect.com, OU=Tech, O=Pieces Tech, L=Dallas, ST=TX, C=US

The current ACL I have specified for it is as follows

/usr/local/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=localhost.connect.com,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US --operation Write --operation Describe --topic ds_FlowsheetResult-pieces2-thr

/usr/local/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list

Current ACLs for resource `Topic:ds_FlowsheetResult-pieces2-thr`:

  User:CN=localhost.connect.com,OU=Tech,O=Pieces has Allow permission for operations: Write from hosts: *

User:CN=localhost.connect.com,OU=Tech,O=Pieces has Allow permission for operations: Describe from hosts: *

However I keep getting the below exception in kafka-authorizer.log

[2017-06-30 16:21:59,230] DEBUG Principal = User:CN=localhost.connect.com,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US is Denied Operation = Describe from host = 127.0.0.1 on resource = Topic:ds_FlowsheetResult-pieces2-thr (kafka.authorizer.logger)

and in the connect logs

[2017-06-30 16:40:50,964] WARN Error while fetching metadata with correlation id 31 : {ds_FlowsheetResult-pieces2-thr=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:707)

[Binoy Thomas]

Hello - 

I wanted to follow up on this to see if I could get help on it...

Essentially wanted Kafka Connect (as a producer) to solely have permission to write to the topic 'ds_FlowsheetResult-pieces2-thr'

[Randall Hauch]

See the documentation for the list of ACLs: http://docs.confluent.io/current/kafka/authorization.html

Note that the DESCRIBE operation is required to get topic metadata, which all producers (including those that Kafka Connect uses) need in order to figure out which broker to send writes for each topic partition.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/8986ae86-ee28-4e62-8fa8-d9f1622479a1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Binoy Thomas]

Thanks for the follow up Randall. I think my confusion is although I have provided authorization for the DESCRIBE operation (initial post). The exception seems to indicate that that operation is denied or am I missing something here.

On Friday, June 30, 2017 at 4:47:37 PM UTC-5, Binoy Thomas wrote:

[Binoy Thomas]

On Friday, June 30, 2017 at 4:47:37 PM UTC-5, Binoy Thomas wrote:

[Randall Hauch]

Yeah, I see that now. Sorry about that.

What other information is included with the error? The "Error while fetching metadata with correlation id ..." is reported by Kafka Connect's producer when no topic exists yet, but if you have the broker set up to auto-create topics then it will do that and the producer will then be able to write to the topic. So you can ignore that message if that is the only problem.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/78713c3a-e492-4320-94b2-d2fd54f51c92%40googlegroups.com.

[Binoy Thomas]

All the topics are created prior to starting any components. However with the ACL setup nothing gets written to the topic.

The only errors I see in the kafka-authorizer.log is 

[2017-07-06 12:26:21,682] DEBUG Principal = User:CN=localhost.connect.com,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US is Denied Operation = Describe from host = 127.0.0.1 on resource = Topic:ds_FlowsheetResult-pieces2-thr (kafka.authorizer.logger)

And in the connect console

[2017-07-06 12:26:23,874] WARN Error while fetching metadata with correlation id 2500 : {ds_FlowsheetResult-pieces2-thr=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:707)

[2017-07-06 12:26:23,925] ERROR Failed to flush WorkerSourceTask{id=ds_FlowsheetResult-pieces2-thr-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:304)

[2017-07-06 12:26:23,925] ERROR Failed to commit offsets for WorkerSourceTask{id=ds_FlowsheetResult-pieces2-thr-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112)

[2017-07-06 12:26:23,979] WARN Error while fetching metadata with correlation id 2501 : {ds_FlowsheetResult-pieces2-thr=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:707)

Just to give a little background on the local setup. 

I have the Kafka Broker & Schema Registry with Owner: CN=localhost, OU=Tech, O=Pieces Tech, L=Dallas, ST=TX, C=US

Below is the SSL / ACL config in server.properties

############################# SSL Configs #############################

ssl.keystore.location=/Users/binoy.thomas/keystores/kafka.server.keystore.jks

ssl.keystore.password=password

ssl.key.password=password

#ssl.truststore.location=/Users/binoy.thomas/truststores/kafka.server.truststore.jks

ssl.truststore.location=/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/security/cacerts

#ssl.truststore.password=password

ssl.truststore.password=changeit

ssl.client.auth=required

security.inter.broker.protocol=SSL

# enable hostname verification

#ssl.endpoint.identification.algorithm=HTTPS

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

############################# ACL Configs #############################

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

#allow.everyone.if.no.acl.found=true

super.users=User:CN=localhost,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US

#####################################################################

ACL created for the brokers and sort of inherited by the Schema Registry as they have the same CN is below

/usr/local/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=localhost,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US --operation ALL --topic * --group * --cluster *

ACL for Kafka Connect 

/usr/local/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=localhost.connect.com,OU=Tech,O=Pieces Tech,L=Dallas,ST=TX,C=US --operation Write --operation Describe --topic ds_FlowsheetResult-pieces2-thr

To post to this group, send email to confluent...@googlegroups.com.

[Binoy Thomas]

Hello...

Wanted to circle back to see if I could get help this issue....