AWS S3 Sink Connector - SignatureDoesNotMatch S3 Exception During Upload though creds exist

[Alex Silva]

Hi there,

I'm using the latest version of the Confluent Open Source and have an S3-Sink configured and loaded, with messages writing to the `mysql-jdbc-ms-user-user` topic. 

Below is my s3-sink config.

name=s3-sink

connector.class=io.confluent.connect.s3.S3SinkConnector

tasks.max=1

topics=mysql-jdbc-ms-user-user

s3.region=us-east-1

s3.bucket.name=confluent-kafka-connect-s3-testing-2

s3.part.size=5242880

flush.size=3

storage.class=io.confluent.connect.s3.storage.S3Storage

format.class=io.confluent.connect.s3.format.avro.AvroFormat

schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator

partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner

schema.compatibility=BACKWARD

#partition.field.name=

partition.duration.ms=1000

path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH/

locale=en

timezone=UTC

I know that my aws creds are configured properly because I'm able to upload files to my buckets via the CLI. For example, this succeeds:

aws s3 cp source_init.sql s3://confluent-kafka-connect-s3-testing-2

Now, with my S3-sink loaded, and after adding new messages to the topic, I see this error in my connect log (`confluent log connect`):

[2017-09-29 09:19:08,008] INFO (Re-)joining group connect-s3-sink (org.apache.kafka.clients.consumer.internals.AbstractCoordi

nator:432)

[2017-09-29 09:19:08,013] INFO Successfully joined group connect-s3-sink with generation 11 (org.apache.kafka.clients.consume

r.internals.AbstractCoordinator:399)

[2017-09-29 09:19:08,013] INFO Setting newly assigned partitions [mysql-jdbc-ms-user-user-0] for group connect-s3-sink (org.a

pache.kafka.clients.consumer.internals.ConsumerCoordinator:262)

[2017-09-29 09:19:08,019] INFO Opening record writer for: topics/mysql-jdbc-ms-user-user/year=2017/month=09/day=29/hour=13//m

ysql-jdbc-ms-user-user+0+0000000000.avro (io.confluent.connect.s3.format.avro.AvroRecordWriterProvider:67)

[2017-09-29 09:19:08,021] INFO Starting commit and rotation for topic partition mysql-jdbc-ms-user-user-0 with start offset {

year=2017/month=09/day=29/hour=13/=0} (io.confluent.connect.s3.TopicPartitionWriter:207)

[2017-09-29 09:19:08,278] INFO Source task WorkerSourceTask{id=jdbc-source-0} finished initialization and start (org.apache.k

afka.connect.runtime.WorkerSourceTask:143)

[2017-09-29 09:19:08,450] ERROR Multipart upload failed to complete for bucket 'wmdev-glue' key 'topics/mysql-jdbc-ms-user-us

er/year=2017/month=09/day=29/hour=13//mysql-jdbc-ms-user-user+0+0000000000.avro' (io.confluent.connect.s3.storage.S3OutputStr

eam:137)

[2017-09-29 09:19:08,450] ERROR Task s3-sink-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtim

e.WorkerSinkTask:455)

org.apache.kafka.connect.errors.DataException: Multipart upload failed to complete.

        at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:138)

        at io.confluent.connect.s3.format.avro.AvroRecordWriterProvider$1.commit(AvroRecordWriterProvider.java:91)

        at io.confluent.connect.s3.TopicPartitionWriter.commitFile(TopicPartitionWriter.java:428)

        at io.confluent.connect.s3.TopicPartitionWriter.commitFiles(TopicPartitionWriter.java:408)

        at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:215)

        at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:180)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:435)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:251)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:180)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148)

        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)

        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Unable to initiate MultipartUpload: com.amazonaws.services.s3.model.AmazonS3Exception: The re

quest signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon 

S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 70E5FFB1A9113286), S3 Extended Request ID: s+Yk0YAD25TKd

OS+veVUMA/oKZWGy5JSIraX101iVbW0Xc5eI1wMV6I4wpFGIIonEk/PJkmFv/4=

        at io.confluent.connect.s3.storage.S3OutputStream.newMultipartUpload(S3OutputStream.java:174)

        at io.confluent.connect.s3.storage.S3OutputStream.uploadPart(S3OutputStream.java:109)

        at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:132)

        ... 16 more

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signatur

e you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; 

Request ID: 70E5FFB1A9113286), S3 Extended Request ID: s+Yk0YAD25TKdOS+veVUMA/oKZWGy5JSIraX101iVbW0Xc5eI1wMV6I4wpFGIIonEk/PJk

mFv/4=

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672)

        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654)

        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518)

        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4185)

        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4132)

        at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3017)

        at io.confluent.connect.s3.storage.S3OutputStream.newMultipartUpload(S3OutputStream.java:170)

        ... 18 more

[2017-09-29 09:19:08,451] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask:456)

[2017-09-29 09:19:08,451] ERROR Task s3-sink-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:148)

org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.

        at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:457)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:251)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:180)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148)

        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)

        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

[2017-09-29 09:19:08,452] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:149)

For the life of me I can't figure out why connect isn't able to reconcile my AWS creds.

 Anyway know how to examine what AWS creds confluent connect has loaded? Or how to explicitly pass creds to the connect process?

Thanks,

Alex

[Konstantine Karantasis]

Hi Alex,

I understand configuring things like this can get frustrating at times.

The S3 connector supports a variety of credentials setups by using the default credential provider chain as described here: 

http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

(also you might want to check the connector's quickstart guide: https://docs.confluent.io/current/connect/connect-storage-cloud/kafka-connect-s3/docs/s3_connector.html )

The ways to configure your Connector's credentials have been listed also in this issue: https://github.com/confluentinc/kafka-connect-storage-cloud/issues/39

Setting the right region might also be required in your setting. 

Hope something of the above helps, 

Konstantine. 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4dad8ae0-3af1-48cc-8729-20826b2ef726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Silva]

Hey Konstantine,

Thanks for the response. I'm familiar with that AWS credentials dev guide and the Confluent S3 quickstart, both of which I've relied on to get going. Thanks for the link to the Github issue, I hadn't seen that.

However, my issue is not that the AWS credentials can't be loaded (documented in the Github issue), it's that my creds are being rejected when the connector makes the request to AWS:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

This is strange because uploading to S3 via the AWS CLI works fine. For some reason the S3 Sink can't act on my behalf properly.

Can you or anyone in the community think of any way to debug this issue? I'm pretty stumped...

Thanks again,

Alex

To post to this group, send email to confluent...@googlegroups.com.

[Alex Silva]

Hey Konstantine, 

I ended up identifying the issue here, which was a dumb error involving conflicting AWS config... I had one set of creds loading from my bash profile, and other set in my aws creds file. 

Apologies for the noise and thanks again for your help!

Thanks,

Alex

[Konstantine Karantasis]

Glad you sorted it out and you let us know what was the issue! 

Cheers, 

Konstantine

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4dad8ae0-3af1-48cc-8729-20826b2ef726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/e5c1c436-e93d-4375-9373-40b0e74fcf32%40googlegroups.com.