Signature issue
235 views
Skip to first unread message
matthieu Labour
Feb 25, 2010, 11:02:14 AM2/25/10
to jets3t...@googlegroups.com
Hi
Not sure if this is a jets3t issue but would appreciate any tips:
My program was inserting documents into S3 and all of sudden, during the night it started continuously failing with the following errors . (I replaced content with BLABLABLA). I restarted it and now it works fine
Thank you !
[2010-02-25 13:39:18] [Thread-27] WARN (RestS3Service.java:411) - Error Response: PUT 'BLABLABLABLABLA' -- ResponseCode: 307, ResponseStatus: Temporary Redirect, Request Headers: [Content-Length: 1020, Content-MD5: mw0aODkQMSw0HxZ2NfSBnA==, x-amz-meta-md5-hash: 9b0d1a383910312c341f167635f4819c, Content-Type: text/plain; charset=utf-8, Date: Thu, 25 Feb 2010 13:39:18 GMT, Authorization: AWS AKIAJVMWT7S24UXBFGIA:0gvKOV3irL5NfZASs4JUkFDydRo=, User-Agent: JetS3t/0.7.1 (Linux/2.6.27-23-xen; i386; en; JVM 1.5.0_20), Host: XXXXXXX.s3.amazonaws.com, Expect: 100-continue], Response Headers: [x-amz-request-id: D42ED41E6FBDB3F3, x-amz-id-2: JEkr7+3z5TllrNeioU6QMzrzhWjXLfJp3YyHW0zwIWtZ7lC1VS3FuH/q+jNrQc/i, Location: https://XXXXXXX.s3-us-west-1.amazonaws.com/BLABLABLA, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Thu, 25 Feb 2010 13:39:18 GMT, Connection: close, Server: AmazonS3]
[2010-02-25 13:39:19] [Thread-27] WARN (RestS3Service.java:411) - Error Response: PUT 'BLABLABLABLA -- ResponseCode: 403, ResponseStatus: Forbidden, Request Headers: [Content-Length: 1020, Content-MD5: mw0aODkQMSw0HxZ2NfSBnA==, x-amz-meta-md5-hash: 9b0d1a383910312c341f167635f4819c, Content-Type: text/plain; charset=utf-8, User-Agent: JetS3t/0.7.1 (Linux/2.6.27-23-xen; i386; en; JVM 1.5.0_20), Expect: 100-continue, Date: Thu, 25 Feb 2010 13:39:18 GMT, Authorization: AWS AKIAJVMWT7S24UXBFGIA:Nenz96iTR6BTs7Vh5rpExX5u8KQ=, Host: XXXXXX.s3-us-west-1.amazonaws.com], Response Headers: [x-amz-request-id: A2F27D2D0FF48EFA, x-amz-id-2: TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Thu, 25 Feb 2010 13:39:18 GMT, nnCoection: close, Server: AmazonS3]
[2010-02-25 13:39:19] [Thread-27] ERROR(RestS3Service.java:571) - Request Failed.
org.jets3t.service.S3ServiceException: S3 Error Message. PUT 'BLABLABLA' on Host 'ixb-us-kikin-com.s3-us-west-1.amazonaws.com' @ 'Thu, 25 Feb 2010 13:39:18 GMT' -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 6d 77 30 61 4f 44 6b 51 4d 53 77 30 48 78 5a 32 4e 66 53 42 6e 41 3d 3d 0a 74 65 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 75 74 66 2d 38 0a 54 68 75 2c 20 32 35 20 46 65 62 20 32 30 31 30 20 31 33 3a 33 39 3a 31 38 20 47 4d 54 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6d 64 35 2d 68 61 73 68 3a 39 62 30 64 31 61 33 38 33 39 31 30 33 31 32 63 33 34 31 66 31 36 37 36 33 35 66 34 38 31 39 63 0a 2f 69 78 62 2d 75 73 2d 6b 69 6b 69 6e 2d 63 6f 6d 2f 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 69 64 25 33 41 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 66 61 63 65 62 6f 6f 6b 2d 66 72 69 65 6e 64 25 32 30 6b 69 6b 69 6e 75 73 65 72 69 64 25 33 41 31 6a 63 74 66 71 6f 4f 62 43 54 2e 39 4f 33 31 38 73 30 6c 42 76 69 2e 32 4f 52 73 52 58 25 32 30 70 6f 73 74 65 72 5f 69 64 25 33 41 31 36 30 33 37 32 30 32 32 37 25 32 30</StringToSignBytes><RequestId>XXXXXXXX</RequestId><HostId>TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga</HostId><SignatureProvided>Nenz96iTR6BTs7Vh5rpExX5u8KQ=</SignatureProvided><StringToSign>PUTmw0aODkQMSw0HxZ2NfSBnA==text/plain; charset=utf-8Thu, 25 Feb 2010 13:39:18 GMTx-amz-meta-md5-hash:9b0d1a383910312c341f167635f4819c/BKLABLABLA</StringToSign><AWSAccessKeyId>XXXXXXXXXX</AWSAccessKeyId></Error>
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:441)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:924)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1510)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1441)
at org.jets3t.service.S3Service.putObject(S3Service.java:1804)
at org.jets3t.service.S3Service.putObject(S3Service.java:2057)
at kikin.kbp.service.common.docmanager.AWSS3DocManager.addDocument(AWSS3DocManager.java:202)
at kikin.kbp.service.common.index.DocumentConverter.transform(DocumentConverter.java:68)
at kikin.kbp.service.common.index.DocumentConverter.transform(DocumentConverter.java:27)
at kikin.kbp.service.common.transformer.QueueTransformer.run(QueueTransformer.java:36)
[2010-02-25 13:39:19] [Thread-27] ERROR(DocumentConverter.java:72) - org.jets3t.service.S3ServiceException: S3 Error Message. PUT '/BLABLABLBA' on Host 'XXXXX.s3-us-west-1.amazonaws.com' @ 'Thu, 25 Feb 2010 13:39:18 GMT' -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 6d 77 30 61 4f 44 6b 51 4d 53 77 30 48 78 5a 32 4e 66 53 42 6e 41 3d 3d 0a 74 65 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 75 74 66 2d 38 0a 54 68 75 2c 20 32 35 20 46 65 62 20 32 30 31 30 20 31 33 3a 33 39 3a 31 38 20 47 4d 54 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6d 64 35 2d 68 61 73 68 3a 39 62 30 64 31 61 33 38 33 39 31 30 33 31 32 63 33 34 31 66 31 36 37 36 33 35 66 34 38 31 39 63 0a 2f 69 78 62 2d 75 73 2d 6b 69 6b 69 6e 2d 63 6f 6d 2f 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 69 64 25 33 41 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 66 61 63 65 62 6f 6f 6b 2d 66 72 69 65 6e 64 25 32 30 6b 69 6b 69 6e 75 73 65 72 69 64 25 33 41 31 6a 63 74 66 71 6f 4f 62 43 54 2e 39 4f 33 31 38 73 30 6c 42 76 69 2e 32 4f 52 73 52 58 25 32 30 70 6f 73 74 65 72 5f 69 64 25 33 41 31 36 30 33 37 32 30 32 32 37 25 32 30</StringToSignBytes><RequestId>A2F27D2D0FF48EFA</RequestId><HostId>TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga</HostId><SignatureProvided>Nenz96iTR6BTs7Vh5rpExX5u8KQ=</SignatureProvided><StringToSign>PUTmw0aODkQMSw0HxZ2NfSBnA==text/plain; charset=utf-8Thu, 25 Feb 2010 13:39:18 GMTx-amz-meta-md5-hash:9b0d1a383910312c341f167635f4819c/BLABLABLA</StringToSign><AWSAccessKeyId>XXXXXXX</AWSAccessKeyId></Error>
Not sure if this is a jets3t issue but would appreciate any tips:
My program was inserting documents into S3 and all of sudden, during the night it started continuously failing with the following errors . (I replaced content with BLABLABLA). I restarted it and now it works fine
Thank you !
[2010-02-25 13:39:18] [Thread-27] WARN (RestS3Service.java:411) - Error Response: PUT 'BLABLABLABLABLA' -- ResponseCode: 307, ResponseStatus: Temporary Redirect, Request Headers: [Content-Length: 1020, Content-MD5: mw0aODkQMSw0HxZ2NfSBnA==, x-amz-meta-md5-hash: 9b0d1a383910312c341f167635f4819c, Content-Type: text/plain; charset=utf-8, Date: Thu, 25 Feb 2010 13:39:18 GMT, Authorization: AWS AKIAJVMWT7S24UXBFGIA:0gvKOV3irL5NfZASs4JUkFDydRo=, User-Agent: JetS3t/0.7.1 (Linux/2.6.27-23-xen; i386; en; JVM 1.5.0_20), Host: XXXXXXX.s3.amazonaws.com, Expect: 100-continue], Response Headers: [x-amz-request-id: D42ED41E6FBDB3F3, x-amz-id-2: JEkr7+3z5TllrNeioU6QMzrzhWjXLfJp3YyHW0zwIWtZ7lC1VS3FuH/q+jNrQc/i, Location: https://XXXXXXX.s3-us-west-1.amazonaws.com/BLABLABLA, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Thu, 25 Feb 2010 13:39:18 GMT, Connection: close, Server: AmazonS3]
[2010-02-25 13:39:19] [Thread-27] WARN (RestS3Service.java:411) - Error Response: PUT 'BLABLABLABLA -- ResponseCode: 403, ResponseStatus: Forbidden, Request Headers: [Content-Length: 1020, Content-MD5: mw0aODkQMSw0HxZ2NfSBnA==, x-amz-meta-md5-hash: 9b0d1a383910312c341f167635f4819c, Content-Type: text/plain; charset=utf-8, User-Agent: JetS3t/0.7.1 (Linux/2.6.27-23-xen; i386; en; JVM 1.5.0_20), Expect: 100-continue, Date: Thu, 25 Feb 2010 13:39:18 GMT, Authorization: AWS AKIAJVMWT7S24UXBFGIA:Nenz96iTR6BTs7Vh5rpExX5u8KQ=, Host: XXXXXX.s3-us-west-1.amazonaws.com], Response Headers: [x-amz-request-id: A2F27D2D0FF48EFA, x-amz-id-2: TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Thu, 25 Feb 2010 13:39:18 GMT, nnCoection: close, Server: AmazonS3]
[2010-02-25 13:39:19] [Thread-27] ERROR(RestS3Service.java:571) - Request Failed.
org.jets3t.service.S3ServiceException: S3 Error Message. PUT 'BLABLABLA' on Host 'ixb-us-kikin-com.s3-us-west-1.amazonaws.com' @ 'Thu, 25 Feb 2010 13:39:18 GMT' -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 6d 77 30 61 4f 44 6b 51 4d 53 77 30 48 78 5a 32 4e 66 53 42 6e 41 3d 3d 0a 74 65 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 75 74 66 2d 38 0a 54 68 75 2c 20 32 35 20 46 65 62 20 32 30 31 30 20 31 33 3a 33 39 3a 31 38 20 47 4d 54 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6d 64 35 2d 68 61 73 68 3a 39 62 30 64 31 61 33 38 33 39 31 30 33 31 32 63 33 34 31 66 31 36 37 36 33 35 66 34 38 31 39 63 0a 2f 69 78 62 2d 75 73 2d 6b 69 6b 69 6e 2d 63 6f 6d 2f 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 69 64 25 33 41 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 66 61 63 65 62 6f 6f 6b 2d 66 72 69 65 6e 64 25 32 30 6b 69 6b 69 6e 75 73 65 72 69 64 25 33 41 31 6a 63 74 66 71 6f 4f 62 43 54 2e 39 4f 33 31 38 73 30 6c 42 76 69 2e 32 4f 52 73 52 58 25 32 30 70 6f 73 74 65 72 5f 69 64 25 33 41 31 36 30 33 37 32 30 32 32 37 25 32 30</StringToSignBytes><RequestId>XXXXXXXX</RequestId><HostId>TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga</HostId><SignatureProvided>Nenz96iTR6BTs7Vh5rpExX5u8KQ=</SignatureProvided><StringToSign>PUTmw0aODkQMSw0HxZ2NfSBnA==text/plain; charset=utf-8Thu, 25 Feb 2010 13:39:18 GMTx-amz-meta-md5-hash:9b0d1a383910312c341f167635f4819c/BKLABLABLA</StringToSign><AWSAccessKeyId>XXXXXXXXXX</AWSAccessKeyId></Error>
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:441)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:924)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1510)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1441)
at org.jets3t.service.S3Service.putObject(S3Service.java:1804)
at org.jets3t.service.S3Service.putObject(S3Service.java:2057)
at kikin.kbp.service.common.docmanager.AWSS3DocManager.addDocument(AWSS3DocManager.java:202)
at kikin.kbp.service.common.index.DocumentConverter.transform(DocumentConverter.java:68)
at kikin.kbp.service.common.index.DocumentConverter.transform(DocumentConverter.java:27)
at kikin.kbp.service.common.transformer.QueueTransformer.run(QueueTransformer.java:36)
[2010-02-25 13:39:19] [Thread-27] ERROR(DocumentConverter.java:72) - org.jets3t.service.S3ServiceException: S3 Error Message. PUT '/BLABLABLBA' on Host 'XXXXX.s3-us-west-1.amazonaws.com' @ 'Thu, 25 Feb 2010 13:39:18 GMT' -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 6d 77 30 61 4f 44 6b 51 4d 53 77 30 48 78 5a 32 4e 66 53 42 6e 41 3d 3d 0a 74 65 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 75 74 66 2d 38 0a 54 68 75 2c 20 32 35 20 46 65 62 20 32 30 31 30 20 31 33 3a 33 39 3a 31 38 20 47 4d 54 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6d 64 35 2d 68 61 73 68 3a 39 62 30 64 31 61 33 38 33 39 31 30 33 31 32 63 33 34 31 66 31 36 37 36 33 35 66 34 38 31 39 63 0a 2f 69 78 62 2d 75 73 2d 6b 69 6b 69 6e 2d 63 6f 6d 2f 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 69 64 25 33 41 6b 69 6b 69 6e 2e 70 61 72 74 6e 65 72 2e 66 61 63 65 62 6f 6f 6b 2d 66 72 69 65 6e 64 25 32 30 6b 69 6b 69 6e 75 73 65 72 69 64 25 33 41 31 6a 63 74 66 71 6f 4f 62 43 54 2e 39 4f 33 31 38 73 30 6c 42 76 69 2e 32 4f 52 73 52 58 25 32 30 70 6f 73 74 65 72 5f 69 64 25 33 41 31 36 30 33 37 32 30 32 32 37 25 32 30</StringToSignBytes><RequestId>A2F27D2D0FF48EFA</RequestId><HostId>TE7gtQizLgZlTdEIkfNx5Du8edvv0vCtjG5S7T3MNx8LcmGWk6eVBoKoo2fvNPga</HostId><SignatureProvided>Nenz96iTR6BTs7Vh5rpExX5u8KQ=</SignatureProvided><StringToSign>PUTmw0aODkQMSw0HxZ2NfSBnA==text/plain; charset=utf-8Thu, 25 Feb 2010 13:39:18 GMTx-amz-meta-md5-hash:9b0d1a383910312c341f167635f4819c/BLABLABLA</StringToSign><AWSAccessKeyId>XXXXXXX</AWSAccessKeyId></Error>
James Murty
Feb 26, 2010, 1:01:37 AM2/26/10
to jets3t...@googlegroups.com
Hi,
I don't see any obvious explanation in the stack trace.
Given it was a persistent problem for the app, but was solved by a restart, one possible explanation is that your requests were being routed via a malfunctioning gateway on the 'net or a failing S3 endpoint. Restarting the app caused you to get a new endpoint S3 IP or a different route to S3. It is very difficult to debug these kinds of transient issues after the fact.
It is also possible that Jets3t started misbehaving though I don't know what would cause it to generate invalid signatures.
James
--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To post to this group, send email to jets3t...@googlegroups.com.
To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.
Spiral
Apr 14, 2010, 6:54:54 PM4/14/10
to JetS3t Users
I am seeing something similar. It is sporadic but can sometimes
continue for quite awhile.
continue for quite awhile.
James Murty
Apr 14, 2010, 8:54:45 PM4/14/10
to jets3t...@googlegroups.com
When this signing error occurs does it resolve itself, or does it only start working after you restart your JetS3t-based app?
On Wed, Apr 14, 2010 at 3:54 PM, Spiral <jmr...@gmail.com> wrote:
I am seeing something similar. It is sporadic but can sometimes
continue for quite awhile.
ethan
May 12, 2010, 11:30:02 AM5/12/10
to JetS3t Users
I am currently experiencing this problem. I recently upgraded to 7.3
from 7.1. I made this move so I could use the getOrCreateBucket()
method with the EU location (I'm uploading from the US). Also 7.1
getOrCreateBucket() didn't pick up the default location I set in my
properties files.
Anyhow I have 4 threads uploading to the same bucket. The first one
after 00:00 GMT ends up creating the bucket with the
getOrCreateBucket(). This thread gets the 307 redirect then 403
signature fail. The other three threads upload to the new bucket fine
and continue doing work. However the thread that got the 307->403
errors coninues to fail and does not appear to recover. I can fix this
problem by bouncing my app.
It makes sense to me that if I'm redirected the signatures wouldn't
match but why doesn't this just fail once and recover. What in jets3t
is "holding" on to this temporary state. Why do I continue to get
307's?
If I can fix this I'm probably just going to move the bucket creation
to a separate workflow and have it create tomorrow's today and just be
done with this problem.
James Murty
May 12, 2010, 9:42:17 PM5/12/10
to jets3t...@googlegroups.com
Thanks for the detailed bug description, this is the first solid lead I've had for this problem.
I have created a bug ticket for the issue here:
Please follow the ticket to receive updates, and feel free to add anything else that may help debug the problem such as more description or stack traces.
Cheers,
James
Reply all
Reply to author
Forward
0 new messages