multipart upload to S3 is failing because of NoHttpResponseException

57 views

Skip to first unread message

ste...@geotoolkit.net

unread,

Apr 12, 2018, 9:55:55 AM4/12/18

to JetS3t Users

I am doing a multipart upload of a large file to Amazon S3. This is failing with a NoHttpResponseException coming from the httpclient. See below stack trace.

The file statistics are

File size: 48031541748

Part upload size: 16777216

Num parts: 2863

Apr 11, 2018 7:28:42 PM org.jets3t.service.multi.ThreadedStorageService$ThreadGroupManager run

SEVERE: A thread failed with an exception. Firing ERROR event and cancelling all threads

org.jets3t.service.S3ServiceException: Request Error: test-2018-04-08-1650-steven-shell-test-1.s3.amazonaws.com:443 failed to respond

at org.jets3t.service.impl.rest.httpclient.RestS3Service.multipartUploadPartImpl(RestS3Service.java:843)

at org.jets3t.service.S3Service.multipartUploadPart(S3Service.java:3686)

at org.jets3t.service.multi.s3.ThreadedS3Service$MultipartUploadObjectRunnable.run(ThreadedS3Service.java:395)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.http.NoHttpResponseException: test-2018-04-08-1650-steven-shell-test-1.s3.amazonaws.com:443 failed to respond

at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)

at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)

at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)

at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)

at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257)

at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:230)

at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)

at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:684)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)

at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:328)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1157)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1968)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1889)

at org.jets3t.service.impl.rest.httpclient.RestS3Service.multipartUploadPartImpl(RestS3Service.java:837)

... 3 more

When I turn on debug logging, it appears this error is not being retried? Here is a later run with debug logging and with threaded-service.ignore-exceptions-in-multi set to true

(log4j properties set to log4j.category.org.jets3t.service.impl.rest.httpclient.RestS3Service=DEBUG)

1226250 [Thread-70] DEBUG org.jets3t.service.impl.rest.httpclient.RestS3Service - Including AWS session token in request: x-amz-security-token=FQoxxxx

1226750 [Thread-71] DEBUG org.jets3t.service.impl.rest.httpclient.RestS3Service - Uploading multipart part data with Content-Length: 16777216

1226751 [Thread-71] DEBUG org.jets3t.service.impl.rest.httpclient.RestS3Service - Including AWS session token in request: x-amz-security-token=FQoxxx

1231756 [main] WARN org.jets3t.service.multi.ThreadedStorageService$ThreadGroupManager - Ignoring exception (property threaded-service.ignore-exceptions-in-multi is set to true)