"The target server failed to respond"

[Xpliant Admin]

Hi-
I'm trying to use JetS3t's synchronize program to backup my company's
files on Amazon S3. Unfortunately I'm running into an error on a
fairly consistent basis. It seems to happen more frequently when I'm
uploading a lot of data. Here's a sample of my script:

JETS3T_DIR=/home/myaccount/GridServers/jets3t-0.9.0/bin
SYNC_PROPS=${JETS3T_DIR}/synchronize.properties
sh ${JETS3T_DIR}/synchronize.sh -p --properties ${SYNC_PROPS} UP
mybucket/weekly/ /mnt/users/

Here's the error I get:

--------------------------------------------
ERROR [org.jets3t.service.multi.ThreadedStorageService
$ThreadGroupManager] A thread failed with an exception. Firing ERROR
event and cancelling all threads
org.jets3t.service.ServiceException: Request Error:
org.apache.http.NoHttpResponseException: The target server failed to
respond
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:
574)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:
281)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:
1043)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:
1862)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:
1791)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:
1782)
at
org.jets3t.service.StorageService.putObject(StorageService.java:813)
at org.jets3t.service.multi.ThreadedStorageService
$CreateObjectRunnable.run(ThreadedStorageService.java:1332)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.http.NoHttpResponseException: The target server
failed to respond
at
org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:
101)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:
252)

[ ... ]

Exception in thread "main" org.jets3t.service.ServiceException:
Request Error: org.apache.http.NoHttpResponseException: The target
server failed to respond
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:
574)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:
281)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:
1043)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:
1862)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:
1791)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:
1782)
at
org.jets3t.service.StorageService.putObject(StorageService.java:813)
at org.jets3t.service.multi.ThreadedStorageService
$CreateObjectRunnable.run(ThreadedStorageService.java:1332)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.http.NoHttpResponseException: The target server
failed to respond
at
org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:
101)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:
252)

[ ... ]
--------------------------------------------

About my environment: I'm running Fedora Linux with Java version
"1.6.0_24", OpenJDK Runtime Environment (IcedTea6 1.11.1)
(fedora-65.1.11.1.fc16-x86_64), OpenJDK 64-Bit Server VM (build 20.0-
b12, mixed mode). I do not set CLASSPATH or JAVA_HOME in my
environment. My machine is behind a firewall, but I've never noticed
any problems due to that. The file "synchronize.properties" contains
only my accesskey and secretkey.

Hopefully I'm not doing something bone-headed like failing to set the
number of retries. I'd really appreciate any help.

Thanks,
- Pete

[James Murty]

Hi Pete,

Given the error you're seeing it's most likely a glitchy firewall or
proxy getting in the way between you and S3.

By default, JetS3t will retry an operation 5 times before giving up,
although it may not consider the NoHttpResponseException to be a
retryable/recoverable exception. You could find out for sure by
increasing the logging level (see the log4j.properties file).

But in any event it's most likely your firewall that's the problem.
Maybe try throttling the upload (with the "httpclient.read-throttle"
setting [1]) to see if it's falling over due to too much traffic.

James

[1]: http://www.jets3t.org/toolkit/configuration.html#jets3t

> --
> You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
> To post to this group, send email to jets3t...@googlegroups.com.
> To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.
>

[Xpliant Admin]

Hi James-

Thanks for the help.  I modified jets3t.properties so that httpclient.read-throttle=300 (which I think should be 2.4 Mbps).  I also changed the following in log4j.properties:

log4j.logger.org.apache.commons.httpclient.HttpMethodDirector=WARN

log4j.logger.org.jets3t=WARN

Now I'm getting the following error:

ERROR [org.jets3t.service.multi.ThreadedStorageService$ThreadGroupManager] A thread failed with an exception. Firing ERROR event and cancelling all threads
org.jets3t.service.ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>JhrBugtH0fE+JmkbXaNhVw==</ExpectedDigest><CalculatedDigest>ElIm3m7uWNPV3HAyQXF+tw==</CalculatedDigest><RequestId>6628B62F1DE70F10</RequestId><HostId>W2Tgg+4QkzyF9KHfSUlSoQLdGVkv/gIt2eQnzLw4IATVJvEBd5RPmJzZ+KsSFQSz</HostId></Error>
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:427)

        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1043)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1862)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1791)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1782)
        at org.jets3t.service.StorageService.putObject(StorageService.java:813)
        at org.jets3t.service.multi.ThreadedStorageService$CreateObjectRunnable.run(ThreadedStorageService.java:1332)
        at java.lang.Thread.run(Thread.java:679)

Exception in thread "main" org.jets3t.service.ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>JhrBugtH0fE+JmkbXaNhVw==</ExpectedDigest><CalculatedDigest>ElIm3m7uWNPV3HAyQXF+tw==</CalculatedDigest><RequestId>6628B62F1DE70F10</RequestId><HostId>W2Tgg+4QkzyF9KHfSUlSoQLdGVkv/gIt2eQnzLw4IATVJvEBd5RPmJzZ+KsSFQSz</HostId></Error>
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:427)

        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1043)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1862)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1791)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1782)
        at org.jets3t.service.StorageService.putObject(StorageService.java:813)
        at org.jets3t.service.multi.ThreadedStorageService$CreateObjectRunnable.run(ThreadedStorageService.java:1332)
        at java.lang.Thread.run(Thread.java:679)

This looks like a different sort of error from what I previously saw.  Still, I'll try my test program again with httpclient.read-throttle=100.  Any thoughts?

TIA,
 - Pete

[James Murty]

Hi Pete,

That error is a worrying one because it means that the data content
being received by S3 does not exactly match the data JetS3t thinks it
is sending.

JetS3t generates an md5 hash based on the data it's about to send and
includes it as the Content-MD5 header included with the upload to S3.
When S3 receives the data, it verifies that the data it is about to
store hashes to this same value. Any discrepancy indicates data
corruption, resulting in S3 returning an error and refusing to store
the object.

This problem could be caused by relatively trivial interference by
your firewall/proxy, such as addition of HTTP headers that aren't set
by JetS3t. Or it could indicate corruption of the actual file data
being sent to S3. Either way it will not be possible to upload data
safely to S3 until you've discovered exactly what is going on. You
could disable Content-MD5 verification of uploaded data but that would
be a risky move.

Is there any way you can route S3-bound traffic through your firewall
with minimal interference?

James

> --
> You received this message because you are subscribed to the Google Groups
> "JetS3t Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/jets3t-users/-/9_nvuToJJGAJ.

[Pete DiMarco]

Hi James-

First, thanks for all the help.  I finally got the folks in charge of our firewall to turn off almost everything.  Unfortunately I'm still getting the same error message (BadDigest).  This might be a dumb question, but does synchronize.sh assume that the files it's been told to upload won't be modified while it's running?  I'm passing a big folder (roughly 26,000 files) to synchronize and I'm not doing any file locking or anything like that.  It's possible that a file could get changed while synchronize runs.

TIA,

 - Pete

[James Murty]

Hi Pete,

Yes, I'm afraid Synchronize can fail if files it is working on change
during the sync process. This will certainly be a problem if a file is
altered while it is being uploaded, as it should be since such a file
is unlikely to contain coherent data, but can also occur in the time
between files being queued for upload and the upload commencing.

You might be able to reduce the occurrence of these kinds of problems
by using batch mode, though at best this would reduce the number of
failures rather than preventing them.

Ideally you should avoid synchronizing files that are subject to
modification. If this isn't possible you could try setting the
"threaded-service.ignore-exceptions-in-multi" property in
jets3t.properties to at least get through an initial synch process
despite upload failures. Running with this switch isn't a great option
long-term, however, because you won't be notified of failures unless
you take the effort to read through the log after each sync.

James

[ppa...@odesk.com]

Hello everyone,

I also get the first error mentioned in this thread in a consistent manner. I tried to throttle down the speed even to 100kb/s but nothing changed. My setup is fully deployed in Amazon AWS inside our company's VPC. Im trying to upload jars from the continuous integration server to an Amazon S3 bucket. So, i cannot really expect to find a solution that has to do with some proxy. 

This error does not lead the code to retry based on the retries mechanism. We changed the code to do so. But then we get the same error whatever the number of retries we set. It seems that we get into some inconsistent state and we don't seem to find a solution out of this.

Any help? 

[James Murty]

Hi,

Sorry for the delayed response to this message but I don't have any
great insight on it and was hoping someone else might weigh in.

The most likely explanation I have found via some Googling is this one
[1], where stale connections may sometimes not be recognised and
recovered by HttpClient automatically. The recommended work-around
there is to do some brute-force connection cleanup per section 2.5 of
[2], could you try this and see if it helps -- or seems likely to
help?

If something like this is necessary to improve JetS3t's reliability we
could add it to the toolkit, despite it seeming like a bit of an ugly
hack.

James

[1]: http://stackoverflow.com/questions/10558791/apache-httpclient-interim-error-nohttpresponseexception
[2]: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e659

---
jamesmurty.com

> --
> You received this message because you are subscribed to the Google Groups
> "JetS3t Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to jets3t-users...@googlegroups.com.

>
> To post to this group, send email to jets3t...@googlegroups.com.

> Visit this group at http://groups.google.com/group/jets3t-users.
> For more options, visit https://groups.google.com/groups/opt_out.

[ppa...@odesk.com]

Hello,

We think we found 2 solutions to the problem. The first, is a hack into code which we consider pushing back as a pull request, in the near future. The second is using parameters to practically disable multiple thread usage, as this seemed to be a complexity we do not need. So using:

httpclient.max-connections=1

threaded-service.max-thread-count=1

threaded-service.admin-max-thread-count=1

seems to solve the issue. Given the setup in the Amazon company cloud VPC that i mentioned before, upload speed is blazingly fast without the need to use multiple threads. Of course, i can't say about other setups.

In any case, a deployment setup where a CI server in cloud tries to publish artifacts into an S3 bucket in the cloud, seems to be a rather well-established approach. If this problem persists to all these situations, then a solution is urgent and necessary.

On Wednesday, June 6, 2012 9:47:06 PM UTC+3, Xpliant Admin wrote: