Failures and failures

[Angel Vera]

Hi Gang,

I have been getting a lot failures lately.. and I am wondering what do you guys have to share..? What has other people experience? what are your settings...?

My job runs under jenkins and from what I can tell it hasn't run succesful since 5 months ago.. LONG TIME eh?, but it only runs every 3 days.

My settings are set to fail an upload if it tries more than 5 times. I checked a couple of the old jobs and they all seem to have died at:

WARN [org.jets3t.service.impl.rest.httpclient.RestS3Service] Retried connection 6 times, which exceeds the maximum retry count of 5

So what are your settings? maybe I have to be less restrictive..? increase the retry to 10.. but that is just too obvious. Let me know your thoughts..


Here are my settings inside of jets3t.properties:
s3service.https-only=true
s3service.max-thread-count=4
s3service.admin-max-thread-count=10
s3service.stream-retry-buffer-size=131072
s3service.internal-error-retry-max=5
s3service.disable-dns-buckets=false

httpclient.connection-timeout-ms=60000
httpclient.socket-timeout-ms=60000
httpclient.max-connections=10
httpclient.stale-checking-enabled=true
httpclient.retry-max=5
httpclient.authentication-preemptive=false
httpclient.proxy-autodetect=true

[James Murty]

Hi Angel,

It depends very much on why your S3 tasks are failing, can you provide more information on that?

Increasing the retry limit will only help if your tasks fail due to transient network or service disruptions, in which case trying again might succeed. 

If every task is failing every time something much more fundamental must be wrong. You will most likely need to find and fix that underlying problem rather than tweaking the settings. 

James

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.
To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at https://groups.google.com/group/jets3t-users.
For more options, visit https://groups.google.com/d/optout.

[Angel Vera]

Sure let me collect some more info about the failures, but at first glance it looked like all are failing at different points in time/different files. Thus my hesitant conclusion :) but let me look at it in more details.

Sent from a mobile device.

[James Murty]

If all the failures are at different points the most likely problem is that you have too many uploads/downloads for your bandwidth, causing transfers to become bandwidth starved and fail -- S3 doesn't wait long for a transfer pause before killing the connection. 

You only have 4 simultaneous threads/connections configured (max-thread-count) which doesn't seem like too many, but you could try reducing that to 2 or 3 if you have low bandwidth to see if that helps. 

[gun...@gmail.com]

[gun...@gmail.com]

So i managed to fix the problem at somepoint.. and got my sync to work for a few days.. but now I seem to be running into low memomry problems..? I have the

that is using only 2 thread count:

# S3Service properties
###
s3service.https-only=true
s3service.max-thread-count=2
s3service.admin-max-thread-count=10
s3service.stream-retry-buffer-size=131072
s3service.internal-error-retry-max=10
s3service.default-bucket-location=US
s3service.s3-endpoint=s3.amazonaws.com
s3service.disable-dns-buckets=false



and with  -Xmx512M already setup.



I am getting:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:2694)
	at java.lang.String.<init>(String.java:203)
	at java.lang.StringBuffer.toString(StringBuffer.java:561)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:317)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)
	at org.jets3t.service.utils.FileComparer.buildFileMapImpl(FileComparer.java:320)

[Angel Vera]

I just updated to the latest jets3t code base to see if the defaults settings with some of the improvements that have undergone synchronize since 0.6.1 fix the problem.. (I was on a pre/2009 version)

--
You received this message because you are subscribed to a topic in the Google Groups "JetS3t Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jets3t-users/dHD6_SEOmcA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jets3t-users+unsubscribe@googlegroups.com.

[Angel Vera]

Okay...same problem.

[Angel Vera]

Huh.. interesting...

I killed a few process and then managed to free some MEM, gave it to java making it 1280M and eventually hit this:

Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.lang.String.substring(String.java:1875)
	at org.jets3t.service.utils.FileComparer.lookupObjectMetadataForPotentialClashes(FileComparer.java:799)
	at org.jets3t.service.utils.FileComparer.buildObjectMapPartial(FileComparer.java:721)
	at org.jets3t.apps.synchronize.Synchronize.compareLocalAndRemoteFiles(Synchronize.java:317)
	at org.jets3t.apps.synchronize.Synchronize.uploadLocalDirectory(Synchronize.java:411)
	at org.jets3t.apps.synchronize.Synchronize.run(Synchronize.java:1033)
	at org.jets3t.apps.synchronize.Synchronize.main(Synchronize.java:1632)

any ideas..?

[Angel Vera]

Well this seems to be related to: https://bitbucket.org/jmurty/jets3t/issues/211/stringindexoutofboundsexception-in