S3 upload takes very long - ~20 seconds to a 3MB file

[Mia Shani]

Please help - what am I doing wrong? this is my code

           RestS3Service s3Service = new RestS3Service(credentials);

       

      // generate the file object using its key

           S3Object fileObject = new S3Object(file);

           fileObject.setBucketName(bucketName);

           fileObject.setContentLength(file.length());

           if (key != null && key.length() > 0)

          fileObject.setKey(key);

           

           fileObject = s3Service.putObject(bucketName, fileObject);

debug prints before the first line of this code and after the last line of this code show about 20 seconds difference.

I have an excellent high-speed connection, and if I upload directly to S3 via the web interface, the same file gets uploaded within about 3-4 seconds or less.

thanks

Mia

[James Murty]

Hi,

Can you confirm that it's the S3Service#putObject method specifically that is taking 20 seconds? And make sure you are not creating a new RestS3Service instance every time you do something, you should create one of these and re-use it everywhere in your application.

Aside from that, if it is indeed the upload that is running slowly, setting the property "log4j.logger.org.apache.http.headers=DEBUG" in the log4j.properties file might log useful information to find out why it is taking so long.

Hope this helps,

James

---
jamesmurty.com

Mia

--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.
To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at http://groups.google.com/group/jets3t-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mia Shani]

Thank you, James. it did help.

the actual time is much shorter (2-3 seconds) and my time measurement was off.

I think that I have to use a new RestS3Service instance every time because I am saving the files to S3 one at a time.

I am not sure it is good practice to use a single global RestS3Service instance in my case, but I will examine this option, I am sure it will improve my timing.

Mia

[Mia Shani]

Hello, James,

Sorry to be such a nuisance... I am running one my debug PC and time is a bit better, however on may production machine the timing is still terrible.

here are my logs, including the Apache debug prints:

this print is just before the call to s3Service.putObject:

timing before s3Service.putObject - bucket digilabs.galleries.4 key - 385964/0742.jpg content length - 4949311 (12:38:43)

then these are Apache logs ...

DEBUG [org.apache.http.headers] >> PUT /385964%2F0742.jpg HTTP/1.1

DEBUG [org.apache.http.headers] >> Date: Tue, 07 May 2013 19:38:43 GMT

DEBUG [org.apache.http.headers] >> Content-MD5: LoAk4JJ2vasltc9bjKk8iw==

DEBUG [org.apache.http.headers] >> Content-Type: image/jpeg

DEBUG [org.apache.http.headers] >> x-amz-meta-md5-hash: 2e8024e09276bdab25b5cf5b8ca93c8b

DEBUG [org.apache.http.headers] >> Authorization: AWS <user id/key removed for security>

DEBUG [org.apache.http.headers] >> Content-Length: 4949311

DEBUG [org.apache.http.headers] >> Host: digilabs.galleries.4.s3.amazonaws.com:443

DEBUG [org.apache.http.headers] >> Connection: Keep-Alive

DEBUG [org.apache.http.headers] >> User-Agent: JetS3t/0.9.0 (Windows 2003/5.2; x86; en; JVM 1.7.0_21)

DEBUG [org.apache.http.headers] >> Expect: 100-continue

DEBUG [org.apache.http.headers] << HTTP/1.1 100 Continue

DEBUG [org.apache.http.headers] << HTTP/1.1 200 OK

DEBUG [org.apache.http.headers] << x-amz-id-2: o/L31ov0Oa7RLNGeBnnqKsvj5FMwtZZuWVmWyeTL5zvkpBTTcxRVJMKZRAfsyC2Z

DEBUG [org.apache.http.headers] << x-amz-request-id: 858143F642961598

DEBUG [org.apache.http.headers] << Date: Tue, 07 May 2013 19:38:44 GMT

DEBUG [org.apache.http.headers] << ETag: "2e8024e09276bdab25b5cf5b8ca93c8b"

DEBUG [org.apache.http.headers] << Content-Length: 0

DEBUG [org.apache.http.headers] << Server: AmazonS3

this print is just after the call to s3Service.putObject - as you see it is almost 30 seconds later!

putObject verification: key = 385964/0742.jpg created fileObject key = 385964/0742.jpg (12:39:13)

uploaded object: S3Object [key=385964/0742.jpg, bucket=digilabs.galleries.4, lastModified=Tue May 07 12:38:44 PDT 2013, dataInputStream=null, Metadata={ETag="2e8024e09276bdab25b5cf5b8ca93c8b", Date=Tue May 07 12:38:44 PDT 2013, Content-Length=4949311, id-2=o/L31ov0Oa7RLNGeBnnqKsvj5FMwtZZuWVmWyeTL5zvkpBTTcxRVJMKZRAfsyC2Z, request-id=858143F642961598, Content-MD5=LoAk4JJ2vasltc9bjKk8iw==, Content-Type=image/jpeg}] (12:39:13)

 file 0742.jpg uploaded to S3 (12:39:13)

this is my code (very similar to before, just added some more debug print information)

           long contentLength = file.length();

           fileObject.setBucketName(bucketName);

           if (key != null && key.length() > 0)

          fileObject.setKey(key);

           // add more metadata information? e.g. content length...

           fileObject.setContentLength(contentLength);

           

           Debug.debug("timing before s3Service.putObject - bucket " + fileObject.getBucketName() 

          + " key - " + fileObject.getKey() 

          + " content length - " + fileObject.getContentLength()

          );

           fileObject = s3Service.putObject(bucketName, fileObject);

           Debug.debug("putObject verification: key = " + key + " created fileObject key = " + fileObject.getKey());

           Debug.debug("uploaded object: " + fileObject);

do you see anything that I am missing? 

thank you

Mia 

[James Murty]

Hi,

I don't see any obvious problems in your code or the HTTP headers.

If there is a pause somewhere in the request/response cycle, configuring your logs to report per-line timestamps (not just "DEBUG [org.apache.http.headers]") might reveal it.

Aside from that I'm not sure where to look. I assume you have tested uploads to S3 from your production machine using some other technique, to be sure that the JetS3t upload is actually slower?

---
jamesmurty.com

[Mia Shani]

Thanks James, actually I just tested saving files "directly" using

          AmazonS3 s3 = new AmazonS3Client(credentials);

          PutObjectResult por = s3.putObject(new PutObjectRequest(bucketName, key, file));

and I get the same long timing.

I suppose I must look at my computer's setting.

If you have any hint from experience, I'd really appreciate it!

thanks a lot

Mia