AmazonS3 putObject with InputStream length example

[JohnIdol via StackOverflow]

I am uploading a file to S3 using java -- this is what I got so far:

AmazonS3 s3 = new AmazonS3Client(new BasicAWSCredentials("XX","YY"));

List<Bucket> buckets = s3.listBuckets();

s3.putObject(new PutObjectRequest(buckets.get(0).getName(), fileName, stream, new ObjectMetadata()));

The file is being uploaded but a WARNING is raised when I am not setting the content length:

com.amazonaws.services.s3.AmazonS3Client putObject: No content length specified for stream > data.  Stream contents will be buffered in memory and could result in out of memory errors.

This is a file I am uploading and the stream variable is an InputStream, from which I can get the byte array like this: IOUtils.toByteArray(stream).

So when I try to set the content length and MD5 (taken from here) like this:

// get MD5 base64 hash
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
messageDigest.reset();
messageDigest.update(IOUtils.toByteArray(stream));
byte[] resultByte = messageDigest.digest();
String hashtext = new String(Hex.encodeHex(resultByte));

ObjectMetadata meta = new new ObjectMetadata();
meta.setContentLength(IOUtils.toByteArray(stream).length);
meta.setContentMD5(hashtext);

It causes the following error to come back from S3:

The Content-MD5 you specified was invalid.

What am I doing wrong?

Any help appreciated!

P.S. I am on Google App Engine - I cannot write the file to disk or create a temp file because AppEngine does not support FileOutputStream.

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/8351886/amazons3-putobject-with-inputstream-length-example

[srikfreak via StackOverflow]

While writing to S3, you need to specify the length of S3 object to be sure that there are no out of memory errors.

Using IOUtils.toByteArray(stream) is also prone to OOM errors because this is backed by ByteArrayOutputStream

So, the best option is to first write the inputstream to a temp file on local disk and then use that file to write to S3 by specifying the length of temp file.

Please DO NOT REPLY directly to this email but go to StackOverflow:

http://stackoverflow.com/questions/8351886/amazons3-putobject-with-inputstream-length-example/8352132#8352132

[tarka via StackOverflow]

If all you are trying to do is solve the content length error from amazon then you could just read the bytes from the input stream to a Long and add that to the metadata.

    /*
     * Obtain the Content length of the Input stream for S3 header
     */
    try {
        InputStream is = event.getFile().getInputstream();
        contentBytes = IOUtils.toByteArray(is);
    } catch (IOException e) {
        System.err.printf("Failed while reading bytes from %s", e.getMessage());
    } 

    Long contentLength = Long.valueOf(contentBytes.length);

    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentLength(contentLength);

    /*
     * Reobtain the tmp uploaded file as input stream
     */
    InputStream inputStream = event.getFile().getInputstream();

    /*
     * Put the object in S3
     */
    try {

        s3client.putObject(new PutObjectRequest(bucketName, keyName, inputStream, metadata));

    } catch (AmazonServiceException ase) {
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("AWS Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    } catch (AmazonClientException ace) {
        System.out.println("Error Message: " + ace.getMessage());
    } finally {
        if (inputStream != null) {
            inputStream.close();
        }
    }

You'll need to read the input stream twice using this exact method so if you are uploading a very large file you might need to look at reading it once into an array and then reading it from there.

Please DO NOT REPLY directly to this email but go to StackOverflow:

http://stackoverflow.com/questions/8351886/amazons3-putobject-with-inputstream-length-example/11116119#11116119