Content-Type on public S3 Blob

[Eric Charles (GMail)]

Hi,

I need to set the content-type when uploading an object to AWS S3 (I use
jclouds 1.6.0-rc.4).

The [1] snippet works well (contentType method), but I need to make the
object public, and if I understand well, this is only possible via the
AWSS3Client.

So I use [2], but does not succeed to enforce the content-type (it is
always knows as 'application/unknown' by S3.


Any idea how to upload a public object with a given Content-Type?

Thx in advance and Congrats for all your work,

Eric


[1]

BlobStoreContext context = ContextBuilder. //
newBuilder("aws-s3"). //
credentials(IDENTITY, CREDENTIALS). //
buildView(BlobStoreContext.class);
BlobStore store = context.getBlobStore();
Blob blob = store.blobBuilder(BUCKET). //
name(IMAGE_NAME). //
payload(new File("src/main/resources/" + IMAGE_NAME)).
contentType("image/jpeg"). //
calculateMD5(). //
build();
store.putBlob(IMAGE_NAME, blob);

[2]

BlobStoreContext context = ContextBuilder. //
newBuilder("aws-s3"). //
credentials(IDENTITY, CREDENTIALS). //
buildView(BlobStoreContext.class);
AWSS3Client s3Client = ContextBuilder. //
newBuilder("aws-s3"). //
credentials(IDENTITY, CREDENTIALS). //
buildApi(AWSS3Client.class);
S3Object s3Object = s3Client.newS3Object();
s3Object.getMetadata().setKey(IMAGE_NAME);
Map<String, String> userMetaData = Maps.newHashMap();
userMetaData.put("Content-Type", "image/jpeg");
s3Object.getMetadata().setUserMetadata(userMetaData);
s3Object.getMetadata().getContentMetadata().setContentType("image/jpeg");
s3Object.getAllHeaders().put("Content-Type", "image/jpeg");
s3Object.setPayload(new File("src/main/resources/" + IMAGE_NAME));
s3Client.putObject(BUCKET, s3Object,
withAcl(CannedAccessPolicy.PUBLIC_READ));

[Andrew Phillips]

Hi Eric

You should be able to get the public URI from the first snippet. See
the following example:

https://github.com/jclouds/jclouds-cloud-storage-workshop/blob/master/exercise1/src/main/java/org/jclouds/labs/blobstore/exercise1/ImageUploaderC.java#L71

Hope that helps!

ap

[Eric Charles (GMail)]

Hi Andrew,

Thx for the pointer, but my issue is not to get the public URI.

I have been confusing in my explanation, so to recap:

- I have a private bucket (only granted to me).
- I want on a per upload basis define the permission (in most case
private, but for some public), a bit like right-clicking on the item via
the S3 Console and selecting "Make Public".

Adding the permission 'read for everyone' solves the code limitation: my
snippet [1] can be used and I have the correct Content-Type defined and
delivered my I browse my image.

My snippet [2] can be used with a private bucket, setting on per item
basis, but I loose the ability to define the Content-Type.

I tried to compare the HTTP requests sent to S3, but working on SSL, I
can't see the headers...

I still would like to understand how to set on a per item basis the
permissions. If there is a bug, I will be happy to try to provide a fix
for this.

Thx, Eric

[Andrew Phillips]

Hi Eric

Try setting the payload *before* setting the content metadata (where
the content length goes [1]):

S3Object s3Object = s3Client.newS3Object();

s3Object.setPayload(new File("src/main/resources/" + IMAGE_NAME));

s3Object.getMetadata().getContentMetadata().setContentType("image/jpeg");
s3Client.putObject(BUCKET, s3Object, withAcl(CannedAccessPolicy.PUBLIC_READ));

Setting the payload *afterwards* causes the content metadata to be
wiped, I think [2]. This may be intentional but I would certainly also
be curious to know *why*, since the generic-blob-to-S3-object
conversion [3] also sets the payload *after* the metadata.

Regards

ap

[1]
https://github.com/jclouds/jclouds/blob/master/blobstore/src/main/java/org/jclouds/blobstore/domain/internal/BlobBuilderImpl.java#L194
[2]
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/domain/internal/S3ObjectImpl.java#l135
[3]
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/blobstore/functions/BlobToObject.java#L48

[Eric Charles (GMail)]

Hi Andrew,

I have tested setting the payload after the content type, but no luck,
S3 still shows Content-Type=application/unknown

Do you have more success running that code snippet?
Once we can upload successfully with a content type, we can look for any
bugs in the methods call order.

Thx, Eric

[Andrew Phillips]

> I have tested setting the payload after the content type, but no luck,
> S3 still shows Content-Type=application/unknown

Hm :-( Can you put the header and wire log [1] up on Pastie - stripped
of keys etc., obviously?

Thanks!

ap

[1] http://www.jclouds.org/documentation/reference/jclouds-logging/

[Eric Charles (GMail)]

Here we go:

[1] http://pastie.org/7634614 (further to [s1])
--> Metadata shows Content-Type=image/jpeg

[2] http://pastie.org/7634665 (further to [s2])
--> Metadata shows Content-Type=application/unknown

Seems like [2] does not send the AccessControlPolicy (XML) and misses a
final step present in [1].

Thx, Eric

[s1]

BlobStoreContext context = ContextBuilder. //
newBuilder("aws-s3"). //
credentials(IDENTITY, CREDENTIALS). //

modules(ImmutableSet.<Module> of(
new Log4JLoggingModule())). //

buildView(BlobStoreContext.class);

BlobStore store = context.getBlobStore();

Blob blob = store.blobBuilder(BUCKET_NAME). //
name(IMAGE_NAME). //
payload(new File("src/main/resources/" + IMAGE_NAME)). //
// or byte[], InputStream...
contentDisposition("attachment; filename=" +
IMAGE_NAME). //

contentType("image/jpeg"). //
calculateMD5(). //
build();

store.putBlob(BUCKET_NAME, blob);

AWSS3Client s3Client = ContextBuilder. //
newBuilder("aws-s3"). //
credentials(IDENTITY, CREDENTIALS). //

modules(ImmutableSet.<Module> of(
new Log4JLoggingModule())). //

buildApi(AWSS3Client.class);
S3Object s3Object = s3Client.newS3Object();
s3Object.getMetadata().setKey(IMAGE_NAME);

s3Object.getMetadata().getContentMetadata().setContentType("image/jpeg");

s3Object.setPayload(new File("src/main/resources/" + IMAGE_NAME));

s3Client.putObject(BUCKET_NAME, s3Object,
withAcl(CannedAccessPolicy.PUBLIC_READ));
s3Client.close();

[Andrew Phillips]

Hi Eric

Puzzling...here's my attempt, with jclouds 1.6.0-rc.4 under Java 1.6 [1]:

AWSS3Client s3Client = ContextBuilder.newBuilder("aws-s3")
.credentials("foo", "bar")
.modules(ImmutableSet.of(new Log4JLoggingModule()))
.buildApi(AWSS3Client.class);
S3Object s3Object = s3Client.newS3Object();
s3Object.setPayload(new File("src/main/resources/cloud.jpg"));
s3Object.getMetadata().setKey("foo");
s3Object.getMetadata().getContentMetadata().setContentType("image/jpeg");
System.out.println(s3Object.getMetadata().getContentMetadata());
s3Client.putObject("test", s3Object, AWSS3PutObjectOptions.Builder
.withAcl(CannedAccessPolicy.PUBLIC_READ));
s3Client.close();

results in

22:59:18,503 DEBUG [headers] >> Content-Type: image/jpeg

Could you try that code snippet and see if you get the same headers I do?

Thanks!

ap

[1] http://pastie.org/private/jfscmj5wnhpkatrgchgsq

[Eric Charles (GMail)]

Thx Andrew for your time and support!

Your snippet works here (jclouds-1.6.0-rc.4 on jdk7): the log shows the
header is sent and S3 console now indicates a friendly image/jpeg
Content-Type.

I have taken back my previous code, put the setPayload as first called
method, and now it also works.

There is definitively something weird to fix and it has to do with the
method call sequence. If the setPayload(...) is called after the
getMetadata().getContentMetadata().setContentType(...), it fails.

The good thing for now is that we have identified a bug and that I have
learned how to log/debug jclouds.

Unless you fix it before me, I will take back in the coming days the
code you pointed in a previous mail and try to see what's happening there.

What is the jclouds practice? Do you expect issues/pull-requests on
https://github.com/jclouds/jclouds?

Thx,

Eric

[Andrew Phillips]

> I have taken back my previous code, put the setPayload as first called
> method, and now it also works.
>
> There is definitively something weird to fix and it has to do with the
> method call sequence. If the setPayload(...) is called after the
> getMetadata().getContentMetadata().setContentType(...), it fails.
>
> The good thing for now is that we have identified a bug and that I have
> learned how to log/debug jclouds.

The code that is causing the observed behaviour is at [1]. There was a
chat about this on IRC, and from what I understand this is, in fact,
intentional.

The thing is that the content type is actually an attribute of the
*payload* (it's part of the content metadata, not of the object
metadata), and so setting the payload with *empty* content metadata
(which is what the setPayload(new File(...)) call does) overwrites and
earlier changes to the content metadata.

If you use the following command sequence:

s3Object = client.newObject()
Payload payload = new FilePayload(new File(...)) // [2]
payload.getContentMetadata().setContentType("image/jpeg")
s3Object.setPayload(payload)
...

then you should get the same result and are also safe from reordering effects.

Glad to hear it worked in the end!

ap

[1]
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/domain/internal/S3ObjectImpl.java#l135
[2]
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/io/payloads/FilePayload.java

[Adrian Cole]

Andrea's right, the behavior was intentional, but we should point out that confusion is not :)  Blob is our oldest object and it hasn't changed fundamentally since it was created.  For example, we didn't update it when other parts of jclouds went immutable although we did make a builder.  Keep an open mind and see if you can help us clean it up!

Cheers,

-A

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jclouds+u...@googlegroups.com.
To post to this group, send email to jcl...@googlegroups.com.
Visit this group at http://groups.google.com/group/jclouds?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Eric Charles (GMail)]

Thx to all for the details.

As a user, I would expect a runtime exception with a friendly message if
I invoke in the wrong order. Of course one could argue that you could
override the existing metadata with a brand new payload, but we this we
unfortunateluy would be far away from any immutable behaviour.

Of course, I don't have the needed background and something else may
come into the play to not trhow runtime exception.

If the runtime exception is not an option, the javadoc should at least
mention this.

Thx, Eric

On 18/04/2013 15:35, Adrian Cole wrote:
>
> Andrea's right, the behavior was intentional, but we should point out
> that confusion is not :) Blob is our oldest object and it hasn't
> changed fundamentally since it was created. For example, we didn't
> update it when other parts of jclouds went immutable although we did
> make a builder. Keep an open mind and see if you can help us clean it up!
>
> Cheers,
> -A
>
> On Thursday, April 18, 2013, Andrew Phillips wrote:
>
> I have taken back my previous code, put the setPayload as first
> called
> method, and now it also works.
>
> There is definitively something weird to fix and it has to do
> with the
> method call sequence. If the setPayload(...) is called after the

> getMetadata().__getContentMetadata().__setContentType(...), it

> fails.
>
> The good thing for now is that we have identified a bug and that
> I have
> learned how to log/debug jclouds.
>
>
> The code that is causing the observed behaviour is at [1]. There was
> a chat about this on IRC, and from what I understand this is, in
> fact, intentional.
>
> The thing is that the content type is actually an attribute of the
> *payload* (it's part of the content metadata, not of the object
> metadata), and so setting the payload with *empty* content metadata
> (which is what the setPayload(new File(...)) call does) overwrites
> and earlier changes to the content metadata.
>
> If you use the following command sequence:
>
> s3Object = client.newObject()
> Payload payload = new FilePayload(new File(...)) // [2]

> payload.getContentMetadata().__setContentType("image/jpeg")

> s3Object.setPayload(payload)
> ...
>
> then you should get the same result and are also safe from
> reordering effects.
>
> Glad to hear it worked in the end!
>
> ap
>
> [1]

> https://github.com/jclouds/__jclouds/blob/master/apis/s3/__src/main/java/org/jclouds/s3/__domain/internal/S3ObjectImpl.__java#l135
> <https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/domain/internal/S3ObjectImpl.java#l135>
> [2]
> https://github.com/jclouds/__jclouds/blob/master/core/src/__main/java/org/jclouds/io/__payloads/FilePayload.java

> <https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/io/payloads/FilePayload.java>
>
> --
> You received this message because you are subscribed to the Google
> Groups "jclouds" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jclouds+u...@googlegroups.com.
> To post to this group, send email to jcl...@googlegroups.com.

> Visit this group at http://groups.google.com/__group/jclouds?hl=en
> <http://groups.google.com/group/jclouds?hl=en>.
> For more options, visit https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

[Adrian Cole]

Andrea's right, the behavior was intentional, but we should point out that confusion is not :)  Blob is our oldest object and it hasn't changed fundamentally since it was created.  For example, we didn't update it when other parts of jclouds went immutable although we did make a builder.  Keep an open mind and see if you can help us clean it up!

Cheers,

-A

On Thursday, April 18, 2013, Andrew Phillips wrote:

[Adrian Cole]

Andrea's right, the behavior was intentional, but we should point out that confusion is not :)  Blob is our oldest object and it hasn't changed fundamentally since it was created.  For example, we didn't update it when other parts of jclouds went immutable although we did make a builder.  Keep an open mind and see if you can help us clean it up!

Cheers,

-A

On Thursday, April 18, 2013, Andrew Phillips wrote:

[Eric Charles (GMail)]

Any love for any runtimeexception on bad sequence call?

(I'm sure I am not the only one always choosing the wrong queue at the
supermarket :)

Especially if rc5 is soon cut soon, the momentum is good for any patch,
at the risk of breaking some code (I'm sure some implementation call the
methods in bad order and think they effectively set the content-type,
but they don't).

Thx, Eric