Moving Images to Amazon S3

[Kyle Fox]

I'm absolutely stuck on this. This question might be more python than
django related, but I figured because it deals with Amazon S3, someone
here may know how to handle it.

Here's what needs to happen:
1. A user uploads an image (part of a Model we have)
2. Create a 100x100 thumbnail using PIL
3. Move the thumbnail to Amazon S3

Step 3 is the tricky part for me. I can't figure out how to do it,
other than first temporarily saving the in-memory thumbnail to the
filesystem, re-reading the file, moving the raw data to S3, and then
deleting the temporary thumbnail file:

# thumb is an already thumbnailed Image instance, in memory
thumb.save( temp_path, "JPEG")
raw = open( temp_path ).read()
# code to move 'raw' to S3
os.remove( temp_path )

This is a pretty heavy way of doing it, and I'm desperate to find a
better way of doing it but am absolutely out of ideas...

The S3 API requires the file data to be in the format the
open(myfile).read() returns (whatever that is).

Is there a way to get the same data from an in-memory Image instance,
so I don't need to save/re-read/delete each thumbnail file?

[Nic James Ferrier]

Kyle Fox <kyle...@gmail.com> writes:

> The S3 API requires the file data to be in the format the
> open(myfile).read() returns (whatever that is).
>
> Is there a way to get the same data from an in-memory Image instance,
> so I don't need to save/re-read/delete each thumbnail file?

Surely the better way to do this is to just use FUSE or something to
map S3 into the filesystem?

--
Nic Ferrier
http://www.tapsellferrier.co.uk

[Frédéric Sidler]

openfount provide something that does the trick
http://www.openfount.com/blog/s3dfs-for-ec2

and adrian holovaty did something like that some weeks ago for chicagocrime.org
http://www.holovaty.com/blog/archive/2006/04/07/0927

2007/5/31, Nic James Ferrier < nfer...@tapsellferrier.co.uk>:

--
<div style="border-top: 1px dotted rgb(153, 153, 153); border-bottom: 1px dotted rgb(153, 153, 153); margin: 6px 0pt; padding: 8px; line-height: 13px; font-family: 'Lucida Grande',Verdana,Arial,Sans-Serif; font-size: 11px; color: rgb(51, 153, 153);">
<img src="http://farm1.static.flickr.com/233/514575183_851b07d59b.jpg" alt="me" style="padding: 2px 6px 0pt 0pt; float: left;">
<strong style="color: rgb(51, 153, 153);">Fr&eacute;d&eacute;ric Sidler</strong>: Test at Mixin.com - <a href="http://www.mixin.com/ " style="border-bottom: 1px dotted rgb(153, 153, 153); color: rgb(51, 153, 153); text-decoration: none;">Mixin.com</a><br>
contact | <a href="mailto: frederi...@gmail.com" style="border-bottom: 1px dotted rgb(153, 153, 153); color: rgb(51, 153, 153); text-decoration: none;"> frederi...@gmail.com</a> - +41 79 704 56 42 | mixin - frederic.sidler
</div>

[Nic James Ferrier]

"=?ISO-8859-1?Q?Fr=E9d=E9ric_Sidler?=" <frederi...@gmail.com>
writes:

> openfount provide something that does the trick
> http://www.openfount.com/blog/s3dfs-for-ec2
>
> and adrian holovaty did something like that some weeks ago for
> chicagocrime.org
> http://www.holovaty.com/blog/archive/2006/04/07/0927

Yes. Implementing S3 at this level actually is not terribly hard... it
works quite well.

And there are good Python bindings for FUSE.

[Kyle Fox]

I think my question wasn't clear: we're having NO problem putting
files on S3, that's dead simple (ie what Holovaty blogged about).

What we need to do is take an *in-memory Image* and put it directly
onto S3. We need a way to convert a PIL Image instance into a format
S3 can accept WITHOUT having to first save the Image to the file-
system. S3 accepts the kind of string that gets returned from
open(afile).read()

I was trying to wrap the result of img.tostring() in the StringIO
class, and then put that on S3, but that doesn't work either...

[SanPy]

Try this:
# in your view
raw_image_data = form.cleaned_data['photo']['content']
thumbnail_content = resize_image(raw_image_data)
filename = form.cleaned_data['photo']['filename']
upload_to_s3(filename, thumbnail_content)

def resize_image(buf, size=(100, 100)):
f = cStringIO.StringIO(buf)
image = PILImage.open(f)
if image.mode not in ('L', 'RGB'):
image = image.convert('RGB')
image.thumbnail(size, PILImage.ANTIALIAS)
o = cStringIO.StringIO()
image.save(o, "JPEG")
return o.getvalue()

def upload_to_s3(filename, filedata):
conn = S3.AWSAuthConnection(AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY)
content_type = mimetypes.guess_type(filename)[0]
response = conn.put(BUCKET_NAME, '%s' % filename,
S3.S3Object(filedata), {'x-amz-acl': 'public-read', 'Content-Type':
content_type})

Regards, Sander.

[SanPy]

One addition:

On 31 mei, 21:35, SanPy <jhmsm...@gmail.com> wrote:
> Try this:
> # in your view
> raw_image_data = form.cleaned_data['photo']['content']
> thumbnail_content = resize_image(raw_image_data)
> filename = form.cleaned_data['photo']['filename']

Change the filename's extension to jpg, because it gets converted to
JPEG in resize_image:
filename = os.path.splitext(filename)[0] + ".jpg"

[Kyle Fox]

SanPy,

Thank you! This works awesome and does exactly what I need!