Trouble writing image to AWS S3

189 views
Skip to first unread message

Michael Wood

unread,
Nov 3, 2015, 2:42:32 PM11/3/15
to Lucee
Hi,

I'm having trouble writing an image into S3 where the object name includes a "sub-folder".

This works:
<cfscript>
   imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/image1.jpg";
   ImageWrite(myImg, imgWritePath, 1, true);
</cfscript>

This does not:
<cfscript>
   imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000/image1.jpg";
   ImageWrite(myImg, imgWritePath, 1, true);
</cfscript>

The second one returns error:
"can't write file [s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file, 
missing parent directory [s3://[secretId]:[secretKey]@/[bucketName]/000]"

There seems to be a spurious "/" inserted after the @ symbol. I didn't put it there...

Mike

Sean Daniels

unread,
Nov 3, 2015, 2:47:03 PM11/3/15
to lu...@googlegroups.com
I'm pretty sure you need to explicitly create the subfolder if it does not yet exist.

<cfscript>
imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000";
imgWritefile = "image1.jpg";

if (!directoryExists(imgWritepath))
directorycreate(imgWritepath)

ImageWrite(myImg, imgWritePath & "/" & imgWritefile, 1, true);
</cfscript>
> --
> Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
> ---
> You received this message because you are subscribed to the Google Groups "Lucee" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
> To post to this group, send email to lu...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ab473c81-3dac-443c-b420-ec7cb59c568d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Michael Wood

unread,
Nov 3, 2015, 2:52:12 PM11/3/15
to Lucee
There aren't actually sub-folders in S3; only objects. But I did try pre-creating the that folder by manually creating another object with that folder name. It didn't fix the issue.

Andrew Dixon

unread,
Nov 3, 2015, 3:00:19 PM11/3/15
to lu...@googlegroups.com
Hi Michael,

S3 is problematic using the built in functions, check out this project:


Kind regards,

Andrew

Message has been deleted
Message has been deleted
Message has been deleted

Michael Wood

unread,
Nov 3, 2015, 10:50:14 PM11/3/15
to Lucee
Hi Andrew,

Sorry about the repeated posts and deletes - I'm trying to figure out how to use this interface.

Thank you for your suggestion and help. Your suggestion REALLY helped but it took me the intervening time to figure out how to get/install/configure/implement it. 

So the Lucee-AWS documentation was of limited use but then I'm not an expert on image processing or handling. To potentially save others my pain here's some instructions and sample code:

1. Get the Lucee-AWS extension installed on your box however you can. There's limited help at https://github.com/mso-net/lucee-aws. I installed and used the recommended CommandBox tool but figuring out how to install CommandBox on my remote Linux server was a bit if a puzzle to me in itself. 

2. Once Lucee-AWS is installed then you can put it to use - And it's pretty simple to use once installed BUT only if you know that you need to use the binary read and base64 conversions.

3. You add this to your Application.cfc file:
this.javaSettings = { loadPaths: [ '/aws/aws-java-sdk/']};

4. Here's sample code how to USE the extension in a CFM template.

<cfscript>
   // Setup to use the method calls
   s3 = new aws.s3( 
      account = '#awsAccessKeyId#',
      secret = '#awsSecretKey#',
      bucket = 'myBucket'
   );

// To read an image from S3
   myObj = s3.getObject( 'myImage.jpg' );
   myImg = imageReadBase64(myObj.content);
   // You can now treat myImg as a ColdFusion image
   
   // To write an image to S3 first read the image file as binary and then convert it to base64
   myImg = toBase64(fileReadBinary('mytest.jpg'));
   // This is how you send the file to S3 - Note that you have to insert the "data:image/jpg;base64," in front of the image base64 string. 
   // Change the file extension as needed but I'm pretty sure it has to match the extension of the file you read in, perhaps not
   s3.putObject( 'myFolder/myImage.jpg' , 'data:image/jpg;base64,#myImg#' );
   // Delete an image object from S3
   s3.deleteObject( 'myFolder/myImage.jpg' );
</cfscript>

Mike

Pete Freitag

unread,
Nov 4, 2015, 9:56:44 AM11/4/15
to lu...@googlegroups.com
You can also upload the file directly to S3 from the client's browser, here's how you can do it in CFML: https://www.petefreitag.com/item/833.cfm  this approach is nice because it does not utilize your server resources for the upload, but may be a bit more work to get going.

--
Pete Freitag
https://foundeo.com/ - ColdFusion Consulting & Products
http://hackmycf.com - CFML Server Security Scanner


Michael Wood

unread,
Nov 4, 2015, 12:48:03 PM11/4/15
to Lucee, pe...@foundeo.com
Hi Pete!

Oh I wish I could but we do processing on the images (re-size, create thumbnails, etc) in EC2 before storing them in S3. I can't think of another way to pre-process the images before storing them in S3, can you?

This is something new we're trying. We have the upload pages working in an iFrame so that the upload and image processing is offloaded entirely to the client and to EC2/S3. They will communicate directly freeing this load from our main server. Running Lucee on Amazon Linux in EC2 and using S3 for storage made this idea possible/practical/affordable...

wish we could do this in Lambda but I'm pretty sure the programming required is too far out of my skill set.

Mike

Michael van Leest

unread,
Nov 4, 2015, 1:22:34 PM11/4/15
to lu...@googlegroups.com
@Michael Wood: Is a intermediate upload bucket an option? After the upload is done, you can get the image from S3, process the image and place it in the final bucket?

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Michael van Leest

Andrew Dixon

unread,
Nov 4, 2015, 1:36:49 PM11/4/15
to lu...@googlegroups.com
HI Michael,

You could use the AWS Lambda service to process the images directly on AWS with the Lambda function triggered by the "store" event in S3, it is one of the examples in the AWS Lambda docs:


Kind regards,

Andrew

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Pete Freitag

unread,
Nov 4, 2015, 1:55:49 PM11/4/15
to lu...@googlegroups.com
As Michael Van Leest suggested you could have another bucket that receives the raw uploads, and then process those using a bucket event notification: https://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html which would be a callback that AWS makes to your server after the upload is complete. 

A lambda function would probably be the ideal solution for this, but if you want to process it on your server you can do a bucket event instead.




--
Pete Freitag
https://foundeo.com/ - ColdFusion Consulting & Products
http://hackmycf.com - CFML Server Security Scanner


--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Michael Wood

unread,
Nov 4, 2015, 3:44:03 PM11/4/15
to Lucee, pe...@foundeo.com
Guys, is there a disadvantage to doing this the way I'm doing it; that is uploading to an EC2 instance, processing the images, and then moving them into S3? 

Or saying it the other way around, are there potential compelling advantages to uploading straight to S3 and then post-processing the images?

Our pre-processing in EC2 includes image/file type validation, security checks, image conversion to jpg (as a standard), image re-sizing, creation of a thumbnail, AND we do a web service call to get an id to name the file which also determines a "folder name" to use in the object name for S3. It seems like we will have a lot more control this way. And amazingly all of this happens pretty quickly. Of course I haven't had an opportunity to do load testing yet. Uploading directly to S3 might be more robust.

The other advantage to pre-processing is that the end user immediately SEES the uploaded/processed image and thumbnail (OR the failure to upload/process and can try again).

Mike Wood

Andrew Dixon

unread,
Nov 4, 2015, 4:21:18 PM11/4/15
to lu...@googlegroups.com
Pretty sure like with most things there are advantages and disadvantages to all ways of doing it. 

If however you do go down the route of uploading directly to S3 I would warn you that doing image manipulation on an image stored on S3 already via Lucee is SLOW!!! If you are going to upload directly to S3 I would suggest the Lambda route would be far quicker.

Kind regards,

Andrew

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages