Atomically check existence of S3 key and put object if it doesn't exist

[Scott]

I need to maintain users' original filenames in S3. But a user might
upload 2 files with the same name (or the same file twice). So I
simply append a _2, _3, etc to the end of the filename. Under
absolutely no circumstances can I overwrite an existing object in S3.
For special reasons I can't do a db transaction like "select count(*)
from files where filename=?; if (result == 0) insert into files values
(userId, filename)".

Is there a way to do this in S3 and jets3t? What I currently do is
check the boolean value of new File("image.jpg").createNewFile().
It's an atomic operation that returns false if the file existed or
true and creates the file if it didn't exist. So if it existed, I try
image_2.jpg, etc. But I need to move this to S3 and I can't use the
filesystem anymore.

[James Murty]

You cannot really accomplish what you want with this approach using S3, or indeed with any distributed file system like S3.

Due to the distributed nature of S3 you cannot assume that a result you get from the service is 100% accurate. Because the service exists over many different servers it may take some time for changes to be synchronized to all of them. In other words, there could be a delay between uploading a file and being able to test its presence within S3. If you rely on looking up file names on S3, there is a good chance that you will overwrite a file sooner or later.

You should consider use a different mechanism to ensure files are not overwritten. For example, you could include the hash of a file's contents in the S3 object name to ensure that every unique file has a unique name. You can still retain the original filename if you set the name in the Content-Distribution header and make the hash value a prefix of the S3 object name, like so:

<md5-hash-value>/original-filename.jpg

This wouldn't produce two different objects if a user uploads the same file twice, but if a user does that I don't see why you can't keep just one copy anyway.

Hope this helps,
James

---
http://www.jamesmurty.com

[Scott]

It's relatively easy to create 2 different files with the same md5
hash value:
http://www.win.tue.nl/hashclash/SoftIntCodeSign/

Of course I very much like the idea of not wasting storage on
identical files...

It was important to me to keep the keys very clean for several
reasons, but I see now that S3 doesn't support true renaming of
directories (anything before a / in the key name) or files. Instead
it simply copies the files individually with a new name. It'll be
very common for me to need to rename many of these virtual directories
with thousands of files in them... so I'm going to just use names
ensured to be unique with my database.. Amazon hasn't given any
indication in their forums that they plan to allow true renaming of
object keys in the future either.

On Jul 26, 1:30 am, James Murty <jamu...@gmail.com> wrote:
> You cannot really accomplish what you want with this approach using S3, or
> indeed with any distributed file system like S3.
>
> Due to the distributed nature of S3 you cannot assume that a result you get
> from the service is 100% accurate. Because the service exists over many
> different servers it may take some time for changes to be synchronized to
> all of them. In other words, there could be a delay between uploading a file
> and being able to test its presence within S3. If you rely on looking up
> file names on S3, there is a good chance that you will overwrite a file
> sooner or later.
>
> You should consider use a different mechanism to ensure files are not
> overwritten. For example, you could include the hash of a file's contents in
> the S3 object name to ensure that every unique file has a unique name. You
> can still retain the original filename if you set the name in the
> Content-Distribution header and make the hash value a prefix of the S3
> object name, like so:
>
> <md5-hash-value>/original-filename.jpg
>
> This wouldn't produce two different objects if a user uploads the same file
> twice, but if a user does that I don't see why you can't keep just one copy
> anyway.
>
> Hope this helps,
> James
>

> ---http://www.jamesmurty.com

[James Murty]

It's relatively easy to create 2 different files with the same md5
hash value:
http://www.win.tue.nl/hashclash/SoftIntCodeSign/

You could always use one of the SHA hashes instead, but the chances of a clash with both a file's MD5 hash *and* its file name would be pretty remote. On the other hand, you could use GUIDs instead of hash values to be even safer.
 

It was important to me to keep the keys very clean for several
reasons, but I see now that S3 doesn't support true renaming of
directories (anything before a / in the key name) or files.  Instead
it simply copies the files individually with a new name.  

That's right. It is unwise to think of S3 as simply an online file system because it has very different capabilities and behavior. The more you can work to S3's strengths the easier time you will have integrating with the service.

[Adrian Cole]

According to hostedftp [1], one can assume a 80ms overhead on any put operation to S3.  Not to bake this into your design, but there are 1024 characters available for use ;)  You can put a millisecond-precise timestamp suffix on your file name. 

If you use a suffix approach (hash or timestamp or both), the benefit is that you can do a bucket listing and match all of them back, then parse the date for the latest one (either the date in metadata, or the date in the suffix. it doesn't really matter). 

As James mentioned, any attempts to use S3 an immediately consistent store are doomed.  You'd either need to balance the consistency problem with something like this (client-side), or put a layer in front that deals with it (ec2 running a cache that supports atomic puts, and then stores in S3).


I hope this helps, Scott.
-Adrian

[1] http://hostedftp.wordpress.com/2009/06/30/hostedftp-amazon-aws-s3-performance-report-how-fast-is-the-cloud/