File Storage API: save file to DB model and disk: separate disk path from file name

[Jon Dufresne]

Hi,

I'm trying to use the File Storage API to create a custom handler for
files. Here are my goals:

Stored files are represented both in the database and on disk with the
following model:

class StoredFile(models.Model):
name = models.CharField(max_length=255)
path = models.CharField(max_length=255, unique=True)

The "name" attribute represents the filename presented to the user:
"My document.txt". The path attribute represents the basename as the
file exists on the disk in MEDIA_ROOT, usually created with mktemp:
"tmpAsdf123" in "/path/to/media". The goal of this configuration:

1. Files are stored on disk
2. There are no filename collisions (duplicate names can exist but the
paths will be unique)
3. Uploaded files preserve the name exactly, may be downloaded later,
are never mangled
4. All files are stored in an identical manner

You can think of "name" as an abstract user-oriented name, and path as
the actual on disk representation.

What I want, saving a file should save the file to path and record an
entry for the StoredFile model. Ideally FileFields would be a FK to
this model--currently it is a string.

I'm having a little trouble getting started. Some issues:

1. FileFields store strings, would like it to be a one-to-one FK (or
an intereger id)
2. Most of the storage API takes a single argument "name" Should this
be the name as stored on disk? The name as presented to the user? The
PK from the StoredFile model?

Is what I'm trying to do achievable? Am I approaching the problem correctly?

This design was taken from an existing application, so
interoperability is also a goal.

Cheers,
Jon

[Russell Keith-Magee]

Achievable? probably. Easy? No. You're going to be digging in the weeds on this one. 

If I were trying to do this, I wouldn't try and get this to work with the existing FileField - I'd be writing my own Field definition. You can crib extensively from the existing definition, but everywhere that the current model points at it's own storage to get a filename, you use the foreign key to get the full file name. 

Essentially, it's going to be a merger of FileField and ForeignKey. I haven't done this myself, but my gut tells me this is in the realm of the "possible, but complicated".

The other way that might be worth looking into is to follow the lead of GenericForeignKey. That's an example of a field that "looks" like a single column, but is implemented as two columns under the hood. You'll be looking to do a similar thing - write a "wrapper" field that points at the name and path fields to produce a composite "full file path" field. That composite field will be a lot closer to what FileField is expecting, so it might be a little easier to merge *that* with FileField than to merge FileField with ForeignKey. Then, you put the composite field on the StoredFile model, and traverse down the foreign key relation whenever you need to access the file.

Of course YMMV, and if it breaks, you get to keep all the shiny pieces :-)

Yours,

Russ Magee %-)

[Jon Dufresne]

On Fri, Oct 17, 2014 at 4:45 PM, Russell Keith-Magee
<rus...@keith-magee.com> wrote:
> Achievable? probably. Easy? No. You're going to be digging in the weeds on
> this one.
>
> If I were trying to do this, I wouldn't try and get this to work with the
> existing FileField - I'd be writing my own Field definition. You can crib
> extensively from the existing definition, but everywhere that the current
> model points at it's own storage to get a filename, you use the foreign key
> to get the full file name.
>
> Essentially, it's going to be a merger of FileField and ForeignKey. I
> haven't done this myself, but my gut tells me this is in the realm of the
> "possible, but complicated".
>
> The other way that might be worth looking into is to follow the lead of
> GenericForeignKey. That's an example of a field that "looks" like a single
> column, but is implemented as two columns under the hood. You'll be looking
> to do a similar thing - write a "wrapper" field that points at the name and
> path fields to produce a composite "full file path" field. That composite
> field will be a lot closer to what FileField is expecting, so it might be a
> little easier to merge *that* with FileField than to merge FileField with
> ForeignKey. Then, you put the composite field on the StoredFile model, and
> traverse down the foreign key relation whenever you need to access the file.

Thanks for the feedback and suggestions.

I'll play around a bit and give it a shot.

My fear with this approach is third party applications with FileFields
will now now handle files differently. Luckily this doesn't affect me
at the moment, but may in the future.

Cheers,
Jon