Existing file system migration

[Stefano Brentegani]

Hi.

This must have been asked many times already, but I can't find where.

We're moving from an existing django-cms implementation using django-filebrowser to django-filer, and I'm trying to figure out the best way to populate the database.

Resources are on a local file system, and most of them are just referenced in HTML pages and mapped by the web server, so I'm looking for a way to:

1. import existing files "in-place", i.e. just insert relevant data in the database by reading them from the file, without overwriting or copying it, preserving the folder structure relative to the root
2. migrate filebrowser fields: this is just a matter of adding a new field, fetching the right Image and finally removing the legacy field and fix field names (and possibly indexes)

Step 2 should be fairly easy to implement within a South data migration (surrounded by schema migrations to add filer's fields and remove filebrowser's).

Now I'm stuck on step 1, because I can't find a way to create a new Image or File without having it fed to the storage and saved as a new file.

I understand that Filer's storage management is much more generic than a basic file system (so, for instance, checking upon path collision whether an existing file is an exact copy of the one I'm saving might not always be easy), so I'm considering writing data directly to the database (I know, it stinks, that's why I'm here). Alternatively we could just import all the files (using Filer's management command) and then just swap the root folders both on the file system and in Filer settings. I'd rather not have to modify folder mappings on the web server.

Has anyone already gone through this? How did you do it? How would you do it?

Thank you.

Stefano

[Stefan Foulis]

Hi Stefano

Unfortunately we don't have a ready solution for this. There is a management command that will import files from a given folder structure (https://github.com/stefanfoulis/django-filer/blob/develop/filer/management/commands/import_files.py). But this will copy the files into it's own structure (using date based folders). That does not seem to be what you want.

1)

You could adapt this command to keep the files in place instead.

With the default storage configuration Filer uses /media/ as the root and automatically places new files into the "filer_public" subfolder with a date based folder structure. If your files are already in /media/ somewhere you could add the files to filer by just creating new "filer.models.File" instances and assigning the file path relative to MEDIA_ROOT as a string to filer.models.File.file.

But keep in mind that some operations in filer (like changing the storage backend) will move the physical file to the filer folder structure.

2)

South is the way to go for your own models with file fields. Migrating cms plugins to their cmsplugin-filer versions will be a bit more tricky, because they have slightly different features. Also you'll have to create a new cms plugin in the same location as the old one and then delete the old one.

If you find a re-usable way to solve these steps, I'd be glad to integrate this somehow. File import would be something to extend the management command to django-filer with.

2) would better be suited as a utility in cmsplugin-filer.

Cheers

Stefan

[Borgstrom]

Hi Brente,

Wondered what your solution to this was? I'm about to go through a similar migration and would love to hear any pointers or lessons learned from you.

LMK

Thanks!

EB