Where are stored static item in EDX Platform?

[Nacho Diaz Asenjo]

Hi!

  I'm looking for all files that i uploaded in "Files & Upload" Section of my course ... but i can't find anything.
 
  I thought that /edx/var/edxapp/data/<COURSE> location would be correct size to find them ;-(

  They are stored in mongo? Or in mysql?

Thanks

[John Eskew]

Those assets are stored in MongoDB using GridFS. If you connect to your mongo instance and use the edxapp database, you'll see the fs.files and fs.chunks collections. The asset file metadata is stored in fs.files and the actual bytes are in fs.chunks.

[Moreno Pinheiro]

Hello.

A curious related issue happened when I tried to backup (mongodump) and restore (mongorestore) those files to a new ubuntu 12.04 server installation of aspen.1 release.

I had no problem running the configuration instruction (edX-Ubuntu-12.04-64-bit-Installation) and restoring the mysqldump backup but when I restored the mongo backups most of the courses images and files content didn't show up (404 code) however I can see they are on mongo database collections.

But the curious thing is I manually uploaded a course image file (I had the same image from the old server) so it works the image shows up (as usual). Then I dropped the edxapp mongodb (complete deleted) and restore the mongodump database backup once again then what happened was the course image started to show up but pointing from the old uploaded file (since the new one I did was complete deleted). Seems like that when I upload the same image file it restored the "link" (or something).

Can someone help me to understand how to get all my courses images/files restored?

Thanks,

[John Eskew]

I've heard another report that a MongoDB backup/restore of GridFS (where the assets are stored) makes all the assets inaccessible. I haven't yet done the research into why this happens. But a way to get your course assets working again are to export the course via Studio from an existing instance and then import the course via Studio into your new instance.

[Moreno Pinheiro]

Is there a way (script or command line code) to backup/restore all courses? Thanks.

[Moreno Pinheiro]

Sorry guys, I couldn't reproduce it (the curious thing). My new suspicious are the MongoDB (2.4.7 / 2.6.6) pymongo (2.4.1 / 2.7.2) version difference.

[Carson Gee]

I’m glad it worked out, but as an FYI, there is a quick command for exporting all your courses to disk. We use this to back up all our courses to a private git repo with a cron task script that looks something like https://gist.github.com/carsongee/9f64e40656c27885ab61. And using a simple crontab for root that looks like:

12 4 * * * /edx/app/course_git_backup/course_git_backup.sh > /dev/null

The notable command in the platform management commands is cd /edx/app/edxapp/edx-platform; /edx/bin/python.edxapp manage.py cms --settings=aws export_all_courses
We have an ansible role to do deploy all this, but we haven’t generalized it enough to open source it yet.

​

[Moreno Pinheiro]

Thank you Carson.

My version of "edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_all_courses.py" is a little outdated so I had to change line 37 from

37                     course_dir = course_id.replace('/', '...')

to

37                     course_dir = course_id.to_deprecated_string().replace('/', '...')

Now that I have all course exported do you know an easy way to import all of them? :)

[Carson Gee]

The import command can import an entire folder filled with courses, so just something like:

sudo su - edxapp -s /bin/bash
. edxapp_env
python manage.py cms --settings=aws import /path/to/export/dir

should work.

​

[Roberto Wills]

Hi Carson Gee, I export all courses and then i tried the commands you said about importing all courses, I point it to the folder where i put the backups of all my courses. But i got an error:

Here's the result:

Traceback (most recent call last):

  File "manage.py", line 15, in <module>

    from safe_lxml import defuse_xml_libs

ImportError: No module named safe_lxml

[Carson Gee]

This seems like you have a platform installation issue. I’m thinking you need to rerun /edx/bin/update, or just making sure requirement/edx/local.txt is reinstalled. There are several threads about that error on the code list and SO.

​

--
You received this message because you are subscribed to the Google Groups "General Open edX discussion" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/00562385-6449-4da8-9933-afae74a1c40a%40googlegroups.com.