Migrating from local file store to Amazon S3

[Evgeniy Skidan]

Hi, all!

Can anyone share the rake task or some thoughts how to move existing files to S3 bucket?

Thanks.

[Ross Gailer]

I just finished moving servers - both were local so I don't know if this applies to moving from local to Amazon, but this is what I had to do.

After Canvas install is complete,  shut down the apache service and the canvas_init service on the local machine, then restart the postgresql service.

  

This will stop all access on the DBs and the site so that nothing will change during the move

Dump the local DBs.

  

Create a tar of the tmp/files

  

Copy both the dump files, and the tar file to the new location - in your case Amazon.

Copy the security.yml file from <canvas install>/config and replace the existing one in the new install.

Make sure that no canvas processes are running on the new install. Not knowing Amazon, I can't tell you how to do that..

Drop the existing canvas_production and canvas_queue_production databases on the new install and re-create them blank.

Load the data from the dumps into the databases.

Unpack the tar file and move the files into the correct position that they came from.

Restart web services and canvas_init.

Hope this is some assistance with the process.

Presbyterian Ladies' College Armidale

Locked Bag 5

ARMIDALE NSW 2350

ABN:79 017 381 796

CRICOS provider code: 02295G

Tel:    +61 2 6770 1700 

Fax:  + 61 2 6772 5697

http://www.plcarmidale.nsw.edu.au

In the interests of environmental sustainability, PLC is endeavouring to communicate electronically where possible.

The contents of this email are confidential. Any unauthorised use of the contents is expressly prohibited. If you have received this email in error, please advise by telephone (reverse charges) immediately and then delete/destroy the email and any printed copies. Thank you.

[Evgeniy Skidan]

Ross,thanks for help, but migration to S3 is little deeper than just file copy.

[James Royal]

Hey Evgeniy,

Were you able to resolve this issue

[Evgeniy Skidan]

Yes, I did.

03 мая 2014 г. 22:09 пользователь "James Royal" <jcro...@gmail.com> написал:

--

---
You received this message because you are subscribed to a topic in the Google Groups "Canvas LMS Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/canvas-lms-users/Vh_xYj2J8uk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to canvas-lms-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Royal]

Hi Evgeiny,

Could I kindly bother you to share how you handled this issue? I am currently faced with the same problem where I used the tmp/files directory for storing files in my production environment and would now like to migrate over to S3 storage. Any help would be extremely appreciated. 

Thanks In Advance, 

[Evgeniy Skidan]

AFAIR, I just wrote simple ruby script which reorganized tmp/files folder for me into S3 structure, then I just upload result into S3 bucket, that's all. Worst thing is that I can't find this script, so you need to write your own, sorry for this. Let me know if I could help you somehow.

--
Evgeniy Skidan

[James Royal]

I know you did not mention this but did you have to worry about database entries at all ? Or is simply reconfigure the file structure of tmp/files directory enough to handle the massive move? My understanding is as follows:

1. Reconfigure tmp/files to represent S3 storage

2. Move all files from tmp/files to S3 storage

3. Update canvas/config settings to point to the S3 files

4. Restart web server and delayed jobs

Would you say this is the approiate set or steps? I apologize for my ignorance. I simply do not want to blow up my canvas installation :).  

Thanks again,

[Evgeniy Skidan]

On 7 May 2014 22:29, James Royal <jcro...@gmail.com> wrote:

I know you did not mention this but did you have to worry about database entries at all ? Or is simply reconfigure the file structure of tmp/files directory enough to handle the massive move? My understanding is as follows:

Simple reconfiguration of the file structure enough. As you can see database entries in "attachment" table don't contain type of the file storage, so you don't need to touch database at all for migration.

 

1. Reconfigure tmp/files to represent S3 storage

2. Move all files from tmp/files to S3 storage

3. Update canvas/config settings to point to the S3 files

4. Restart web server and delayed jobs

Would you say this is the approiate set or steps? I apologize for my ignorance. I simply do not want to blow up my canvas installation :).  

Steps look good, but I suggest to make a snapshot and launch temporary staging environment from it to avoid total blow up :)

--
Evgeniy Skidan

[joshk...@gmail.com]

Can you explain what you mean by "Reconfigure tmp/files to represent S3 storage"?

Thanks.

[Eugene Vlasov]

Hi Community!

This is an old thread, but I will reply anyway. I got into this problem recently when I wanted to migrate files from local storage at tmp/files to Amazon S3 storage.

On local file system Canvas stores files and attachments like this:

tmp/files/0000/0001/file_name.xyz

tmp/files/0000/0002/...

tmp/files/0001/0001/...

tmp/files/0001/9999/...
and so on

On Amazon S3 file structure is different. It looks something like this (inside s3 bucket):

account_1/attachments/XXXX/file_name.xyz

account_1thumbnails/XXXX/...

Where XXXX is attachment id. Attachment ID is created from concatenation of two digits from local storage path. This way tmp/files/0000/0001/file_name.xyz becomes account_1/attachments/1/file_name.xyz and tmp/files/0001/9999/file_name.xyz becomes account_1/attachments/19999/file_name.xyz

As we can see from the above Canvas files cannot be copied as-is from local storage to Amazon S3 storage, but file structure should be converted.

Hope this helps others who get stuck over this problem.

Yevhen Vlasov

https://www.webhostingzone.org/