Alaveteli cloud storage approach

[Laurent Savaëte]

Hi everyone,

I am in the process of migrating our files (raw emails+attachments) storage from local disk to cloud. I've setup the mirrored service as per https://alaveteli.org/docs/installing/storage/#mirrored-services, the cron jobs are running so that files older than 7 days are shifted to cloud. All is well, I think.

However I am wondering about 2 things:

- is it required to fully migrate to cloud storage? The docs (both alaveteli's and rails's https://guides.rubyonrails.org/active_storage_overview.html#mirror-service make it sound like the mirrored service is meant to act as a transition). If so, what is the right way to fully switch off local storage? I would assume tweaking the storage:promote script to move everything to cloud, not just files older than 7 days, and then updating the config to remove disk storage. I've not seen any script to do this in a somewhat atomic way. I could just stop the site at night for a few minutes, but wanted to check if there was a cleaner way.

- what do you recommend to replicate/backup files once in cloud? so far, I've been using restic to backup all files on disk to a remote storage, but this does not really work in a full cloud scenario.

Thanks for any tips!

Laurent

[Graeme Porteous]

Great to hear this is working for you.

It's not required to fully migrate to cloud services. We still run WDTK using the mirrored service. Eventually we will also be looking to fully migrate. When we do we will keep the local/mirrored options as they are so you can rely on those if you choose to.

Your plan to fully switch over seems good to me. There isn't a cleaner way at present.

As for backups, we use Amazon S3 and their S3 versioning to cover app/user level mistakes.

--
Graeme Porteous
gra...@mysociety.org

--
You received this message because you are subscribed to the Google Groups "Alaveteli Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alaveteli-de...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/alaveteli-dev/421c3c09-80cc-4c40-b8af-ba78d62eb3b4%40gmail.com.

[Laurent Savaëte]

Thanks for your answers!

About backups, I was actually also considering the risk of datacenters going up in smoke, etc... (we obviously never put bugs in our software ;0) it's rare, but it does happen.

 I'm not sure if AWS does implicit off-site replication of data on s3, but the provider we use (exoscale) offers this as an option. I will explore that route, so that at least 2 copies of each file exist in different geographic locations.

To view this discussion visit https://groups.google.com/d/msgid/alaveteli-dev/CAD%3D4AygJX-a8ipOkfe9KwSimdcS%3D_eoUVWX-W0RdMBwY0TtP3w%40mail.gmail.com.

[Graeme Porteous]

AWS offers a number of different options for data protection [1] and it is worth noting S3 has an 11 9s durability SLA [2] so chances of something going missing is very remote. Can't really advise of what is best for your data stored with different providers.

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/data-protection.html

[2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html

--
Graeme Porteous
gra...@mysociety.org

To view this discussion visit https://groups.google.com/d/msgid/alaveteli-dev/2200da40-7a06-4e4c-90ef-f4456d67b971%40gmail.com.