Solutions for using virtual server with multiple terabyte storage?

[Brian Gollands]

I'm currently running RS10.1 with Steve Bowman's s3_storage plugin on an AWS Lightsail server. I've been happy with this setup and am grateful to Steve for making the plugin. As I'm transitioning into retirement, I've been revisiting this setup in terms of passing along a system that is as simple and robust as possible.

Using a virtual server is the only viable alternative for my organization. Are there others out there who also use a virtual server with a different solution to being able to store multiple terabytes of assets? With my beginner's knowledge of AWS, there seem to be technical and economical constraints on having a local /filestore/ directory that is more than 100 or so GB. I'd really appreciate hearing what others have done in this situation.

Thanks,
Brian

[Dan Huby]

Hi Brian,

Personally I have always seen the S3 integration as unnecessary given you can mount an EBS volume and those can be up to 64TB - and also ResourceSpace's file architecture makes it easy to subdivide across multiple volumes.

I hope that helps, and happy retirement!

Dan

[Brian Gollands]

Dan --

Clearly I have to do more learning about EC2/EBS and RS with multiple volumes. Is there any documentation you can point me to regarding the latter?

Thanks!

Brian

[Dan Huby]

Hi Brian,

There's nothing specifically ResourceSpace - it's just mounting the volume then creating a symbolic link from "filestore" to a folder on that.

There's a guide to making EBS volumes available to Linux here:

https://docs.aws.amazon.com/ebs/latest/userguide/ebs-using-volumes.html

Dan

-- 
Dan Huby
Founder and Chief Technology Officer

Montala Limited
http://www.montala.com

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/KKY9yNAqa9k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/f1f47cbf-92aa-46f9-a648-4ba80dcaa724n%40googlegroups.com.

[Maria Youngman]

We are using EFS.  Amazon Elastic File System (EFS) provides a simple, serverless, and elastic file storage solution for AWS compute services, allowing you to share file data without managing storage capacity or performance.

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---

You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/ca8a48af-6764-4769-9bab-acf2e90caaebn%40googlegroups.com.

[Dan Huby]

Even better - as it can be accessed from multiple EC2 instances you could have a more distributed installation with several web servers.

You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/KKY9yNAqa9k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/CAA5XHRP7LG8wSqX6wP%2B%2BM4Rms_dmV4rr78fE%2BtiaT%3DvEe2zncQ%40mail.gmail.com.

[Maria Youngman]

Certainly, but for simplification and cost in this case, the application is for a small group so we are using only one EC2 Ubuntu instance and AWS Backup.

[Brian Gollands]

Thanks for the suggestion, Maria. 

I had looked into EBS after Dan's suggestion the other day. I think the cost on that puts it out of our reach (we're a struggling non-profit science research institution), but I didn't know about EFS. 

While I really appreciate the ability to quickly spin up virtual servers in AWS and try different things, the number and complexity of options available in the AWS world is daunting. The TCO on EFS looks good, but as I've read, one has to be careful with how your cloud is set up or it can get very expensive. As a self-taught IT manager, I'm going to have to do more research to figure out the pros and cons for our situation. One of the advantages of using S3 is that, conceptually, it's relatively easy for me to understand what's going on. But there is no dearth of documentation from AWS on EC2 and EFS, so I'll go back to school. :-)

[Brian Gollands]

An aside to Dan Huby -- Thanks for making a great product open source for all of us non-profits AND being so active on this forum. Much appreciated!

[Maria Youngman]

Hi Brian,

When you create an EC2 instance, you also create a volume (EBS). Then move your local filestore (volume) to a mounted drive (EFS) using a symbolic link. Basically, we are using an EC2 instance t3.small created from an AMI Ubuntu Server 24.04, the EBS volume is 60 (GiB) and  EFS mount filestore unencrypted (since ResourceSpace already encrypts data).

To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/b40288e4-22b4-474e-ab55-f2faf130104cn%40googlegroups.com.

[Dan Huby]

ResourceSpace doesn't encrypt data - it stores files as-is.

If you want encrypted data you have to do that at the file system level.

To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/CAA5XHRNUVad1Kj5zjogu-r%3D3qwB9fFDwLhjo0tqbubS0Q47sxw%40mail.gmail.com.

[Maria Youngman]

Hi Dan,

Apologies for the confusion. It looks like ResourceSpace uses a form of obfuscation to organize files, likely for structural and performance reasons. Since we don't require encryption, we're not encrypting the files stored on EFS.

[Brian Gollands]

Maria and Dan --

Thanks for this. Maria, you make it sound so simple :-).  We don't need encrypted data either, so that's not a problem. Will try out your suggestion. 

[David Dwiggins]

EBS storage is substantially more expensive than S3 storage, though - in USD, baseline of $0.022/GB/Mo vs $0.05/GB/Mo, and it gets even cheaper if you use tiering on S3. At scale this can start to make a difference. for 35 TB of data would be $1,750/month vs $770/month, or almost $12,000 per year more expensive even without tiering.

We maintain our primary repository on premise on network attached storage, and then mirror it to S3 tiered storage for backup and use in online apps. But I do think native support for object storage (at least for original resources and larger derivatives) would be a worthwhile feature.

-David

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/903ad164-b333-4c1f-afde-402349690c8cn%40googlegroups.com.

[Brian Gollands]

@David -- You're right on the Standard pricing of EFS and it is out of the question of us. However, the Infrequent Access (IA) price 0.016/GB is more attractive. I'm currently experimenting with a test instance linked to a EFS volume set to a lifecycle of 1 day (after which it goes to IA). It's not clear to me what the consequences of IA are in terms of response by RS. Given our small staff and what I know will be relativelyl infrequent use, my hope is that much of the file store can sit in IA with primarily the thumbs being what sits in Standard. Again, I haven't yet found the doc that describes what exactly IA means for performance, so I'm unsure of my theory.