s3_storage Plugin v1.5 for RSv10.1

[Steve]

An updated v1.5 s3_storage plugin enabling Simple Storage Service (S3) storage of resource original files is available for ResourceSpace v10.1.

Plugin Changes
Plugin v1.5 is for ResourceSpace v10.1.
The SDK has been updated to v3.261.1 in the plugin ../lib files.
Updated S3 bucket owner check on the Admin/System/S3 Installation Check page.

Hope this updated plugin helps the ResourceSpace community.

Thanks, Steve

[Mike Perry]

Steve -- while I currently have no need for the capabilities of this plugin I want to thank you for your efforts and contributions.

👏

[Roger Howard]

Hey Steve,

Thanks for all your work on this! 

Would you be interested in some help moving hosting of this plugin to Github or elsewhere? I'm keeping a private repo of this project, but it'd be great if there was an official repo somewhere. I'd be more than happy to help you in any way to make this happen, and it'd definitely make it a lot easier for me and others to contribute.

Thanks again!

On Wednesday, March 1, 2023 at 1:55:55 PM UTC-5 Steve wrote:

[Steve]

It would be helpful to have a location for community plugins, say in GitHub.  I am ok with someone adding the s3_storage plugin to such a community space.

Thanks, Steve

[Justin Snair]

Steve,

Thanks so much for this awesome plugin. Looking in to migrating a system from local filestore to S3 and I notice in the README for v.15 of the plugin that you mention not to use the filestore_to_s3.php script as they still need further updates. Can you share what isn't working?

[Billy Jack F]

Steve,  I am getting a "FAIL" on S3 admin page for "Is the local storage directory set ($storagedir)?"

I checked the path hundred of time and it's still showing failed and now I am scratching my head on this one. Could you check the add ons that I made to my config.php file below and see if I am doing it wrong?

Everything passed or looks good on the admin_system_s3.php page except the storagedir set which is showing a fail. 

My RS info:

ResourceSpace version: SVN 10.1 r22253

S3 Plugin version 1.5 (s3_storage_v1.5.zip)

config.php:

# S3 Plugin
$storagedir="/home/usernameeditedout/public_html/filestore";
$storageurl="https://domainnameeditedout/filestore";
$exiftool_write = true;
$exiftool_write_metadata = true;
$custompermshowfile = false;

Thank you,

Bill

[Billy Jack F]

Will the Filestore to S3 script be updated soon?

Thank you for the plugin

Bill

[Steve Bowman]

I suggest removing the $storageurl="https://domainnameeditedout/filestore"; or commenting out via //$storageurl="https://domainnameeditedout/filestore";

Will try and update the filestore_to_s3.php script in the next few weeks.  Unfortunately, have a lot going on lately.

Glad to have others be able to use the plugin.  It has saved us significant cost with the Utah Aerial Imagery Database (https://imagery.geology.utah.gov) with over 400,000 air photos and related items in the database. 

Steve

Steve D. Bowman, Ph.D., P.E., P.G. Geologic Hazards Program Manager O: (801) 537-3304 E: steve...@utah.gov Utah Department of Natural Resources Utah Geological Survey geology.utah.gov
The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion so that we can ensure such a mistake does not occur in the future.

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/UhNO16_a89I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/33872d38-cb5f-4c06-8a76-6f63b2c6b800n%40googlegroups.com.

[amanda.chan]

Thank you again Steve for all your work on this plugin!!!

[amanda.chan]

Hi Steve,

If I implement your plugin after I've already started my library, do I need to replace files for the images/videos I've already uploaded for them to get to the S3 bucket?

Thanks!

On Wednesday, March 1, 2023 at 12:55:55 PM UTC-6 Steve wrote:

[Steve Bowman]

Yes, if you start using the s3_storage plugin, it will not automatically move existing original resources in the filestore.  The filestore_to_s3.php script will do that; however, it needs to be updated for RS v.10.2.

Steve

Steve D. Bowman, Ph.D., P.E., P.G. Geologic Hazards Program Manager O: (801) 537-3304 E: steve...@utah.gov Utah Department of Natural Resources Utah Geological Survey geology.utah.gov
The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion so that we can ensure such a mistake does not occur in the future.

--

ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/UhNO16_a89I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/c4488592-4fb3-4051-b44a-a335c57aa168n%40googlegroups.com.

[amanda.chan]

Hi Steve,

Are the originals supposed to be located on the local and the S3 filestores? I was under the impression that the originals were not kept on the local filestore. Thanks as always!!!

[Steve]

Yes, the original files are only stored in S3; however, there is a zero-byte original placeholder file in the usual filestore folder. 

Steve. 

[Seth Gitner]

@stevebowman if the files are being put on the s3 but still exist on the application server does that mean my resource space is out of date? I am also getting
Is the local storage directory set ($storagedir)? FAIL

where can I fix that?

I want all of my images to exist in was only.

thank you.

Seth

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/302a28f0-c4a8-40f1-9848-112335208668n%40googlegroups.com.

[amanda.chan]

My originals are also staying on the server. I don’t get that error but I get: Undefined index: DisplayName

Seth, do you mind sharing your settings for ffmpg preview creation? Mine makes the video file larger than the original. Thanks!

[Chris]

I also receive that "DisplayName" Error when checking the S3 Installation.
Is it because I do not use RS 10.1 but 10.3?

[Steve Bowman]

Yes, it is likely due to using RS v10.3, as the current plugin was designed for v10.1.  A new s3_storage plugin for RS v10.3 should be ready in a week or so.

Thanks, Steve

Steve D. Bowman, Ph.D., P.E., P.G. Geologic Hazards Program Manager O: (801) 537-3304 E: steve...@utah.gov Utah Department of Natural Resources Utah Geological Survey geology.utah.gov
The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion so that we can ensure such a mistake does not occur in the future.

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/fbe101db-acba-42a1-bb6f-5b7cafec4d76n%40googlegroups.com.

[Livia Neagu]

Hi Steve,

Thank you so much for providing all the information regarding the S3 Storage Plugin!

I've been trying to upload files to AWS S3 for a few days now, but haven't been successful yet. I'm currently using RS version 10.4, and I wanted to ask if there's a way to connect ResourceSpace to AWS with this version. If not, could you please let me know where I can find the latest version of the S3 Storage Plugin that might work?

I appreciate your help!

Best regards,
Livia

[Tim L]

As far as I know, there is not updated version yet. I have tried to change the code myself and make the plugin working on 10.3, but I failed.

We are using s3fs right now on Linux. It basically mounts the S3 in a folder on the system level. ResourceSpace just doesnt even know, that its on S3. We are handling 700GB so far. People said, that this solution might have performance issues, but so far, we are quite happy with it. :)

I see if I can find some time to help Steve with the plugin update.

[Steve Bowman]

An updated version of the s3_storage plugin for ResourceSpace v10.4 is now available:

 rs_v10.4_plugins_9-4-2024.zip

Thanks, Steve

Steve D. Bowman, Ph.D., P.E., P.G. Geologic Hazards Program Manager O: (801) 537-3304 E: steve...@utah.gov Utah Department of Natural Resources Utah Geological Survey geology.utah.gov
The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion so that we can ensure such a mistake does not occur in the future.

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/1ac5eed5-81b7-4710-80c0-7cb20cd2ab03n%40googlegroups.com.

[Adegoroye Owolabi]

Mounting Amazon S3 for external storage

https://www.resourcespace.com/knowledge-base/systemadmin/s3storage

Has anyone here been able to follow this documentation successfully?

Does this work, Is there any issue to be aware of?

[Steve Bowman]

The s3fs connector can be used, but is not always reliable in a server environment.

Steve

Steve D. Bowman, Ph.D., P.E., P.G.

Senior Scientist/Geologic Hazards

O: (801) 537-3304 E: steve...@utah.gov Utah Department of Natural Resources Utah Geological Survey

geology.utah.gov

The content of this email is confidential and intended for the recipient specified in the message only. It is strictly forbidden to share any part of this message with any third party without the written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion so that we can ensure such a mistake does not occur in the future.

To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/acddd831-18d3-4073-886c-d64d9a39d444n%40googlegroups.com.

[Brian Gollands]

Steve (or anyone else who has successfully installed the latest version) --

Sorry to ask a dumb question, but as I've had no trouble installing your plugin for older versions of RS, I want to make sure that I've not making a stupid assumption that's causing me problems now. I downloaded the new version from the link you provided above and copied the /rs_v10/plugins/s3_storage/ folder to the plugins folder in my fresh install of rs10.4 running in ubuntu24. I then copied /rs_v10/plugins/s3_storage/copy.php and /rs_v10/plugins/s3_storage/download.php to the places mentioned in your readme.txt, ignoring /rs_v10/plugins/s3_storage/crop_new1.php and any other copy.php and download.php files located within the parent rs_v10 installation from your zip archive. After activation and configuration, my S3 installation check is OK, but I get blank thumbs and no downloading action when clicking the Download button. Nothing helpful from the log. Am I doing something obviously wrong?

TIA,

Brian

[Adegoroye Owolabi]

Could you please share the issue you encountered during installation within your server?

[Brian Gollands]

I went back and retraced my steps and still am not able to get Steve's S3 plugin working. Here's what I've done:

1. installed RS10.4 in an AWS instance of Ubuntu 24.04 LTS with PHP 8.3 using the instructions found at https://www.resourcespace.com/knowledge-base/systemadmin/install_ubuntu. However, I did not install a MySQL read-only username or the cron jobs.
2. without adding in the plugin files (i.e., a base install of RS10.4), I have a working DAM that uploads and download files. The installation check is all OK.
3. I then copy in the s3_storage plugin folder to /plugins/ and _move_ the included crop.php file to /plugins/transform/pages/ and the included download.php to /pages/.
4. after activating the plugin and configuring for the s3 bucket, the S3 installation check is OK.
5. I then find that all thumbnails have been replaced with differently colored uni-colored rectangles and the intermediate images on the Details page are missing. New uploads appear the same way and clicking the Download button results in no downloading action. However, the originals of the new uploads _do_ show up in the bucket.
6. Uninstalling the S3 plugin (i.e., deactivating it and copying back in the originals of crop.php and download.php to their original places), results in the thumbnails and intermediate images returning, as well as the normal functionality of uploading and downloading.

I'm willing to believe that I'm doing something wrong, but can't see what it is.

Am I wrong in thinking that this should work in Ubuntu 24 with PHP 8.3? Does anyone else have the plugin working with these versions?

Thanks for any suggestions,
Brian

[Steve Bowman]

It is just a general issue, not something specific.

Steve

To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/f8cd83ca-1d4a-457b-98af-70eca0ff5008n%40googlegroups.com.