Re: TESTERS NEEDED! Amazon S3 File Store Support

[Troy Yeager]

I’d like to hear where this is at as well. I had gotten it to work with a single bucket, but I was hoping to use it across multiple buckets and was never able to get that to work correctly.

Thanks,

Troy

On Jul 27, 2022, at 10:42 AM, 'Colin P' via ResourceSpace <resour...@googlegroups.com> wrote:



Hi, I see this thread hasn't had any posts in awhile. Is S3 file store support working? I'm setting up a new install for a client and I'd love to use S3 for the storage.

On Monday, August 2, 2021 at 6:24:57 PM UTC-5 nae...@gmail.com wrote:

Another note to add here. I transferred images to S3, and then transferred by installation and previews to an EC2 instance on amazon. One thing that got me during that process was after I loaded the database and files into EC2 I had a persistent mkdir() permissions issue. Turns out the config_json field on the plugins table in the database for s3_storage has the filestore location stored, which was being used to supply the $storagedir variable, which was throwing the error. So, if you are moving your installation you will want to go into the database and edit this manually.

On Sunday, August 1, 2021 at 1:31:46 PM UTC-4 Andrew McCullough wrote:

Also, if you have a big filestore, don't forget to increase your $php_time_limit in config. I just set mine to 0 (no limit) and will change it back when I'm all done.

On Sunday, August 1, 2021 at 1:17:11 PM UTC-4 Andrew McCullough wrote:

A couple of notes for those coming after.

I managed to get filestore transfer tool working. A few things:

- variables in the migration script are called things like $s3_bucket. In the configuration file, the names of these variables has been updated to $s3_storage_bucket, so you will need to change the name in the migration script for it to work.

- I had trouble with file extensions on original files. Images with .jpeg extensions would result in the wrong file path and were therefore not found when doing the upload. Apparently, this is just one of the limitations of the get_resource_path() API function? So in order to pass an explicit file extension to get_resource_path, I performed a get_resource_data on each $ref (this same change has to be made in step 2 and step 3):

// Build array of the resource original file and alternative files to upload to a S3 bucket.

$ref_original[0]['ref'] = 0;

$ref_original[0]['file_extension'] = get_resource_data($ref)['file_extension']; #2021-07-31 ADDED file extension for original resource

$ref_files = array_merge($ref_original + $alt_files);

// Loop through resource original files and upload to S3.

foreach($ref_files as $file)

    {

    // Setup AWS SDK S3 putObject parameters.

    if($file['ref'] == 0)

        {

        $s3filepath = get_resource_path($ref, true, '', false, $file['file_extension']); #2021-07-31 ADDED FILE EXTENSION to FIX FILE COULD NOT BE FOUND ERRORs

        $file_output = 'Uploading original file (';

        $file_info = 'Original file: ';

        }

    else

        {

        $s3filepath = get_resource_path($ref, true, '', false, $file['file_extension'], true, 1, false, '', $file['ref']);

        $file_output = 'Uploading alternative file (';

        $file_info = 'Original alternative file #' . $file['ref'] . ': ';

        }

 

The script did not seem to work with any alternative files, which really didn't matter with my implementation, but if alternative files are important to you then it might be something you want to look into.

A final note. Each time I ran the script, it started uploading files from the beginning. I think the intention is that the script will skip through resources that already exist in the S3 bucket, but that wasn't the case for some reason. No matter, but something to think about. I ended up hard-coding starting ref numbers on subsequent runs to avoid repeat uploads when I was debugging.

I hope this helps someone in the future!

Best,

Andrew

On Thursday, July 29, 2021 at 12:37:09 PM UTC-4 Steve wrote:

Updated ../hooks/upload_plupload.php to resolve issue with the Upload Log displaying twice.  Also added installation instructions.

On Wednesday, July 28, 2021 at 4:21:08 PM UTC-6 Steve wrote:

Glad it is working ok. 

On Tuesday, July 27, 2021 at 6:25:40 PM UTC-6 nae...@gmail.com wrote:

Agreed with Brian, thank you for your work on this--it is so great!

On Tuesday, July 27, 2021 at 5:49:58 PM UTC-4 bgol...@gmail.com wrote:

Steve --

Wow! Christmas came early! Thanks very much, Steve!

Brian

On Tuesday, July 27, 2021 at 10:41:49 AM UTC-4 Steve wrote:

The plugin is now available in GitHub: https://github.com/sdbowman/ResourceSpace-S3-Storage-Plugin.

On Tuesday, July 27, 2021 at 7:45:13 AM UTC-6 Steve wrote:

Brian- The attached ZIP file contains the latest S3 storage plugin that works with ResourceSpace v9.6.  I suggest using this plugin, rather than the older branch that is not up to date.  Place the plugin in the ResourceSpace ../plugins folder with the default plugins.

Thanks, Steve.

On Monday, July 26, 2021 at 4:35:59 PM UTC-6 bgol...@gmail.com wrote:

I'm new to RS but have a working installation up. What I really need for our institution is to use it with S3 storage. Before attempting this, I want to confirm that I'm on the right track:

1. I gather that, even with the new RS 9.6, Steve's way is the only way to enable S3 support. Thus, rather than install the latest distro from RS, I need to install the one from the following? https://svn.resourcespace.com/svn/rs/branches/sbowman/20201026_sbowman_s3_3/

2. What version of RS is this branch based on?

Thanks!

Brian

On Saturday, January 16, 2021 at 12:08:24 PM UTC-5 nae...@gmail.com wrote:

This is great and super timely for us (as we are moving a bunch of stuff over to AWS). I tested the S3 connection and so far it is working super well with uploads, download, and all the rest. A strange thing about the ResourceSpace installation from the branch though is that my resource view pages stop rendering after the download region:

Not sure why this is the case and can't find anything in the logs, user is super admin, etc. etc.

Sorry, this probably isn't a very helpful contribution but I thought I would put it out there in case anyone else is encountering the same...

Andrew

On Monday, November 23, 2020 at 12:37:48 PM UTC-5 Steve wrote:

The SVN branch https://svn.resourcespace.com/svn/rs/branches/sbowman/20201026_sbowman_s3_3/ contains the latest S3 object-based storage integration.  The PDF at the branch root contains an overview, setup, and how the integration works.

Steve

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/JT833klfwjc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/e35c5293-49a9-45ad-957e-e8eddc77144en%40googlegroups.com.

[Steve]

Version 1.3 of the s3_storage plugin is attached and works with ResourceSpace v9.8.  I need to take a look at RS v10.0 and see if any changes will be needed.

Thanks, Steve

[Steve]

I also need to update the filestore migration tool, so suggest trying in a testing environment only (not on a production level system).

Steve

[Brian Gollands]

Steve --

Is the v1.3 zip you attached complete? To upgrade from v1.2, do we just drop it in /plugins/ as a replacement for the old s3_storage folder? Previously, there were files like 
collections_functions.php and collection_download.php that needed to be copied over the originals, too. Are those unchanged from v1.2? Thanks.

[Brian Gollands]

I've installed Steve Bowman's S3 storage plugin in the following:

OS: Ubuntu 20.04
PHP: 7.4.3
MySQL: 8.0.30
Storage: AWS S3

with 2 installations:
1. ResourceSpace v9.7 and s3_storage v1.2
2. ResourceSpace 9v.8 and s3_storage v1.3 (used the v. 1.2 replacement files for /transform/, etc. with the v1.3 plugin he links to above)

Uploading works but downloading from Resource View does not. In both installations, I get the following error in the apache error log after clicking one of the Download buttons. This one is from clicking the original image download button (IP addresses obfuscated):

------------------
[Wed Sep 28 20:34:15.734064 2022] [php7:error] [pid 132772] [client XXXXXXXXXXX:51327] PHP Fatal error:  Uncaught ArgumentCountError: Too few arguments to function HookS3_storageDownloadDownload_resource_extra(), 1 passed in /var/www/html/resourcespace/include/general_functions.php on line 3569 and exactly 2 expected in /var/www/html/resourcespace/plugins/s3_storage/hooks/download.php:14
Stack trace:
#0 /var/www/html/resourcespace/include/general_functions.php(3569): HookS3_storageDownloadDownload_resource_extra('/var/www/html/r...')
#1 /var/www/html/resourcespace/include/general_functions.php(3655): hook('download_resour...', 'download', Array, false)
#2 /var/www/html/resourcespace/pages/download.php(216): hook('download_resour...', 'download', Array)
#3 {main}
  thrown in /var/www/html/resourcespace/plugins/s3_storage/hooks/download.php on line 14, referer: http://XXXXXXXXXXX/resourcespace/pages/download_progress.php?ref=1&size=&ext=jpg&k=
------------------
 
Note the following:
- downloading from the Image Tools DOES work.
- The RS installation check passes, as does the plugin's installation check.

Has anyone else seen this? Or have an idea what the problem is?

Thanks for any suggestions!

Brian

[Steve]

Working on RS v10 compatibility now. Error above is from the hook function; will be fixed in the v1.4 plugin shortly. 

Thanks, Steve

[Brian Gollands]

That's great! Thanks, Steve!

Brian

[Regis]

Trying to set up to test this awesome plugin but am getting this error:

PHP Fatal error:  Uncaught Error: Call to undefined function sql_query() in /var/www/html/plugins/s3_storage/hooks/upload_batch.php:34\nStack trace:\n#0 /var/www/html/include/general_functions.php(3613): HookS3_storageUpload_batchUpload_original_extra(12)\n#1 /var/www/html/include/general_functions.php(3699): hook('...', '...', Array, false)\n#2 /var/www/html/pages/upload_batch.php(689): hook('...', '...', Array)\n#3 {main}\n  thrown in /var/www/html/plugins/s3_storage/hooks/upload_batch.php on line 34, referer: https://files.media-graph.com/pages/upload_batch.php?replace=&batch_replace_min=0&batch_replace_max=0&batch_replace_col=0&alternative=&collection_add=&resource_type=1&no_exif=&autorotate=&replace_resource=&archive=&relateto=&filename_field=&keep_original=&replace_resource_original_alt_filename=&single=&status=0&k=

[Colin P]

Hi Steve,
How is the 1.4 plugin coming along? Jut installed a v10 instance for a client and would love to use this plugin.

Thanks!

Colin

[amanda.chan]

Also hoping to install your plugin! Do you need more testers?

[Steve]

Sorry for the delay in getting the s3_storage plugin working with RSv10.0.  A few other priorities came up lately, and having a few difficulties with the changes present in v10.0 in getting the plugin working correctly.  I may be able to get something by late Sunday for testing.

Thanks for understanding, Steve.

[Steve]

An initial v1.4.0 of the s3_storage plugin is attached that has compatibility with ResourceSpace v10.0.  

Changes:

• Compatibility with RS10.0.
• New streaming ZIP downloader.
• Error with the Transform plugin is fixed.
• S3 SDK updated to the latest version.

Install:  Copy the s3_storage folder to the ResourceSpace ../plugins folder, replace the download.php at ../pages, and replace the crop.php file at ../plugins/transform/pages.  Configure the plugin using the Manage Plugins page.

Any testing would be greatly appreciated.  Highly suggest to not test on active, production systems in case there is a problem with the plugin.

Thanks, Steve

[Colin P]

The 1.4.0 plugin seems to work well. As I mentioned in an email to Steve, I do get a "FAIL" message for S3 Bucket owner and S3 Bucket owner ID in the plugin check. BUT, the plugin seems to still work fine. Files are uploaded to my S3 bucket successfully, and deleting them also works.

Thanks for the awesome plugin!

[amanda.chan]

Hi Steve,

I successfully installed your plug-in to version 10.1. In the installation check I also receive an error for S3 Bucket owner and S3 Bucket owner ID. But I can upload to ResourceSpace and it is saves in my S3 bucket immediately.

This is probably a stupid question, but after 90 days the file will be deleted from my local "temp storage" filestore and remain in the S3 bucket filestore, is that correct?

Thanks for this awesome plugin! After some more playing around I plan to implement for my work's instance.

On Sunday, November 27, 2022 at 9:44:29 AM UTC-6 Steve wrote:

[Steve]

Not sure why the plugin is showing an error on the check page for the bucket owner, as I have not been able to reproduce. 

Uploaded original files are immediately uploaded to S3 storage, it is just the various preview images and a zero byte placeholder original file that are stored in the filestore. 

Hope this helps, Steve. 

[amanda.chan]

Yes, that makes sense. Thank you! It is working perfectly except for that error for S3 Bucket owner and S3 Bucket owner ID. But that is not a big deal.Thank you so much for the great plug-in!

[Steve]

Glad the plugin is working well for you. It certainly helped solve an expensive storage issue for us with large size aerial imagery. 

Thanks, Steve.  

[webadmin]

Hello! It's been 2 years since this plugin was beta tested. Is there any progress to this project?  Thanks!