Is _this_ possible with RS?

[Douglas Thompson]

So, we're a university with thousands of photos. We have a limited amount of storage space that our IS folks have given us, so we have to do a yearly "purge" of "older" resources from the system in order to free up some of that space.

Older stuff (purged and otherwise) exist on a couple USB drives that we keep protected and only plug in to a computer when needed.

What we'd like to be able to do is:

1. Keep the thumbnails/previews and metadata (keyword, mostly) search capability in RS. For resources that are no longer in RS, though, we'd like some sort of "offline" indicator (or something along those lines), so that the people doing the searching will know when they've found an older photo that they need to grab off a USB drive.
2. Not just do this as a one-time thing...we'll no doubt have to purge things again next academic year (each academic year is a good sectioning point for this stuff, really).

For #2, even if it's something that can only be accomplished by manually running a script (or something), that's still a lot better than not being able to deal with it at all.

I've looked into the Archive functionality and the Offline Archive plugin, but haven't yet been able to get it all to work the way I want.

The wrinkle is that I'm not really much of a programmer, so I'm pretty much going to have to stick with out-of-the-box solutions here.

Any ideas how I can accomplish the basic goal of having a robust RS but balancing that with the fact that we have a finite amount of storage space to work with?

[Douglas Thompson]

So I just had a thought that might help our situation a tad...right now, all "originals" we upload to RS are resized down to (where necessary) 4000 pixels wide. So, for the most part, the "original" size is 4000 pixels wide by whatever the height is.

What if we were to change that to 3000 pixels wide instead of 4000 pixels wide? That's a reduction of 25% in size, so probably comes out to roughly that amount in disk space usage, right?

My question is...is there a way for me to resize the originals after-the-fact without having to re-upload all of them? I'm betting there _is_, but I don't know for sure...

(we keep the full-size originals on the USB drive mentioned above, so no worries if something larger than 3000 pixels wide is needed)

[Joseph Brooke]

Hi there,

To address your first message there is the config option $originals_separate_storage - if you enable that in your ResourceSpace config.php file and then run the script pages/tools/filestore_separation.php it will split your filestore into two folders - "original" and "resized". Original uploaded resource files will be stored in the "original" folder while the preview files generated for that resource will be stored in the "resized" folder. You could then try writing a script that regularly purges resources in the "original" folder when they reach a certain age or have an ID less than a certain value. Unfortunately there are no built in scripts to do this in ResourceSpace, ChatGPT may be able to help with writing the script. 

For your second question - if you change the size of the previews created via the Manage Sizes area - https://www.resourcespace.com/knowledge-base/systemadmin/manage_sizes - you need to recreate previews for your resources in order to generate the smaller sizes. You can do this by adding all your resources into a collection and in the Actions menu select "Edit collection previews". 

[Douglas Thompson]

I'll give the first one a try when I get a chance...does just deleting the "originals" automatically make them go offline or something?

For the second one, does regenerating the previews also alter the sizes of the originals? I'm betting it doesn't, because I only see the resized-down sizes in that System -> Image Sizes screen. If I do the first thing and then download all of the originals and do a batch-resizing (with Adobe Bridge or whatever) on those, can I just FTP upload them back to the separate "originals" filestore folder structure, or will that screw things up?

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to a topic in the Google Groups "ResourceSpace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/resourcespace/hhvmvn8nbXI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to resourcespac...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/380f2678-29d6-47d1-950d-9cca875311a3n%40googlegroups.com.

--

Douglas E. Thompson Manager of Web and Electronic Communications Office of University Communications Ohio Wesleyan University 61 S. Sandusky St. Delaware, OH 43015 Email: deth...@owu.edu Office Phone: 740-368-3356 Cell Phone: 614-582-6564 Website: owu.edu

[Joseph Brooke]

If you delete the original file for a resource the record will still exist and will be searchable and viewable in the ResourceSpace interface, you will see "Offline resource" in the download area for the original file but you can still download the preview sizes. 

Regenerating previews doesn't alter the size of the original file - it uses the original file to create the previews but that remains unchanged. So if you need to do any changing of preview image sizes you need to do that first before you delete any original files, otherwise when you recreate previews you will get no previews at all. 

FWIW I've updated the Knowledge Base article for the offline archive plugin with more steps on how to use it - https://www.resourcespace.com/knowledge-base/plugins/offline_archive It still might be useful for your situation. 

[Douglas Thompson]

Clarification needed...I think I have a general process worked out in my head:

1. Decide what needs to be purged (we probably don't want/need to do this to everything after initial upload (so that the "resized" stuff is created), so we'll still likely periodically need to identify stuff that we definitely don't need to keep the originals in RS.
2. Add everything (or just a few collections at a time...it really doesn't matter) to a single temporary collection that we make just for this.
3. Use the offline archive plugin in some way, so I'll know which ones to delete from the "original" folder structure (or to automatically delete them or something)
4. Done. The previews and keywords and other metadata and whatnot for everything will still be in RS, but the original size (which, really, is the one that takes up the most space) will say "offline resource" instead of having a Download button. When we see that, we'll know to grab the original from the disk.

So it's entirely down to figuring out how #3 can/should work in our situation...what did you mean by "it still might be useful for your situation"?

[Douglas Thompson]

Might it be possible to specify some sort of non-existent/null/etc. directory for $offline_archive_archivepath for the Offline Archive plugin so since the plugin basically moves the "originals" to that folder, it would effectively be deleting them?

Alternatively, maybe the solution is to create some sort of directory for $offline_archive_archivepath and (via CRON job or something) periodically delete the contents of that and then use the "Offline archiving steps" (as explained in the Knowledge Base article for the plugin that you mentioned upthread)?

Thoughts?

[Douglas Thompson]

Okay so I just tried the steps in the "Offline Archiving Steps" section on the Offline Archive plugin's page:

https://www.resourcespace.com/knowledge-base/plugins/offline_archive

And everything is working as expected up until step #7.

This entry on the Installation Check screen:

Tells me that there was no cron job set up...no scheduled tasks have run for 3853 days, which would put it back to where we _first_ started _using_ RS.

Sure enough, when I looked at the cron.d directory, there was _no_ "resourcespace" entry.

So, I made one, following the Linux steps on _this_ RS knowledgebase page: https://www.resourcespace.com/knowledge-base/systemadmin/cron

But then when I manually ran cron_copy_hitcount.php (via the "nice ionice php" thing), it spit out this:

[dethomps@tabit2 tools]$ nice ionice php cron_copy_hitcount.php
2025-04-14 11:22:56 https://tabit2.owu.edu/rsspace Starting cron process...
2025-04-14 11:22:56 https://tabit2.owu.edu/rsspace Executing job: 002_copy_hitcount.php

Sorry, an error has occurred. /var/www/html/rsspace/include/general_functions.php line 1875: file_put_contents(/var/www/html/rsspace/filestore/tmp/process_locks/copy_hitcount): Failed to open stream: Permission denied

#0  errorhandler()
#1  file_put_contents() called at [/var/www/html/rsspace/include/general_functions.php:1875]
#2  set_process_lock() called at [/var/www/html/rsspace/batch/cron_jobs/002_copy_hitcount.php:14]
#3  include(/var/www/html/rsspace/batch/cron_jobs/002_copy_hitcount.php) called at [/var/www/html/rsspace/batch/cron.php:35]
#4  include(/var/www/html/rsspace/batch/cron.php) called at [/var/www/html/rsspace/pages/tools/cron_copy_hitcount.php:5]

So it starts to work, but then fails with the "Permission denied" error. Is fixing that super-simple, or has this moved beyond my limited capabilities for this stuff?

If I can get the cron to work, then hopefully I have a pretty much workable solution to my original quandry:

I've created a "temp" folder in my home directory on the server and have added the $offline_archive_archivepath line to the RS config file that lists the full PWD for that. I then do step #2 (from the Offline Archive plugin's page) to set a photo to "Waiting to be archived" and then do the other steps. Once the cron is working, it should move the original file for the affected image out of the filestore (which I've already split into "original" and "resized") and into the new "temp" folder. I can hopefully then just delete the contents of that folder periodically and achieve the "all the thumbnails and preview sizes and metadata and whatnot for the resource remain in the system and are searchable, but the most-diskspace-taking original is no longer there" thing mentioned upthread.

Right?

[Douglas Thompson]

Just an update that I manually ran that cron_copy_hitcount.php with sudo and it worked...or at least it didn't error out with a "permission denied" message. But, the archived file was copied to the "temp" directory that I made (it was still in RS as well, only named whatever I put for the archive prefix)

So, I'm still missing something with the configuration or how I think this might work...any help would be greatly appreciated.

[Douglas Thompson]

Just replying to this to hopefully resurrect this conversation, as I finally have more time to look into this stuff. I've done nothing since my last "just an update" message, so something isn't quite working properly still...

[David Ryan]

Hi Douglas,

I *think* I may be interested in a similar solution, but there's so much in the notes above that it's hard to tell quite how far you got with your own work.

Starting from "we have an RS server with a lot of stuff in the filestore and want to move some of the older stuff to a USB stick but still have the previews show up" can you maybe outline what you have working and what still isn't working for you?

Thanks,

David

[Douglas Thompson]

I've been trying to figure this out since early February. I've got bits and pieces of it working, but have never successfully got it all to "play nice" together (and have never got all of what's in my head about it to work even manually).

My challenge is that I'm not much of a programmer or anything, so I've basically had to rely on other people helping with various aspects of nearly all of it. Unfortunately, that means that I don't really have anything along the lines of "what you have working and what still isn't working for you"

You summed up nicely what I'd like to be able to do (and what I think should be doable if the corrects bits and pieces are all set up properly). We have a bunch of stuff in the filestore and want to move the older stuff to external storage but leave its thumbnails and searchable metadata in the system so that current RS users can still find it if needed. We'll just have to pull it from the external storage in those cases. I'm just trying to figure out how to get all of that to work. I'd get one thing to (mostly) work, but then ran into a roadblock on something related to that and would ask about it and get no answer and then try some other aspect of it a month or two later and run into roadblocks with that.

Unfortunately, it seems like every time I think I have time to look into it closer, something comes up to up-end that and so I move on and never get back to what I wanted to do in the first place :P

[Douglas Thompson]

Case in point, it's nearly five months later and I'm just now getting around to being able to look into this stuff.

So getting back to basics (and a post much earlier in this thread from Joesph Brooke), if I were to manually delete the originals (which is easy for me to do, since I have RS set to separate the "originals" from the "generated"...I'd just have to delete the files in the "originals" files, correct?) am I correct that the already-generated thumbnails/previews/downloads/metadata/etc. would remain in the system...it's just that the "Original JPG File" would show as "offline"...correct?

So, "all" that I need to figure out is how to automate the process so that only originals before a certain date (or that I manually select somehow) would be deleted...is that something that the Offline Archive plugin could accomplish?

[Douglas Thompson]

Crap and I just realized that it's not as easy as deleting some of the originals because of the nearly-impossible-to-determine directory structure of the RS filestore...I'd have no way of knowing which original is for which resource, basically, which would make purging just the ones that I want to near-impossible.

I feel like I'm so close to figuring this all out, but I keep running into roadblocks...

[Jim Kirkpatrick]

Actually you can figure out the resource ID from the directory structure.

Resource 1234 is stored in 

Filestore/1/2/3/4_somerandomchars

If you have $filestore_evenspread=true; in your config.php it would be stored like this:

Filestore/4/3/2/1_somerandomchars

Does that help you at all?

Jim

You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/resourcespace/081cbdbb-e774-419a-981a-ed42940b5823n%40googlegroups.com.

[Douglas Thompson]

Hrmmm yes I do believe that will help. Thanks!

[Douglas Thompson]

Now if only there were a way to (semi)automate this, either an "input a list of RS ID#s and the 'originals' will be deleted"-type thing, or some sort of mechanism for deleting just the originals for resources within the RS back-end...