Moving ResourceSpace to a new server?

[Douglas Thompson]

So, we currently have ResourceSpace v8.6.12603 installed and running on a server that is needing to be "decommissioned" by our Information Services folks. We haven't been able to upgrade ResourceSpace beyond v8.6.12603 because the OS on the server is old enough and our IS folks have "rules" in place about upgrading individual components of the OS beyond their default packages...

As such, they have spun up a new server for it to be housed on that has the latest OS and whatnot installed and can be upgraded further if needed.

So, we need to move ResourceSpace from the old server to the new server. Normally, I'd think "it's a LAMP install, so copy the database and the files and you're basically done" would work, but ResourceSpace has the additional complication of the filestore. Ours is huge...something like 540 Gb (nearly 55,000 images)!

Before I start that process, I was wondering, though...is it better to copy things over as-is and then upgrade ResourceSpace, or to install a clean copy of the latest version of ResourceSpace on the new server and then copy over the filestore from the old server (or something along those lines)? My hunch is the former is the better option, but is there anything that can/should be done to make the process work better?

I realize that whenever files are being transferred over a network, inherent slowdowns can happen, and that's fine...we can tackle the filestore in "chunks" if needed. But if I copy over the old version of ResourceSpace and its database to the new server without all of the filestore being copied over, will it work enough that I can test it before the entire filestore is copied over?

I searched in the group before posting this and see some conversations about similar situations but they're mostly from a few years ago, so I didn't know if anything had changed about the process that I should be aware of.

Basically, I'm wanting to make sure I have all the info I need before I start this undertaking so I'll know what to avoid doing (and not avoid doing, as the case may be)

Thanks in advance for any help on this,

Doug Thompson

Manager of Web and Electronic Communications

Ohio Wesleyan University

[Mike Perry]

Hey Doug --

I've done both. Philosophically I prefer a clean new install then migrating the existing resources and database. However if you've made any significant changes to the existing code outside of config.php you will lose those with a clean install.

If you choose a clean install you will need to transfer the secure keys from your current config.php to the new one and comment out the installed lines for those. You will also need to copy over any other config.php customizations. Permission issues sometimes spring up with resources as well. Do the clean install, back up the clean install db, replace the db with your existing db, then swap the secure key settings and any other config.php customizations you may have made. Login in to the new install with your old credentials and check things out.

If you copy the existing installation to the new server before upgrading you'll probably need to update some config.php settings (base installation URL. paths, etc.)

If your existing 8.6 install was done with Subversion the process is much simpler. Just be sure to do a full backup of your existing installation (including database) before doing the Subversion upgrade. You can also try to covert the standard installation to a Subversion; I've never had good luck with that. Since RS has moved away from a downloaded install set of files and gone to strictly Subversion I believe starting with a clean Subversion installation is the best route.

My best experience is starting with a clean Subversion installation and transferring resources and settings. Others may have a different view, and your mileage may vary!

Good Luck. . .

[Douglas Thompson]

No customizations were made outside of the config file, so a clean install sounds like it would be the easier/better thing to deal with...

Question...does anything have to be done (e.g., any special/hidden ResourceSpace scripts that have to be run to rebuild its index, etc.) after the filestore is copied over, or will ResourceSpace just "see" the filestore once it's copied over?

As I understand it, the general process should be:

1. Do a Subversion install of the latest version of ResourceSpace (our old version was not a Subversion install, but starting Subversion now will make it easier/faster to upgrade in the future, correct?).
2. Copy over the pertinent bits of the old version's config.php file to the new one.
3. Backup the new version's database and replace it with your old database.
4. Login to the new version with your old credentials and make sure it's working.
5. Copy over the filestore (and this step is where I don't know if there's anything additional that has to be done or if it will just work)

Correct?

[Mike Perry]

After you do the clean install and it's checked out with the new install credentials THEN copy existing filestore (I would rename the newly installed filestore beforehand so you have it if you need to go back and check/fix the new installation) to new server. Update the new config.php with existing settings. Then backup new db and replace with existing db. You shouldn't need to make any other adjustments to filestore (absent any permission hiccups!)

I think that's it -- been a little sleep deprived the past few nights so use your common sense!!

[Douglas Thompson]

K sounds like a plan. I won't be doing any of this until next week due to various scheduling issues, so just trying to get all my "ducks in a row" in time for that...

Thanks!

[Dan Huby]

This might be useful - https://www.resourcespace.com/knowledge-base/systemadmin/restore

[Douglas Thompson]

Perfect...thanks!

[Douglas Thompson]

Question...I've got RS installed on the new server and am going through its config file now to see what options I need to adjust...and one thing I'm noticing is that a lot of the things that were hard-coded in the old RS version's config file are now things that can be adjusted within RS.

So, would it be better for me to start with a "clean" config file and only copy over the "connect to the database" stuff from the old RS install when the time comes?

Also, I adjusted the RS image sizes about halfway through our run with the older version, but never bothered to go back in and adjust anything...so only newly uploaded stuff got the new sizes. Is there some sort of script (or something) that I can run that will basically rebuild those various sizes from the original images. I know it'll take a while to run such a thing, but this would be one of those "start it at the end of the day and leave it running overnight" types of things, so...no worries about that...

[Dan Huby]

It doesn't matter - you can stick with your existing config. If those
things are changed in the UI they'll automatically take precedent over
your config - the config files control the default value prior to
changing in the UI at the system-wide or user level (User Preferences).

There's a script in pages/tools/ to update your previews.

I hope that helps.

On 08/06/2021 20:52, Douglas Thompson wrote:
> Question...I've got RS installed on the new server and am going through
> its config file now to see what options I need to adjust...and one thing

> I'm noticing is that a /lot/ of the things that were hard-coded in the

> old RS version's config file are now things that can be adjusted

> /within/ RS.

>
> So, would it be better for me to start with a "clean" config file and
> only copy over the "connect to the database" stuff from the old RS
> install when the time comes?
>
> Also, I adjusted the RS image sizes about halfway through our run with
> the older version, but never bothered to go back in and adjust
> anything...so only newly uploaded stuff got the new sizes. Is there some
> sort of script (or something) that I can run that will basically

> /rebuild/ those various sizes from the original images. I know it'll

> take a while to run such a thing, but this would be one of those "start
> it at the end of the day and leave it running overnight" types of

> things, so...no worries about /that/...

>
> On Thursday, June 3, 2021 at 12:50:04 AM UTC-4 Douglas Thompson wrote:
>
> Perfect...thanks!
>
> On Tuesday, June 1, 2021 at 11:04:21 AM UTC-4 Dan Huby wrote:
>
> This might be useful -
> https://www.resourcespace.com/knowledge-base/systemadmin/restore
> <https://www.resourcespace.com/knowledge-base/systemadmin/restore>
>
> On Friday, 28 May 2021 at 16:00:32 UTC+1 Douglas Thompson wrote:
>

> K sounds like a plan. I won't be /doing/ any of this until
> /next/ week due to various scheduling issues, so just trying
> to get all my "ducks in a row" in time for /that/...

>
> Thanks!
>
> On Friday, May 28, 2021 at 9:48:02 AM UTC-4 Mike Perry wrote:
>
> After you do the clean install and it's checked out with
> the new install credentials THEN copy existing filestore
> (I would rename the newly installed filestore beforehand
> so you have it if you need to go back and check/fix the
> new installation) to new server. Update the new
> config.php with existing settings. Then backup new db
> and replace with existing db. You shouldn't need to make
> any other adjustments to filestore (absent any
> permission hiccups!)
>
> I think that's it -- been a little sleep deprived the
> past few nights so use your common sense!!
>
> On Friday, May 28, 2021 at 9:36:20 AM UTC-4 Douglas
> Thompson wrote:
>

> /No/ customizations were made outside of the config

> file, so a clean install sounds like it would be the
> easier/better thing to deal with...
>
> Question...does anything have to be done (e.g., any

> special/hidden ResourceSpace /scripts/ that have to
> be run to rebuild its index, etc.) /after /the

> filestore is copied over, or will ResourceSpace just
> "see" the filestore once it's copied over?
>
> As I understand it, the general process should be:
>

> 1. Do a Subversion install of the latest version of
> ResourceSpace (our old version was /not/ a

> Subversion install, but starting Subversion

> /now/ will make it easier/faster to upgrade in
> the /future/, correct?).
> 2. Copy over the pertinent bits of the old

> version's config.php file to the new one.

> 3. Backup the new version's database and replace it
> with your old database.
> 4. Login to the new version with your old

> credentials and make sure it's working.

> 5. Copy over the filestore (and /this/ step is

> where I don't know if there's anything

> /additional/ that has to be done or if it will
> just /work/)

> As such, they have spun up a /new/ server

> for it to be housed on that has the latest

> OS and whatnot installed and /can/ be
> upgraded further if needed.
>
> So, we need to /move/ ResourceSpace from the

> old server to the new server. Normally, I'd
> think "it's a LAMP install, so copy the
> database and the files and you're basically
> done" would work, but ResourceSpace has the

> additional complication of the /filestore/.
> Ours is /huge/...something like 540 Gb
> (nearly 55,000 images)!
>
> Before I start /that/ process, I was

> wondering, though...is it better to copy

> things over as-is and /then/ upgrade

> ResourceSpace, or to install a clean copy of
> the latest version of ResourceSpace on the

> new server and /then/ copy over the

> filestore from the old server (or something

> along those lines)? My /hunch/ is the
> /former/ is the better option, but is there

> anything that can/should be done to make the
> process work better?
>

> I realize that /whenever/ files are being

> transferred over a network, inherent

> slowdowns can happen, and that's /fine/...we

> can tackle the filestore in "chunks" if
> needed. But if I copy over the old version
> of ResourceSpace and its database to the new

> server /without/ all of the filestore being

> copied over, will it work enough that I can

> test it /before/ the entire filestore is

> copied over?
>
> I searched in the group before posting this
> and see some conversations about similar
> situations but they're mostly from a few
> years ago, so I didn't know if anything had

> /changed/ about the process that I should be

> aware of.
>
> Basically, I'm wanting to make sure I have

> all the info I need /before/ I start this

> undertaking so I'll know what to avoid doing

> (and /not/ avoid doing, as the case may be)

>
> Thanks in advance for any help on this,
> Doug Thompson
> Manager of Web and Electronic Communications
> Ohio Wesleyan University
>

> --
> ResourceSpace: Open Source Digital Asset Management
> http://www.resourcespace.com <http://www.resourcespace.com>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ResourceSpace" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/resourcespace/bzYI1_a4iX0/unsubscribe
> <https://groups.google.com/d/topic/resourcespace/bzYI1_a4iX0/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> resourcespac...@googlegroups.com
> <mailto:resourcespac...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/resourcespace/7a4e3076-e54e-447a-ae92-e50a41118932n%40googlegroups.com
> <https://groups.google.com/d/msgid/resourcespace/7a4e3076-e54e-447a-ae92-e50a41118932n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Douglas Thompson]

Perfect...thanks for the info!

[Douglas Thompson]

So question...which script in pages/tools are you talking about? update_previews.php? update_resource_dimensions.php? Something else?

And, when I started to go through the config file, I see that there's now a way to store "original" photos separately from their "resized" versions. I kinda like that option and want to enable it, but I see that it requires that the pages/tools/filestore_separation.php script be run...so do I need to run both the filestore_separation.php script and whatever preview-updating script is needed (see above), or just one of them, or what?

The end goal in all this is that my filestore will be separated into "original" and "resized" directories and the pixel dimensions of the images in the "resized" directory structure will match up with the new pixel dimensions I've specified as Image Sizes within RS...

[Dan Huby]

Probably both, but at this point I'll leave it for others to answer more
definitively :)

On 09/06/2021 14:15, Douglas Thompson wrote:
> So question.../which/ script in pages/tools are you talking about?

> update_previews.php? update_resource_dimensions.php? Something else?
>
> And, when I started to go through the config file, I see that there's

> now a way to store "original" photos /separately/ from their "resized"
> versions. I kinda /like/ that option and want to enable it, but I see
> that /it/ requires that the pages/tools/filestore_separation.php script
> be run...so do I need to run /both/ the filestore_separation.php script
> /and/ whatever preview-updating script is needed (see above), or
> /just/ one of them, or what?

>
> The end goal in all this is that my filestore will be separated into

> "original" and "resized" directories /and/ the pixel dimensions of the

> images in the "resized" directory structure will match up with the

> /new/ pixel dimensions I've specified as Image Sizes within RS...

> <https://groups.google.com/d/msgid/resourcespace/7a4e3076-e54e-447a-ae92-e50a41118932n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/resourcespace/7a4e3076-e54e-447a-ae92-e50a41118932n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> ResourceSpace: Open Source Digital Asset Management
> http://www.resourcespace.com <http://www.resourcespace.com>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ResourceSpace" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/resourcespace/bzYI1_a4iX0/unsubscribe
> <https://groups.google.com/d/topic/resourcespace/bzYI1_a4iX0/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> resourcespac...@googlegroups.com
> <mailto:resourcespac...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/resourcespace/6bdcce7a-146a-405b-9fe7-cc0293978734n%40googlegroups.com
> <https://groups.google.com/d/msgid/resourcespace/6bdcce7a-146a-405b-9fe7-cc0293978734n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Douglas Thompson]

So I now have the latest RS fully installed and configured and working, and I'm thus ready to copy over my filestore from our old RS instance..

I've established new image sizes and want to "regenerate" the already-generated image-size files that I'll be copying over to be the new sizes. At the same time, I've turned on the option for separating the filestore into "original" and "resized" directories.

I asked earlier in the thread, but which script(s) should I be planning to run (from pages/tools) after I get these files copied over in order to achieve this?

[Douglas Thompson]

Okay...I've decided to not mess with the "separate 'original' and 'resized'" thing with the filestore...so I only have the image sizes thing to worry about...

[Peter Payzant]

Hi, Doug -

I had this same need recently. I was unable to get any ideas for doing it quickly and easily via php code or the API, so in the interests of just getting it done I went full manual and uploaded all the resized images via the web interface. You probably already know this, but just in case you don't, here's the method:

1. Click "Edit metadata".
2. Click "Replace file".
3. Drag the new file to the upload area and click "Start upload".

Of course, I had a list of the images that I wanted to update and kept track of my progress that way. It was pretty slow, but it had the advantage of being fairly risk-free. I just picked away at it until it was done.

Regards

Peter Payzant

[Douglas Thompson]

Unfortunately, since I have just over 55,000 images to move, doing them one-at-a-time manually isn't really an option for me... :P

[Peter Payzant]

That is a big job. Given that no-one has proposed a ready-made solution,
it may be that the only options are some sort of custom software
development, or hiring a number of temps to grind through the photos
manually. Both are going to cost you and expose your database to risk.

If you don't need to supply the higher-resolution versions very
frequently, it might be more practical to just retrieve them manually
when needed, and wait until someone else comes up with an automated
solution.

Regards

Peter Payzant

[Douglas Thompson]

Yeah. Luckily, though, I think there's a script (or scripts) in ResourceSpace's pages/tools folder that will redo what's needed after I copy over the "filestore" of photos from the old version of ResourceSpace...it's just a matter of me figuring out which script(s) are the ones to use...

[Douglas Thompson]

So, any word on whether it's possible for me to move over my 50,000+-resource filestore from our old RS installation to our new RS installation and use pages/tools script(s) to make the changes in the filestore structure and pre-generated image size variants mentioned above in the thread?

I was hoping to get the filestore copying process started today, but it looks like that's not going to happen...

Any helpful advice would be much appreciated, and if the answer is that I can't make the changes I'm trying to make, then so be it...I just need to know either way what my options are.

[Charlie Alonso]

Hi there

Could be rsync or rclone tools a good tool to sync the older filestore to the new server?

Charlie Alonso

https://zonadam.com

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com

---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/7c21fa23-ebb2-46e5-9cd5-91356d364015n%40googlegroups.com.

[Douglas Thompson]

Our IS guy is going to be using rsync, so the actual copying of the filestore files is squared away...I just need to know which script(s) in pages/tools are the best to run to achieve:

1. Adjusted image sizes (I'd like to take this opportunity to change what image sizes RS uses so that people can download what they need for use on the website (currently, what they download is 600 pixels of width larger (at most) than what is needed on the website.
2. Doing the $filestore_evenspread=true; thing so that RS uses the last character of resource IDs to set up the filestore structure rather than the first character of resource IDs. A more equal filestore structure, basically.

What I would like to do is be able to copy over the RS filestore (which has 50,000+ images!) from our old RS installation as-is and then run one or more scripts from pages/tools to rebuild the filestore in such a way that it achieves both #1 and #2 above.

Looking through the comments (or lack thereof, in a couple instances where I'm just going on filename), I've identified the following candidates for possible pages/tools scripts:

• database_prune.php
• dbstruct_create.php
• filestore_migrate.php
• filestore_separation.php
• healthcheck.php
• migrate_data_to_fixed.php
• prune_filestore.php
• reindex.php
• reindex_by_collection.php
• relate_resources.php
• staticupdateexif.php
• test_coverage.php
• update_paths.php
• update_previews.php
• update_resource_dimensions.php
• update_resource_file_sizes.php
• update_sizes.php
  

I think the two that are most pertinent to what I'm needing to do are "filestore_migrate.php" and "reindex.php", but I'm not 100% sure...

And if it's not possible to do the two changes listed above, then I can live with keeping everything as-is with the filestore. I just figured that this would be the perfect time to make changes to the filestore, if possible, since we're basically starting with a new install at this point...

Any help/advice on this would be much appreciated.

[rwr...@sbbg.org]

At the risk of stating the obvious, trying it out on a dev server seems like a reasonable strategy.

Randy Wright

[Douglas Thompson]

If we had fewer than 50,000+ resources that took up less storage space than 650 GB, I would agree. As it is, however, we don't have that amount of extra storage space floating around where we could feasibly try this stuff ahead of time...not to mention the whole "it would require us to copy everything twice" aspect of it...

[Charlie Alonso]

Not sure is you would dump the database and import im the new installation to keep the metadata of all your pictures. If this is your plan, I would configure the preview sizes an regenerate preview after migrate filestore and reindex

Charlie Alonso

https://zonadam.com

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/1e3fc048-0deb-456a-8b73-8aa14ec307b6n%40googlegroups.com.

[Douglas Thompson]

That's my plan...copy over the entire database and filestore as-is from the existing RS installation, then run whatever pages/tools scripts are necessary to achieve the new sizes and filestore structure...

So maybe there's only two pages/tools scripts that are needed: "update_previews.php" and "reindex.php"?