Filestore Path and Scramble Key

[dave]

Hello,

I'm in the process of putting together a script to rebuild my ResourceSpace database after a motherboard failure. I have the original ResourceSpace directories including the config.php and the original scramble key (as well as the 'filestore' directory containing the assets). 

I would like to know exactly how a 'filestore' file path (and filename) is generated using the scramble key.

I found the line in include/general.php:

$scramblepath=substr(md5($ref . "_" . $scramble_key),0,15);

I assume $ref is the path to the asset. Could someone please give an example? is this a path pointing to a location within the 'filestore' directory? does it include the original filename of the asset or is the filename not used?

For an asset with a reference ID of 3, I have the following path pointing to its location in the 'filestore':

/var/www/resourcespace/filestore/3_1c7731363e2c53d/3_37f9cb87a14b3bf.mts   (this is a .MTS format video file with an original filename of '00882.MTS').

I would like to know specifically how the hashes in the path shown above were generated (i.e. what path information was used to generate the hashes).

Thank you!

Dave

[dave]

Are there any developers following the list that might be able to help me out with this question?

(I'd be happy to give the finished recovery script to anyone interested, once it's completed.)

[David Dwiggins]

If you have the original files and scramble key, I'm not sure what you need to write any new code for. Just put the scramble key in the config file, and the system will find the files.

If you're saying that you have the files but not the database, I already wrote a script to do this for a user in the same situation. This was done last spring, so I think it should be in the current release.  It's not perfect, but it does about as good a job as possible given the limitations of the data available on disk. Take a look at pages/tools/recover.php

Note that I think I did fix one or two things about the on-disk XML files after this (for example recording the resource types and file extensions in the on-disk XML file, I think) but I don't think I updated the recovery script to take advantage of this. (It probably won't matter that much if you have an old database, since these changes would have only started with new resources.)

Take a look and let me know if you have questions.

-David

On Mon, Nov 5, 2012 at 9:48 AM, dave <dave.e...@gmail.com> wrote:

Are there any developers following the list that might be able to help me out with this question?

(I'd be happy to give the finished recovery script to anyone interested, once it's completed.)

--
 
 

[dave]

Thanks a lot! I'll check it out.

[dave]

Just following up...

The recover.php script worked great and appears to have recovered all of my resources. Reconstructing my collections was not a problem as they were quite simple.

I did comment out the "create_preview" lines (in two places) which sped up the actual importing of the data by eliminating preview creation. Now that the database seems to have been reassembled, I'm running the "update_previews.php" to recreate the previews.

Also, at one point several hours into the recovery, the script failed because of an "empty filename" error. I checked out the resource storage location (by ID) in the filestore and just deleted it -- there appeared to be nothing stored at that ID. I had to re-run the script (at which point I commented out the preview creation lines to save time) which started the recovery from the beginning again. I do not believe that the recovery.php script allows recovery starting from where it previously left off (in the event of an error) -- someone please correct me if I'm mistaken. 

Thanks, David, for pointing out the recover.php script! It saved me a lot of time that would have been spent "re-inventing the wheel" to regenerate my ResourceSpace database.

On Monday, November 5, 2012 9:55:24 AM UTC-5, David Dwiggins wrote:

[David Dwiggins]

Glad it worked for you!

You can tell it to restart after an error. If you edit the 

// first resource to recover

$start_id = 1;

in the config section, you can tell it to start recovering from a later resource. So just enter the number of the last one that worked and start it again, and it will keep plugging along.

I imagine that we could trap for the error you experienced so that the script wouldn't die, but given how infrequently this script is used, I'm not inclined to spend much time into it.

You're right that regenerating previews afterward rather than during would substantially speed up the initial import.

As always, the best way to ensure you never need to do this is to ensure that you have complete backups of the filestore and config files, and complete MySQL dumps of the MySQL database. Restoring with this script is a last ditch approach!

-David

--