Dspace Data Migration with SAF

[Rupinder Singh]

I have a problem with SAF, as I am Migrating from Dspace 6.2 to DSpace 7.4. I exported my collections from DSpace 6.2, But as I upload the SAF items to my Dspace 7.4 everything goes ok, Except for the items which are part of multiple collections. The import process exists with an error saying that the Item/Handle Already Exists.
How can I resolve this conflict? There are many items that appear is more than one collection in my existing Dspace 6.2.

[Rupinder Singh]

Can someone please help me with this? 

I am planning for migration and am stuck on this. AIP backup and restore are of no use because it throws too many errors. I found SAF to be simple and reliable, but the only problem is with these conflicts of items appearing in multiple collections. 

Please help 

Thank you

Rupinder Singh

[Mohammad S. AlMutairi]

Hello  Rupinder,

You really don't have to use SAF or AIP for this migration since you own both systems ( old & new ). Both should work but It's time consuming and sometimes flaky. There is a faster and a shorter path. Try it and let us know in the list if you still need more help. I'll walk you through.

1) Dump the old database in a binary format ( pg_dump -Fc dspace > dspace-6x.sql ) and copy it along with the old assetstore to the new server.

2) su - posgres

3) Restore the database into the new server using ( pg_restore -v -c -d dspace dspace-6x.sql ). You might need to run ( [dspace]/bin/dspace database clean ) first and for this to work you need to add ( db.cleanDisabled=false ) at the top of ( [dspace]/config/dspace.cfg ) and no need to restart tomcat.

4) exit

5) Run [dspace]/bin/dspace database migrate ignored

6) Check [dspace]/bin/dspace status & info

7) [dspace]/bin/dspace create-administrator

8) Copy the old assetstore folder to where it should be in the new server [dspace]/assetstore

9) Chown -R tomcat:tomcat [dspace]

10) systemctl restart tomcat & finally check if everything is O.K.

** You can if you need to or have to dump the database in a text format and run search and replace on it for anything ( old domain and URL's that has xmlui in it comes in mind ). Do it after finishing step 3 above.

a) su - postgres

b) Dump the databse in text format ( pg_dump -c dspace > dspace-migrated.sql )

c) Search and replace using ( sed, awk, perl, vi or any text edit you feel comfortable with ).

d) Restore it ( cat dspace.sql | psql -d dspace < dspace-migrated.sql )

 

Good luck

[Mohammad S. AlMutairi]

Correction. I wasn't focusing ;-).

d) Restore it ( cat dspace.sql | psql -d dspace ).

[Rupinder Singh]

Thanks, Mr. ALMutairi, but no luck with that. 

Surprisingly the old Dspace 6.2 server does not have any assetstore folder, but the repository is working somehow. I checked the assetstore location from the config file too, but, it's all on default, Searched for other installation locations, but found nothing. Actually, this server was installed long back by someone, who left the organization later. So this old DSpace server was left unattended for quite a long time. 

I created a test server with Dspace 6.2 with the same config, and tried restoring with AIP and SAF on it, Still got the same problems. AIP commands throws error

java.lang.UnsupportedOperationException: Could not find a parent DSpaceObject referenced as '32116/180' in the METS Manifest for object hdl:32116/181. A parent DSpaceObject must be specified from either the 'packager' command or noted in the METS Manifest itself. 

and SAF import works only till it encounters an item that is part of another collection, then it gives an error saying 

Notice
Import failed
Attempted to create a handle which is already in use: 32116/2483

[Mohammad S. AlMutairi]

Please forget AIP and SAF.

Can you get me privately a copy of local.cfg, dspace.cfg and  [dspace]/config/spring/api/bitstore.xml .. You can remove anything private data you think is worth removing ( database password, domain ..etc ).

You need to read this: https://wiki.lyrasis.org/display/DSDOC6x/Storage+Layer#StorageLayer-ConfiguringtheBitstreamStore

[Mohammad S. AlMutairi]

I wanted to warn you that bitstore.xml might have your Amazon awsAccessKey, awsSecretKey, and bucketName so remove them if you are going to send the files or you can take it from here and move your missing assetstore to the server filesystem and follow the usual upgrade procedure.

[Rupinder Singh]

Dear Mr. AlMutairi, 

I sent you copies of dspace.cfg, local.cfg, and bitstore.xml. on your personal mail.

However, after your suggestion, I checked the local.cfg file and found that all the configuration settings were stored there and the assetstore folder is set to  /mnt/dspace/assetstore . So I have taken a backup of it, and will now try restoring it on my DSPACE 7 test server as you suggested in your first mail.  

Please have a look at the config files though, we may come to find something more that went unnoticed before. 

Thank You 

Rupinder Singh

[Rupinder Singh]

Dear Mr. AlMutairi, 

That worked, and I was able to restore the database and the assetstore on my new DSpace 7  installation. I can now see all the collections, communities, and items on my Angular GUI. But as soon as i open any item, It shows the metadata of the document uploaded but gives the following error when is click the bitstream (a PDF dissertation in my case).

Whitelabel Error Page

This application has no explicit mapping for /error, so you are seeing this as a fallback.

Wed Jan 11 17:56:55 IST 2023

There was an unexpected error (type=Internal Server Error, status=500).

An internal read or write operation failed

[Mohammad S. AlMutairi]

Hello  Rupinder,

You are almost there. You should have some clues in the log [dspace]/log/dspace.log ... This error  ( An internal read or write operation failed ) suggest it could be a permission problem or the the assetstore is not where it expected to be (guessing) so check them both also make sure dc.identifier.uri links are correct for all the moved items and finally you need to do a discovery indexing [dspace]/bin/dspace discovery-index -b 

Good luck

[Mohammad S. AlMutairi]

Maybe I should retire. :-)

[dspace]/bin/dspace index-discovery -b

[Rupinder Singh]

Dear Mr. AlMutairi, 

You were right, it was an issue with assetstore folder permissions and ownership. just one chown command and it was fixed. everything is now working as expected. 

But I still don't understand why AIP didn't work, and why it is so complicated. Anyway, I will continue to experiment with that.   

Thank you so much for all your help. That is all I needed, for now, to migrate my repository from the old server to a new one. 

We shall consider this case closed. 

Thank you once again.

with regards

Rupinder Singh

Assistant Librarian

Central University of Punjab

[Mohammad S. AlMutairi]

Hello Rupinder,

I'm so glad you finally had it sorted out. You are most welcome.

Be safe & Cheers