Database items "disappear"

[Chris Clawson]

Before going public, I need to be practiced in backing up and re-installing my database, in it's entirety. The AIP packager seems to be my best method to preserve everything, including Communities and E-Person users. It is easy to export to a AIP zip archive and FTP easily downloads it.

I have a strange problem, when installing it to another machine. I have a local Ubuntu 18.04 server on my local network at home. I have a working, virgin DSpace install and database (xmlui default). I attempt to install my AIP with the following command:

./dspace packager -r -a -k -t AIP -e webm...@domain-name.com -i 123456789/0 -o skipIfParentMissing=true /home/backup/sitewide-aip.zip

This completes after ingesting for several hours. I check my site – Wow, all the communities are there. Good.... oh.. WAIT : Where are the items? Any search returns nothing and there is no “recently added” items listed anywhere.

Okay – I run command line database utilities, ./dspace database index-discovery, ./dspace database repair, and ./dspace filter-media.

Great! It now all appears and works fine … UNTIL

I go to bed, and try to access it again in the morning. All gone. It is back to the same condition and problem I first encountered. The system is behind my firewall, so nobody has been tampering. I have attempted to access it with two other machines within my network – same results. If I try and address a specific URI to an item, that works fine (thumbnail and all).

The system has reverted to where it was before, all the communities (and their introductions) and collections will be listed, but no items are listed, browse-able, or searchable.

Sorry this post is long, but I am stumped. Does anybody know where to look for the problem? Why does my database items “go away” some hours after being “fixed”?

[Tim Donohue]

Hi Chris,

That obviously is not normal behavior.

That said, from what you describe, it sounds to me like maybe an issue in Solr or perhaps a simple caching issue?  Though, I'd first recommend checking your logs for any errors there. See https://wiki.lyrasis.org/display/DSPACE/Troubleshoot+an+error as a guide.

From the steps you described, it sounds like this is what is happening:

1. First you import a set of AIPs.  After importing them, they should be automatically indexed into Solr at this time. But, it sounds like that might be the first problem as that doesn't sound to be happening.
2. Then you run the reindex, which fixes the issue by indexing them in Solr (as the "recently added" list is loaded from that Solr index).
3. After sometime the issue returns. This is the part doesn't make sense to me, as it's unexplainable behavior (but a few possible ideas below)

If you are using the XMLUI, you may want to first check that this isn't a simple caching issue in the XMLUI itself.  After step #1, clear the XMLUI cache (login as an Admin, go to "Control Panel" -> "Java Information" and click "Clear Cache Immediately" near the bottom).  See if they appear in the "recently added" list immediately after clearing the cache.  If not, also try a hard refresh in your browser (Ctrl + refresh button) to make sure your browser isn't caching things & showing you outdated info.

If none of that has an impact, I'd recommend looking closer at your logs.  Either something is messing with your Solr index (less likely) or maybe Solr itself is throwing errors? It's hard to say without more details.

Good luck and let us know on this list what you find!

Tim

---

From: dspace-c...@googlegroups.com <dspace-c...@googlegroups.com> on behalf of Chris Clawson <industr...@gmail.com>
Sent: Wednesday, June 17, 2020 10:10 AM
To: DSpace Community <dspace-c...@googlegroups.com>
Subject: [dspace-community] Database items "disappear"

 

--
All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-communi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-community/69b20fa9-b802-47c8-b596-00c39c67814fo%40googlegroups.com.

[Chris Clawson]

Thanks Tim,

Here is a bit more info...

I have also checked the installation with JSPUI enabled. Same issue. The Communities, Collections and E-Persons are all there, but none of the items added to collections. I have yet to re-force all the database indexing, etc, as yet. Perhaps before doing that, I will clear all my logs first and just watch for this issue. I have tried using two different client machines as well as Firefox and Chrome. I do a lot of web testing, so I do not cache webpages. I thought of Solr. (Dumb Question) - Does JSPUI use Solr as well? If so, do you have specific logs and things to watch, if Solr is suspect?

The Java clear cache had no effect in improving the symptom (with the system in this present condition)

I have also been sure to check that all of the Dspace working directory and Recursive directories are all owned by tomcat:tomcat. My install is using PostgreSQL.

One possible error was that I installed DSpace as the root user. All my directories had to then be chown'd for Tomcat (8.5). I can't think of how that might affect this problem, but At least it is more information.

With no further immediate suggestions, I will clear out my logs, do a fresh reboot, mess again with the database utilities (if needed) and leave it hands off and see what logs again when the problem returns.

Thanks for the support!

To unsubscribe from this group and stop receiving emails from it, send an email to dspace-c...@googlegroups.com.

[Chris Clawson]

Possible correction: I just hand entered a known URI for several known items. On one, the page displays with the correct title, but there is no other metadata or thumbnail displayed. Another displayed properly, with all it's metadata and Thumbnail. Both the simple and full metadata views worked properly and I was able to view the full resolution bitstream image. The third hand entered URI returned a 'Page Not Found' error.

This is a test system, so I am not beyond starting fresh over, if there was a reason for it.

[Chris Clawson]

Update:

My /dspace/solr/statistics/data directory doesn't exist. Solr tries to use directories in this area and throws errors. Perhaps this is what breaks things. I have no idea why these directories were not created.

After installing Dspace, with maven and ant, all I did before importing the AIP was to define an admin user.

[Chris Clawson]

Finished:

(or so it seems)

After examining the Solr logs, I discovered many complaints about the 'tlog' files not being in /dspace/solr/statistics/data/tlog. In fact there were no 'data' directory created at all.

I performed a full system reboot, and then conducted the dcspace utility /dspace/bin/dspace index-discovery (which took a while to perform). The Dspace installation was running normally again and now the previously missing directories and files had appeared.

Summary: I am not exactly sure where my problem came from, but it may be important to repeat... This was a fresh install, made as a practice effort to see if I could fully backup and restore a live machine on to another remote server. I installed the prerequisites on my new, test system. I performed a DSpace installation and was able to reach the default home page with my web browser (from another machine on my network). I used the Dspace command line utility (via SSH) to create an administrator user. I then used AIP package import utility to install a pre-existing database, as mentioned earlier in this thread. This all completed well, but here is where I began to see problems. I believe my error was in not re-loading daemons or rebooting entirely, after the basic DSpace installation. Although Tomcat was running and was able to serve the default DSpace page, I am presuming there were other initial tasks needed, which would happen with a reboot.

My test system has now been idle for about 10 hours and is still performing normally, so I am presuming my strange problem is fixed. Restart, or re-boot after making major system changes! It seems to be a wise general policy.