Internal System Error
JIS PUNNASSERY
Dear DSpace Technical Support Team,
I am writing to request your urgent assistance in resolving a critical issue with our DSpace repository.
We are currently using DSpace 6.3 (JSPUI) on Ubuntu 20.04 LTS. Our repository was working previously, but it is now displaying an Internal System Error, preventing users from accessing the repository.
System EnvironmentDSpace Version: 6.3 (JSPUI)
Operating System: Ubuntu 20.04 LTS
Database: PostgreSQL 12
Application Server: Apache Tomcat 9
Search Engine: Apache Solr
When opening the DSpace website, the following message appears:
Troubleshooting PerformedInternal System Error
The system has experienced an internal error. Please try to do what you were doing again, and if the problem persists, please contact us so we can fix the problem.
To resolve the issue, I have already performed the following checks:
Verified that PostgreSQL is running normally.
Verified that Apache Tomcat 9 is running.
Confirmed that the DSpace database is accessible and contains approximately 1,352 items.
Checked the DSpace configuration files.
Reviewed Tomcat and DSpace log files.
Corrected ownership and permissions of the /dspace directory.
Corrected permissions on the /dspace/log and /dspace/solr directories.
Added the Tomcat user to the DSpace group.
Restarted Tomcat several times.
Checked the Solr application.
Attempted to rebuild the Solr search index.
Verified that the filesystem is writable.
During troubleshooting, I observed the following issues:
Solr search core initialization failure.
Search core reported as unavailable.
Permission-related errors involving the /dspace/log and /dspace/solr directories.
Previous log entries showed "Permission denied" and "Read-only file system" errors.
Although these permission issues have been corrected, the repository still displays the same Internal System Error.
PostgreSQL service: Running
Apache Tomcat 9: Running
Solr application: Accessible
DSpace Web Interface: Displays Internal System Error
I would greatly appreciate your help in identifying the root cause of this problem and restoring the repository to normal operation.
Could you please advise:
What could be causing the Internal System Error?
How can I resolve the Solr search core initialization problem?
Are there any additional configuration files or services that should be checked?
What diagnostic steps do you recommend to identify the exact cause?
Is there any procedure to safely rebuild the DSpace search indexes without affecting the existing repository?
If required, I can provide the following information:
DSpace log (dspace.log)
Tomcat logs (catalina.out)
Solr logs
PostgreSQL logs
local.cfg
Configuration files
Directory permissions
Any additional information needed for diagnosis
This repository is actively used by our institution, so your assistance in resolving this issue would be greatly appreciated.
Thank you very much for your time and support. I look forward to your guidance.
Titus
From your description, the key observation is this:
- PostgreSQL is running.
- Tomcat is running.
- The repository previously worked.
- You have 1,352 items in the database.
- The error started after Solr permission/read-only filesystem problems.
- The JSPUI now only shows "Internal System Error".
This strongly suggests that the JSPUI itself is healthy, but one backend component is failing during a request. In DSpace 6.3 the most common causes are:
- Solr search core unavailable (most likely)
- Database connection issue
- Corrupted Assetstore
- Configuration mismatch
- Java memory/classpath issues
Since PostgreSQL and Tomcat are working, I would investigate Solr first.
Step 1: Check the REAL exception
The JSP page hides the actual error.
Run
tail -200 /dspace/log/dspace.log
and
tail -200 /var/log/tomcat9/catalina.out
Look for
Caused by:
or
Exception
or
ERROR
The first exception is the important one.
Without that exception we're only guessing.
Step 2: Verify Solr cores
Open
Do you see something like
search
statistics
oai
authority
?
If the Solr Admin page loads, execute
curl http://localhost:8080/solr/admin/cores?action=STATUS
Expected output
search
status:
If instead you get
Core search is unavailable
or
Error loading core
then Solr is the problem.
Step 3: Verify search core exists
Check
ls -l /dspace/solr
Typical DSpace 6.3 layout
authority/
oai/
search/
statistics/
Inside search
conf/
core.properties
data/
Verify
ls /dspace/solr/search
Expected
conf
core.properties
data
If any are missing, Solr cannot initialize.
Step 4: Check ownership
Run
ls -ld /dspace
ls -ld /dspace/solr
ls -ld /dspace/log
ls -l /dspace/solr/search
Usually they should be
tomcat:dspace
or
dspace:dspace
depending on your installation.
Permissions
775
or
755
Files should not be owned by root.
Step 5: Check if filesystem is still read-only
Earlier you mentioned
Read-only filesystem
Sometimes Ubuntu remounts the filesystem read-only after disk errors.
Check
mount
Look for
(ro,
If you see
ro
instead of
rw
then the filesystem is still read-only.
Also check
touch /dspace/testfile
If
Permission denied
or
Read-only file system
appears,
that's the actual problem.
Step 6: Check available disk
Run
df -h
Especially
/
and
/dspace
If disk usage is
100%
Solr may fail to start.
Step 7: Check inode exhaustion
df -i
Sometimes there is disk space available but no inodes.
Step 8: Verify Java can write to Solr
Delete only the Solr lock files.
find /dspace/solr -name "*.lock"
If any exist
rm *.lock
inside the data directories.
Never delete
schema.xml
or
solrconfig.xml
Step 9: Verify search index
You can safely rebuild the search index.
This does not delete items.
Only Solr indexes are recreated.
First stop Tomcat
sudo systemctl stop tomcat9
Delete only the index
/dspace/solr/search/data/index
NOT
conf
NOT
core.properties
Restart Tomcat
Then run
[dspace]/bin/dspace index-discovery -b
This rebuilds the Discovery index.
For 1352 items it should finish quickly.
Step 10: Verify database connectivity
Run
[dspace]/bin/dspace database test
or simply
psql -U dspace dspace
If the CLI connects,
database is fine.
Step 11: Check local.cfg
Verify
solr.server = http://localhost:8080/solr
and
db.url
matches PostgreSQL.
Also verify
assetstore.dir=
points to an existing directory.
Step 12: Test Discovery manually
Open
http://SERVER:8080/solr/search/admin/ping
Expected
status: OK
If instead
500
or
Core unavailable
then Discovery cannot start.
Step 13: Check Tomcat deployment
Confirm
loads.
Also verify
ls /var/lib/tomcat9/webapps
contains
xmlui
jspui
solr
depending on your installation.
Step 14: Check Java memory
Sometimes Solr cannot initialize because of heap exhaustion.
Check
journalctl -u tomcat9
Look for
OutOfMemoryError
or
GC overhead limit exceeded
Most likely causes (based on your symptoms)
· Based on the sequence you described, I'd rank the likely root causes as:
· Solr search core failed to initialize after the filesystem/permission issue
· Corrupted Solr index after the read-only filesystem event
· Incorrect ownership or permissions on /dspace/solr or /dspace/log
· Disk or filesystem corruption causing incomplete writes
· Database problem (less likely if you can query all items)
--
All messages to this mailing list should adhere to the Code of Conduct: https://lyrasis.org/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dspace-tech/94f90a98-6398-43e8-bee3-55105452e576n%40googlegroups.com.