Unable to start Nexus repository on CentOS 7 running on AWS

[S Roy]

Our Nexus repository ran out of space and now we cannot start the repository. Unfortunately, we do not have a backup of the repository that we can revert back to.

After we made some space on the disk 3.4. GB is free. Now we are trying to fix the corrupt Orient databases namely config, and component. After we start the Orient DB console via

*sudo java -jar ./lib/support/nexus-orient-console.jar*

and try to connect to the config DB via

*CONNECT plocal:/mnt/sonatype-work/nexus3/db/config admin admin*

we are getting error com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/mnt/sonatype-work/nexus3/db/config' with mode=rw DB name="config"

Connecting to database [plocal:/mnt/sonatype-work/nexus3/db/config] with user 'admin'... 2018-11-24 02:23:00:977 WARNI {db=config} Storage 'config' was not closed properly. Will try to recover from write ahead log... 2018-11-24 02:23:00:983 WARNI {db=config} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OCheckpointEndRecord{lsn=LSN{segment=215, position=52}} will be skipped during data restore$ANSI{green {db=config}} Exception during storage data restore java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.cutTill(ODiskWriteAheadLog.java:919) at

$ANSI{green {db=config}} Error during storage close java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.cutTill(ODiskWriteAheadLog.java:919) at

Exception 475646D4 in storage plocal:/mnt/sonatype-work/nexus3/db/config: 2.2.31 (build 285537d2767275f460df32c6a3be01bfff6a517c, branch 2.2.x) com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/mnt/sonatype-work/nexus3/db/config' with mode=rw DB name="config" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:323)

If we look at the processes running via htop we see multiple java -jar ./lib/support/nexus-orient-console.jar running.

Not sure if that is causing a problem connecting to the config db.

Any ideas or recommendations to fix the issue?

[Peter Lynch]

On Fri, Nov 23, 2018 at 10:35 PM S Roy <shank...@gmail.com> wrote:

Our Nexus repository ran out of space and now we cannot start the repository. Unfortunately, we do not have a backup of the repository that we can revert back to.

What exact version of Nexus? 

After we made some space on the disk 3.4. GB is free. Now we are trying to fix the corrupt Orient databases namely config, and component. After we start the Orient DB console via

*sudo java -jar ./lib/support/nexus-orient-console.jar*

and try to connect to the config DB via

*CONNECT plocal:/mnt/sonatype-work/nexus3/db/config admin admin*

we are getting error com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/mnt/sonatype-work/nexus3/db/config' with mode=rw DB name="config"

Connecting to database [plocal:/mnt/sonatype-work/nexus3/db/config] with user 'admin'... 2018-11-24 02:23:00:977 WARNI {db=config} Storage 'config' was not closed properly. Will try to recover from write ahead log... 2018-11-24 02:23:00:983 WARNI {db=config} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OCheckpointEndRecord{lsn=LSN{segment=215, position=52}} will be skipped during data restore$ANSI{green {db=config}} Exception during storage data restore java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.cutTill(ODiskWriteAheadLog.java:919) at

$ANSI{green {db=config}} Error during storage close java.lang.NullPointerException at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.cutTill(ODiskWriteAheadLog.java:919) at

Exception 475646D4 in storage plocal:/mnt/sonatype-work/nexus3/db/config: 2.2.31 (build 285537d2767275f460df32c6a3be01bfff6a517c, branch 2.2.x) com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/mnt/sonatype-work/nexus3/db/config' with mode=rw DB name="config" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:323)

The database looks in bad shape. When attempting to connect to it, orient detects the corruption and tries to replay transactions from the write ahead logs. The error above suggests the write ahead logs themselves are also corrupt.

If you are using a recent version of Nexus 3 but have no backup, here are some options:

1) Before doing anything else, stop all processes that may be accessing the databases - like console and Nexus.

2) Take a backup of directory /mnt/sonatype-work/nexus3/db/ before you attempt any more changes to it

3) Remove all files ending in *.wal in the directory of the config database ie. /mnt/sonatype-work/nexus3/db/config/ - or any other database that fails with NullPointerException against the write ahead log . This could result is losing some transactions that were in play prior to running out of disk, but since you have no backup, no real choice here.

rm /mnt/sonatype-work/nexus3/db/config/*.wal

4) Then try to start the console and repair the database

Option 1: At the orientdb prompt run the following:

connect plocal:/mnt/sonatype-work/nexus3/db/confg admin admin

repair database --fix-links

rebuild index *

disconnect exit

Then restart Nexus.

Option 2: If the above does not work (and it often doesn't) an export/import of the config database may work.  The instructions are the same as above, just a different set of commands at the console:

connect plocal:/mnt/sonatype-work/nexus3/db/confg admin admin export database config-export
drop database
create database plocal:/mnt/sonatype-work/nexus3/db/confg
import database config-export.json.gz -preserveClusterIDs=true
rebuild index *
disconnect
exit

Note : A duplicate index error may occur in the "import" step. This can be ignored, since the next step after it is rebuilding all indexes.

If we look at the processes running via htop we see multiple java -jar ./lib/support/nexus-orient-console.jar running.

This is odd. More than one process should not be attempting to open and modifying the same orient databases. If there is only one Nexus instance and one person trying to recover, there should only be one of these java orient console processes running.

 

Not sure if that is causing a problem connecting to the config db.

Any ideas or recommendations to fix the issue?

 

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/32a69f3c-2773-44fb-b8c0-875693170594%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Roy]

Hi Peter,

You are a godsend. I was able to recover the repository.

Our Nexus version is 3.8.0-02.

Thanks,
Shankar

[Peter Lynch]

Glad it worked out. You should promptly upgrade that version. It has some issues you are going to run into.