OrientDB manual clean up?

4,655 views
Skip to first unread message

NoMoreASAP

unread,
Sep 19, 2017, 10:19:43 AM9/19/17
to Nexus Users
Hey guys,
For some time now my nexus.log has been getting cluttered with these WARN/ERROR messages:

2017-09-19 08:40:00,820-0500 WARN  [quartz-1-thread-12] *SYSTEM org.sonatype.nexus.repository.storage.internal.StorageFacetManagerImpl - Unable to delete bucket with repository name uds-snapshots$05e10ee3-e5f1-4d7f-89c6-59db5b41cbfc, will require manual cleanup
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #28:166784 (cluster: asset_3)
        DB name="component"
...
Caused by: java.lang.OutOfMemoryError: Java heap space

and

2017-09-19 08:50:00,853-0500 ERROR [quartz-1-thread-12] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=component}} Exception `00867EF0` in storage `component`
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #28:166784 (cluster: asset_3)
        DB name="component"
...
Caused by: java.lang.OutOfMemoryError: Java heap space

Current memory settings are as follows:
-Xms2000M
-Xmx2000M
-XX:MaxDirectMemorySize=2G
Server has 8GB RAM available.

These should be sufficient according to Nexus documentation available at:
https://help.sonatype.com/display/NXRM3/System+Requirements
https://support.sonatype.com/hc/en-us/articles/115007093447-Optimizing-OrientDB-Database-Memory-
so I thought I would listen to what the log sais and try to clean up the problematic record in the database manually by:

1) Stop Nexus OSS app
2) cd into the application directory
3) java -jar ./lib/support/nexus-orient-console.jar to launch the console
4) CONNECT PLOCAL:/u02/sonatype-work/nexus3/db/component admin admin
5) delete from asset_3 where @rid=#28:166784
6) Restart Nexus OSS app

Does that look right to you guys? Has anyone actually played with the underlying OrientDB for Nexus?

Rok Prodan

unread,
Sep 19, 2017, 10:47:04 AM9/19/17
to NoMoreASAP, Nexus Users
Hello,

we had similar issues with retrieving malformed DB records on Nexus OSS 3.3.0-01 
I doubt you will succeed with step 5) because the record is malformed and thus not available.

We managed to "cure" our corrupted database with performing the following:
1) Exporting database components - EXPORT DATABASE /opt/components.export
2) Delete the entire database components physically (drop and truncate didn't work because of malformed records).
3) Create database - CREATE DATABASE PLOCAL:/home/nexus/sonatype-work/nexus3/db/component admin admin
4) Import database - IMPORT DATABASE /opt/components.export.gz -preserveClusterIDs=true

I would like to emphasize that this procedure worked for our case. 


Kind regards,

Rok Prodan 

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/d5c80036-26c7-4a02-b9b9-2d576506102e%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

Peter Lynch

unread,
Sep 19, 2017, 2:00:22 PM9/19/17
to NoMoreASAP, Nexus Users
On Tue, Sep 19, 2017 at 11:19 AM, NoMoreASAP <acxk...@gmail.com> wrote:
Hey guys,
For some time now my nexus.log has been getting cluttered with these WARN/ERROR messages:

2017-09-19 08:40:00,820-0500 WARN   [quartz-1-thread-12] *SYSTEM org.sonatype.nexus.repository.storage.internal.StorageFacetManagerImpl - Unable to delete bucket with repository name uds-snapshots$05e10ee3-e5f1-4d7f-89c6-59db5b41cbfc, will require manual cleanup
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #28:166784 (cluster: asset_3)
              DB name="component"
...
Caused by: java.lang.OutOfMemoryError: Java heap space

and

2017-09-19 08:50:00,853-0500 ERROR [quartz-1-thread-12] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=component}} Exception `00867EF0` in storage `component`
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #28:166784 (cluster: asset_3)
              DB name="component"
...
Caused by: java.lang.OutOfMemoryError: Java heap space

Current memory settings are as follows:
-Xms2000M
-Xmx2000M
-XX:MaxDirectMemorySize=2G
Server has 8GB RAM available.

These should be sufficient according to Nexus documentation available at:
https://help.sonatype.com/display/NXRM3/System+Requirements
https://support.sonatype.com/hc/en-us/articles/115007093447-Optimizing-OrientDB-Database-Memory-
so I thought I would listen to what the log sais and try to clean up the problematic record in the database manually by:

IF no other significant processes are running on the same host as Nexus, the System requirements page lists this as an example maximum configuration for a system with 8GB RAM:

-Xms2703M -Xmx2703M -XX:MaxDirectMemorySize=2703M

You really need to prevent the Out of memory errors first before adjusting anything else. If the problem continues please open an issue at https://issues.sonatype.org in the NEXUS project so that we can examine your complete log file and configuration privately.
 
1) Stop Nexus OSS app
2) cd into the application directory
3) java -jar ./lib/support/nexus-orient-console.jar to launch the console
4) CONNECT PLOCAL:/u02/sonatype-work/nexus3/db/component admin admin
5) delete from asset_3 where @rid=#28:166784
6) Restart Nexus OSS app

Does that look right to you guys? Has anyone actually played with the underlying OrientDB for Nexus?

NoMoreASAP

unread,
Sep 20, 2017, 6:54:47 AM9/20/17
to Nexus Users
There are no other significant processes running on that server, it is dedicated to running Nexus.
Whenever I try to bump heap memory and MaxDirectMemorySize above 1,8GB and 2GB respectively Nexus won't start at all or will ERROR out and die very quickly.

Could the server OS and architecture: RHEL5 32 Bit be the cause?

Grierson, David

unread,
Sep 20, 2017, 7:17:34 AM9/20/17
to NoMoreASAP, Nexus Users

32bit architecture will prevent you from addressing more than 2GB within a single process.

 

You need to get off of that onto a 64bit architecture.

 

Dg.

--

You received this message because you are subscribed to the Google Groups "Nexus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.


To post to this group, send email to nexus...@glists.sonatype.com.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

Peter Lynch

unread,
Sep 20, 2017, 8:57:01 AM9/20/17
to Grierson, David, NoMoreASAP, Nexus Users
On Wed, Sep 20, 2017 at 8:17 AM, Grierson, David <David.G...@sky.uk> wrote:

32bit architecture will prevent you from addressing more than 2GB within a single process.

 

You need to get off of that onto a 64bit architecture.

 


+1
 

Dg.

 

From: nexus...@glists.sonatype.com [mailto:nexus-users@glists.sonatype.com] On Behalf Of NoMoreASAP
Sent: 20 September 2017 11:55
To: Nexus Users <nexus...@glists.sonatype.com>
Subject: Re: OrientDB manual clean up?

 

There are no other significant processes running on that server, it is dedicated to running Nexus.
Whenever I try to bump heap memory and MaxDirectMemorySize above 1,8GB and 2GB respectively Nexus won't start at all or will ERROR out and die very quickly.

Could the server OS and architecture: RHEL5 32 Bit be the cause?

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

NoMoreASAP

unread,
Sep 20, 2017, 10:12:48 AM9/20/17
to Nexus Users, David.G...@sky.uk, acxk...@gmail.com
Thanks for advice - I will let you know once I've migrated to RHEL6.7/64Bit server.


On Wednesday, September 20, 2017 at 2:57:01 PM UTC+2, Peter Lynch wrote:
On Wed, Sep 20, 2017 at 8:17 AM, Grierson, David <David.G...@sky.uk> wrote:

32bit architecture will prevent you from addressing more than 2GB within a single process.

 

You need to get off of that onto a 64bit architecture.

 


+1
 

Dg.

 

From: nexus...@glists.sonatype.com [mailto:nexus...@glists.sonatype.com] On Behalf Of NoMoreASAP
Sent: 20 September 2017 11:55
To: Nexus Users <nexus...@glists.sonatype.com>
Subject: Re: OrientDB manual clean up?

 

There are no other significant processes running on that server, it is dedicated to running Nexus.
Whenever I try to bump heap memory and MaxDirectMemorySize above 1,8GB and 2GB respectively Nexus won't start at all or will ERROR out and die very quickly.

Could the server OS and architecture: RHEL5 32 Bit be the cause?

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

NoMoreASAP

unread,
Nov 3, 2017, 8:53:56 AM11/3/17
to Nexus Users
Hey,
I followed the recommendation and moved Nexus to a fresh RHEL6.7/64Bit server with 16GB of RAM available solely to the application.
Current memory settings:
-Xms4G
-Xmx4G
-XX:MaxDirectMemorySize=4014M

After starting I am still getting the WARN messages: Error on retrieving record indicating a corrupted db
however this time the stack trace didn't show them as Caused by: java.lang.OutOfMemoryError: Java heap space but rather: Caused by: java.nio.BufferUnderflowException: null
Manually deleting the corrupted record:
DELETE FROM CLUSTER:asset_3 WHERE @rid=#28:166784
doesn't work and throws java.nio.BufferUnderflowException

I went through the procedure described by Rok Prodan to cure the database.
1) During the export there were still a lot of java.nio.BufferUnderflowException's messages
but eventually the EXPORT completed with a summary:
Done. Exported 972978 of total 973491 records. 366 records were detected as broken

2) I deleted the database physically.

3) Created empty database

4) During the database import from the backup I get an error on duplicate indexes:
Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'component' from file: /u01/backup/components.export.gz

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #25:28352: found duplicated key 'OCompositeKey{keys=[null, null, null]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #25:11139
        DB name="component" INDEX=asset_bucket_component_name_idx RID=#25:11139

So I'm assuming the import doesn't succeed, any hints on how to get the OrientDB back in shape?
(Sorry if this is something obvious - I'm new to OrientDB)

Rok Prodan

unread,
Nov 3, 2017, 10:09:58 AM11/3/17
to NoMoreASAP, Nexus Users
Hello,

have you tried to manually rebuild index 'asset_bucket_component_name_idx'  again after import finished?
After try to rebuild all indexes on component database.


Best regards,

Rok Prodan

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.
Message has been deleted

NoMoreASAP

unread,
Nov 8, 2017, 7:31:03 AM11/8/17
to Nexus Users
Hey,
Yes, when trying to rebuild this mentioned index the import was complaining about:
orientdb {db=component}> REBUILD INDEX asset_bucket_component_name_
idx
Rebuilding index(es)...$ANSI{green {db=component}} Exception `10DB6131` in storage `component`
com.orientechnologies.orient.core.exception.OCommandExecutionException: Index 'asset_bucket_component_name_idx' not found
        DB name="component"
Error: com.orientechnologies.orient.core.exception.OCommandExecutionException: Index 'asset_bucket_component_name_idx' not found
        DB name="component"

I tried rebuilding all existing indexes for component database:
orientdb {db=component}> REBUILD INDEX *
Rebuilding index(es)...
Rebuilt index(es). Found 511296 link(s) in 21.903999 sec(s).
Index(es) rebuilt successfully

I also ran all of the REPAIR DATABASE commands:
orientdb {db=component}> REPAIR DATABASE --fix-graph
orientdb {db=component}> REPAIR DATABASE --fix-links
orientdb {db=component}> REPAIR DATABASE --fix-ridbags
orientdb {db=component}> REPAIR DATABASE --fix-bonsai
All 4 executions didn't find any errors:
Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors)

Check database also doesn't show any errors:
orientdb {db=component}> CHECK DATABASE
Check of storage is started...Check of storage completed in 396ms.  without errors.

I'm still seeing the warning messages on some records:
2017-11-08 06:04:50,745-0600 WARN  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.repository.storage.internal.StorageFacetManagerImpl - Unable to delete bucket with repository name xxx-releases$debe6a88-1a4e-4771-843f-66ec6fdde45b, will require manual cleanup
com.orientechnologies.orient.core.exception.ORecordNotFoundException: The record with id '#25:20945' was not found
        DB name="component"

Any other thoughts on how to cure the database?

NoMoreASAP

unread,
Nov 20, 2017, 7:45:33 AM11/20/17
to Nexus Users
In case anyone ever stumbles upon a similar problem: I have went through the procedure once again but this time it worked (there were still corrupted records upon export but no issues with the indexes after the import and Nexus was able to clean up the malformed records).
The only difference this time was using Nexus 3.6.1-02 instead of 3.6.0-02

Thanks for all the advice guys!
Reply all
Reply to author
Forward
0 new messages