Nexus3.26.1-02 got JVM out

[Mariska]

Hi,

Nexus was running very stable for the last year and I upgrade very regulary, however after the last upgrade to 3.26.1-02 (last Friday), I saw the following WARN's in the nexus.log starting this Monday:

```

2020-08-31 09:36:23,095+0200 WARN  [qtp1168505502-12066]  username org.sonatype.nexus.repository.npm.internal.NpmAuditErrorHandler - Can't get hashsum for the appId [null] type [npm] name [jsonstream] version [1.3.5] package
org.sonatype.nexus.repository.vulnerability.exceptions.TarballLoadingException: Can't get hashsum for the appId [null] type [npm] name [jsonstream] version [1.3.5] package
       at org.sonatype.nexus.repository.npm.internal.NpmAuditTarballFacet.download(NpmAuditTarballFacet.java:131)
       at org.sonatype.nexus.repository.npm.internal.NpmAuditFacet.getAuditRepositoryComponents(NpmAuditFacet.java:238)
       at org.sonatype.nexus.repository.npm.internal.NpmAuditFacet.getComponentsVulnerabilityFromRemoteServer(NpmAuditFacet.java:211)
       at org.sonatype.nexus.repository.npm.internal.NpmAuditFacet.analyzeComponents(NpmAuditFacet.java:186)
       at org.sonatype.nexus.repository.npm.internal.NpmAuditFacet.audit(NpmAuditFacet.java:140)
       at org.sonatype.nexus.repository.npm.internal.NpmGroupAuditQuickHandler.handle(NpmGroupAuditQuickHandler.java:41)
       at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

```

and after 2 days of about 20 of these warnings, nexus became read-only last night with java memory errors:

```

2020-09-01 19:44:55,839+0200 ERROR [qtp1168505502-20083]  *SYSTEM org.eclipse.jetty.util.log.StdErrLog - Exception in thread "event-12-thread-7288" Exception in thread "nexus-httpclient-eviction-thread" 2020-09-01 19:43:13.154
:WARN:oeji.ManagedSelector:qtp1168505502-20083: java.lang.OutOfMemoryError: Java heap space
```

2020-09-01 21:27:56,025+0200 WARN  [quartz-9-thread-20 <query>SELECT FROM healthcheckconfig WHERE property_name = ?</query>]  *SYSTEM org.sonatype.nexus.quartz.internal.orient.JobStoreImpl - Execution failed
com.orientechnologies.orient.core.exception.OJVMErrorException: JVM error 'OutOfMemoryError : Java heap space' occurred during data processing, storage is switched to 'read-only' mode. To prevent this exception please restart
the JVM and check data consistency by calling of 'check database' command from database console.
       DB name="config"
```

After restart everything seems to work again, but I'm not very confident that it won't happen again. I'm not sure if the WARN "org.sonatype.nexus.repository.npm.internal.NpmAuditErrorHandler - Can't get hashsum for the appId" is related to the out of memory issue, but it seems to be a new Warning.

Could it be something changed in the last release that caused this? Of course I can increase the java heap memory, but that won't help if this is caused by some other error.

Any help would be appreciated.

Mariska.

[Joseph Stephens]

Hi Mariska,

We believe the exception you see is related to this bug https://issues.sonatype.org/browse/NEXUS-24917. This is fixed in 3.27.0 which is expected to land early next week.

We're not sure why that would cause an Out of Memory error but the team is taking a look. Could you file a bug at https://issues.sonatype.org and if possible include a heap dump?

Thanks,

Joe

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/0299605c-d0ad-4278-92ab-c2f802cd310en%40glists.sonatype.com.

--

Joseph Stephens NXRM Product Manager

[Mariska]

Thanks, and I will do so, but it appears that creating a heap dump in a docker container with only the jre installed is not trivial. Tried with a piggy-back container with the jdk installed, but no luck yet. I'll try to reproduce the same in my test environment and run it in a container with full jdk. So, this is definitely something for me to add: being able to monitor better...

[Rich Seddon]

You could try adding this to the java startup arguments, this will cause a heap dump to be generated in the path specified if there is another out of memory error:

-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/path/to/directory

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/552765df-98e6-43e7-b98e-6c6dc76e8bb7n%40glists.sonatype.com.

--

Richard Seddon Manager of Product Support

[Mariska]

Thanks, Rich. I've already done that in our test environment and will do that in production as soon as possible.

[Mariska]

Hi Rich, 

I've also enabled the GC settings according to: https://support.sonatype.com/hc/en-us/articles/360000870607-Java-Garbage-Collection-Logging, as I noticed that the heap grows to over 90% before starting to cleanup. I find that pretty high. We have set the recommended settings of 4GB heap size.

Are there any settings recommended with respect to GC for nexus3? Or would you recommend setting the heap beyond 4GB?

Thanks,

Mariska

[Rich Seddon]

The default Java GC algorithm tends to allocate memory towards the top of the configured heap space before doing a full GC sweep.  So you'll see a sawtooth pattern in heap usage. As long as it goes back down to a reasonable level after the full sweep it should be fine, more memory isn't needed.

Rich

[Mariska]

Hello,

We had again an out of memory issue with NXRM3 3.27.0-03 and I've got a head dump now. It's rather big and I suspect it might also contain sensitive data and therefore  I'am hesitant to share it in the sonatype issue tracker.  Maybe you can let me know what to look for in the heap dump?

I've also gone through the nexus and request log and on the day of the dump, I saw a huge number of bad requests (more than 3500 before the out-of-memory) in the request.log:

```
"GET /repository/cc-nuget HTTP/1.1" 400
```

resulting in the same number of WARNs in the nexus log:

```
WARN  [qtp73856825-731]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Bad request. Reason: Repository path must have another '/' after initial '/'
```

I'm already in contact with the developers making these bad requests to change the nuget.config, do not really believe it might be causing this out-of-memory error, but I do wonder if this somehow caused the problem. I will try to reproduce this in my test environment.

Other ERROR's in the nexus log before the memory dump:

```
2020-10-06 08:58:28,960+0200 ERROR [qtp632432402-87402]  *SYSTEM org.eclipse.jetty.util.log.StdErrLog - 2020-10-06 08:58:28.960:WARN:oejs.HttpChannel:qtp632432402-87402: /static/rapture/baseapp-prod.js
java.lang.IllegalStateException: ABORTED
      at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:896)
      at org.eclipse.jetty.server.Response.sendError(Response.java:466)
```

And just before the memory dump:
```
2020-10-06 16:16:40,947+0200 ERROR [nexus_QuartzSchedulerThread]  *SYSTEM org.quartz.core.ErrorLogger - An error occurred while scanning for the next triggers to fire.
org.quartz.JobPersistenceException: com.orientechnologies.orient.core.exception.OCommandExecutionException: Cannot execute query "SELECT * FROM quartz_trigger WHERE state IN [ ?,?,? ]": low heap memory
       DB name="config"
       at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.acquireNextTriggers(JobStoreImpl.java:957)
       at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:287)
Caused by: com.orientechnologies.orient.core.exception.OCommandExecutionException: Cannot execute query "SELECT *
FROM quartz_trigger WHERE state IN [ ?,?,? ]": low heap memory
       DB name="config"
```
Please let me know what to do with the heap dump. I will try to investigate myself as well, but any help is highly appreciated. 

We are monitoring the heap memory now and it looks like:

I've also got garbage collect logging.

[Rich Seddon]

If the cause is indeed NuGet, have a look here... there was a change in 3.23.0 which can cause this:

https://support.sonatype.com/hc/en-us/articles/360054068493-Solving-NuGet-Performance-Problems-in-Nexus-Repo-3 

[Mariska]

Ah thanks. I will investigate if we can move to nuget V3. It could indeed be that this causes the memory issues in our instance. I will let you know.

[Mariska]

Hello Rich, 

one question: as an alternative to temporary disabling the fix (nexus.nuget.multiple.latest.fix=false), could we also increase the heap memory to above 4GB and optimize garbage collection until all projects have moved to nuget v3?

Please let me know your thougts on that.

Regards, Mariska