Upgrade to 5.6.4 blocked due to OOM error in web service

[Bernd G.]

My plan was to migrate Sonarqube from version 5.1 to version 5.6, but I have to stay with 5.1 for now due to a OutOfMemory issue I cant' get rid of.

I'm running a batch of analysis jobs using the Sonar Runner 2.8. I have about 60 projects that I analyze one after the other. After about 10 projects, the sonar runner can't access the web service anymore, and the web service is in a critical state with the initial OOM error below:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at org.joni.Matcher.<init>(Matcher.java:61) ~[jruby-complete-1.7.9.jar:na]
    at org.joni.StackMachine.<init>(StackMachine.java:44) ~[jruby-complete-1.7.9.jar:na]
    ...

The full stack trace is in the attached file.

I'm running on Windows 7, with Java version:

> java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

I increased Xmx values for the service, with no change in behavior.

Any suggestions?

[nicolas...@sonarsource.com]

Hi there,

Hmm, seems like you're still the embedded H2 database:

2017.01.10 12:02:39 INFO  web[o.sonar.db.Database] Create JDBC data source for jdbc:h2:tcp://localhost:9092/sonar
2017.01.10 12:02:39 WARN  web[o.s.d.DatabaseChecker] H2 database should be used for evaluation purpose only

And you're starting to pay the price (memory exception directly related to embedded database):

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.h2.engine.Database.openDatabase(Database.java:277)
... 7 more

What was your plan for upgrading anyhow ? SonarQube does not support upgrading H2 databases (should only be used for testing/evaluation purpose).

Looks like you really need to get rolling with a full-fledged supported database (see SonarQube v5.6 requirements ).

Best regards,

Nicolas

[Bernd Götz]

Hi Nicolas,

I always use a local Sonar instance with an h2 database for testing purposes. I use such a local setup for SonarQube upgrades and plugin testing before rolling those changes out to our main live instance. That live instance is connected to an Oracle schema, of course. As you recommend.

I've never had any such issues with 5.1.2 and previous versions, and my test case is one single analysis loop over all 60+ projects with the sonar loader. Never a single OOM issue. Something has changed in behavior in 5.6+. Imho, the way how the previous versions handled the max number of analysis import requests was more robust. It seems that version 5.6+ doesn't have such a throttling in place, so that after a number of requests running in parallel on the server, it runs into trouble, but more tasks keep rolling in. It does not check capacity. Just a thought.

Why do you think it's connected to the use of the H2 database? Maybe the H2 database version got more memory greedy? I haven't checked the H2 versions used in 5.1 and 5.6.

I would very much like to migrate to 5.6 but I have to stay with 5.1 for now. I've back ported my internal plugin changes from 5.6 API to the 5.1 API. When time allows, I willl look into the topic again, but not now.

Let me know if you want me to collect some more data for you.

Best regards,

Bernd

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/ZNe-HpJ58fE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/a704a4c7-3ca5-4243-9b98-59b0112751cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bernd Götz]

Hi Nicolas,

I've switched to an Oracle schema today to rule out any H2 related memory issues. I'm still experiencing the same behavior. I've configured Sonarqube heap sizes as follows:

Web service: -Xmx2G

Compute engine: -Xmx2G

Search engine: -Xmx2G

After a few sonar runner (2.8) runs, the Web service process is really very busy, and turns non reponsive after a few analysis runs, here are two screenshots of the  processes with the Process Explorer. The first is with -Xmx1G, the second with -Xmx2G:





Heap space is used up quite quickly, and the CPU is being heavily used by the Web service, too, as you can see below:

Another example, from a later test cycle:

This never happens to me with 5.1 and blocks me from upgrading to 5.6 and beyond. What do you suggeest as next steps? What further information should I provide to you? How should I analyze the Web service process in more details?

[nicolas...@sonarsource.com]

Hi Bernd,

FYI there's been heavy architectural changes between SonarQube v5.1 and v5.6.x, allowing scanners to not require any connection to database (see 5.2 release blog). The new architecture is detailed in the Architecture and Integration documentation. One of the key takeaways here being that scanner-side activity triggers processing of analysis reports on the server side (Compute Engine), and it's a good practice to have monitoring in place to make sure that JVMs can cope with the load (depending on your analysis activity, project size etc. really deployment specific considerations).

In your screenshots I do not see any specific mention of Web JVM as the one being under pressure. In terms of next steps I would suggest you opt more for Java-oriented monitoring tools (no specific recommendation there) that will let you understand more precisely behaviour of each JVM. On a more advanced note, there is a also some documentation page about SonarQube monitoring with JMX beans.

All that should let you move forward and fine-tune your SonarQube configuration along the way.

Best regards,

Nicolas

[Bernd Götz]

Hello Nicolas,

thanks for the input. I need to decide to invest more in 5.6 or to look into the later versions. I then decided to upgrade to version 6.2, and I haven't seen any of my issues with 6.2, that I had seen with 5.6 before. 6.2 seems to be more stable with the workload it gets. I am losing the dashboarding feature of 5.6/6.0/6.1, that is very unfortunate but I think I can sell this in my organization.

Best regards,

Bernd

--

You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/ZNe-HpJ58fE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/31928922-8763-4cc0-984a-13c9ad1db4db%40googlegroups.com.