5.2: java.lang.OutOfMemoryError: Java heap space

[andre....@gmail.com]

Hi,

 

since migration to 5.2 for my largest project, I am getting the following error in the background task logs

there is no change in analysis, just the migration to 5.2 from 5.1.2

 

2015.12.07 22:44:12 INFO  [o.s.s.c.s.ComputationStepExecutor] Load measure computers | time=0ms
2015.12.07 23:02:09 ERROR [o.s.s.c.c.ComputeEngineContainerImpl] Cleanup of container failed
java.lang.OutOfMemoryError: Java heap space
        at java.io.WinNTFileSystem.list(Native Method) ~[na:1.8.0_60]
        at java.io.File.list(File.java:1122) ~[na:1.8.0_60]
        at java.io.File.listFiles(File.java:1207) ~[na:1.8.0_60]
        at org.sonar.api.internal.apachecommons.io.FileUtils.cleanDirectory(FileUtils.java:1645) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.internal.apachecommons.io.FileUtils.deleteDirectory(FileUtils.java:1535) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.internal.apachecommons.io.FileUtils.forceDelete(FileUtils.java:2270) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.internal.apachecommons.io.FileUtils.cleanDirectory(FileUtils.java:1653) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.internal.apachecommons.io.FileUtils.deleteQuietly(FileUtils.java:1566) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.utils.internal.DefaultTempFolder.clean(DefaultTempFolder.java:89) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.api.utils.internal.DefaultTempFolder.stop(DefaultTempFolder.java:94) ~[sonar-plugin-api-5.2.jar:na]
        at org.sonar.server.computation.filesystem.ComputationTempFolderProvider.stop(ComputationTempFolderProvider.java:65) ~[sonar-server-5.2.jar:na]
        at org.picocontainer.DefaultPicoContainer.stopAdapters(DefaultPicoContainer.java:1048) ~[picocontainer-2.15.jar:na]
        at org.picocontainer.DefaultPicoContainer.stop(DefaultPicoContainer.java:803) ~[picocontainer-2.15.jar:na]
        at org.sonar.core.platform.ComponentContainer.stopComponents(ComponentContainer.java:131) ~[sonar-core-5.2.jar:na]
        at org.sonar.core.platform.ComponentContainer.stopComponents(ComponentContainer.java:126) ~[sonar-core-5.2.jar:na]
        at org.sonar.server.computation.container.ComputeEngineContainerImpl.cleanup(ComputeEngineContainerImpl.java:109) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.taskprocessor.report.ReportTaskProcessor.process(ReportTaskProcessor.java:55) [sonar-server-5.2.jar:na]
        at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.executeTask(CeWorkerRunnableImpl.java:78) [sonar-server-5.2.jar:na]
        at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.run(CeWorkerRunnableImpl.java:55) [sonar-server-5.2.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_60]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_60]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
2015.12.07 23:02:09 ERROR [o.s.s.c.t.CeWorkerRunnableImpl] Failed to execute task AVF-QAWZB_ii9u5J1Iqo
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149) ~[na:1.8.0_60]
        at java.lang.StringCoding.decode(StringCoding.java:193) ~[na:1.8.0_60]
        at java.lang.String.<init>(String.java:426) ~[na:1.8.0_60]
        at java.lang.String.<init>(String.java:491) ~[na:1.8.0_60]
        at com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:438) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:2503) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:193) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:132) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2082) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2067) ~[sqljdbc41.jar:na]
        at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2408) ~[sqljdbc41.jar:na]
        at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213) ~[commons-dbcp-1.4.jar:1.4]
        at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213) ~[commons-dbcp-1.4.jar:1.4]
        at org.sonar.db.source.FileSourceDao.selectLineHashes(FileSourceDao.java:74) ~[sonar-db-5.2.jar:na]
        at org.sonar.server.computation.issue.TrackerBaseInputFactory$BaseLazyInput.loadLineHashSequence(TrackerBaseInputFactory.java:62) ~[sonar-server-5.2.jar:na]
        at org.sonar.core.issue.tracking.LazyInput.getLineHashSequence(LazyInput.java:34) ~[sonar-core-5.2.jar:na]
        at org.sonar.core.issue.tracking.LazyInput.getBlockHashSequence(LazyInput.java:42) ~[sonar-core-5.2.jar:na]
        at org.sonar.core.issue.tracking.BlockRecognizer.match(BlockRecognizer.java:40) ~[sonar-core-5.2.jar:na]
        at org.sonar.core.issue.tracking.Tracker.detectCodeMoves(Tracker.java:62) ~[sonar-core-5.2.jar:na]
        at org.sonar.core.issue.tracking.Tracker.track(Tracker.java:45) ~[sonar-core-5.2.jar:na]
        at org.sonar.server.computation.issue.TrackerExecution.track(TrackerExecution.java:41) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:67) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:59) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:83) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:97) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:80) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:90) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:77) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:90) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:77) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:90) ~[sonar-server-5.2.jar:na]
        at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:77) ~[sonar-server-5.2.jar:na]

[Sebastien Lesaint]

Hello,

SonarQube 5.2 has much more processing going on the server side than SonarQube 5.1 had.

This explain why more memory might be required to process big projects.

The JVM process of the WebServer (in which the Compute Engine is running) might run out of memory to allocate and throw a OutOfMemory error.

Are you using the default memory configuration of SQ 5.2?

Check property "sonar.web.javaOpts" in your sonar.properties file. If it is commented out, you are using the default configuration.

Try and increase the value of the -Xmx argument (eg. -Xmx1024m, -Xmx1280 and so on...).

Please let us know how much memory was required for your project

And, if you don't mind, could you give us some figures to get an idea how big your project is, please ?

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/ee5cbbc4-02cd-4997-bd93-efbed0861a70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[andre....@gmail.com]

Hi,

which information do you want, then I can see what I can give you.

I am currently still in investigation and test, as after a sonar restart the application does not want to start with this error

WrapperSimpleApp: Encountered an error running main: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.io.WinNTFileSystem.list(Native Method)
    at java.io.File.list(File.java:1122)
    at java.io.File.listFiles(File.java:1207)
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1645)
    at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
    at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
    at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
    at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
    at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
    at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
    at org.apache.commons.io.FileUtils.deleteQuietly(FileUtils.java:1566)
    at org.sonar.application.PropsBuilder.initTempDir(PropsBuilder.java:102)
    at org.sonar.application.PropsBuilder.build(PropsBuilder.java:73)
    at org.sonar.application.App.main(App.java:115)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
    at java.lang.Thread.run(Thread.java:745)
WrapperSimpleApp: start(args) end.  Main Completed=true, exitCode=1

i had to delete the temp folder content

some infos from first run with sonarqube 5.2:
21:54:21.733 INFO  - Analysis reports generated in 471705ms, dir size=1 GB
22:01:25.176 INFO  - Analysis reports compressed in 423443ms, zip size=328 MB
22:01:45.732 INFO  - Analysis reports sent to server in 20556ms
INFO: ------------------------------------------------------------------------

Total time: 1:52:24.558s
Final Memory: 31M/4775M

compared to run with sonarqube 5.1.2:
03:52:38.094 INFO  - Store results in database
04:01:08.665 INFO  - Analysis reports generated in 114846ms, dir size=84 MB
04:03:27.216 INFO  - Analysis reports compressed in 138551ms, zip size=46 MB
04:03:29.759 INFO  - Analysis reports sent to server in 2543ms
INFO: ------------------------------------------------------------------------
Total time: 2:21:37.330s
Final Memory: 167M/5097M

On Tuesday, December 8, 2015 at 6:46:23 PM UTC+1, Sebastien Lesaint wrote:

Hello,

SonarQube 5.2 has much more processing going on the server side than SonarQube 5.1 had.

This explain why more memory might be required to process big projects.

The JVM process of the WebServer (in which the Compute Engine is running) might run out of memory to allocate and throw a OutOfMemory error.

Are you using the default memory configuration of SQ 5.2?

Check property "sonar.web.javaOpts" in your sonar.properties file. If it is commented out, you are using the default configuration.

Try and increase the value of the -Xmx argument (eg. -Xmx1024m, -Xmx1280 and so on...).

Please let us know how much memory was required for your project

And, if you don't mind, could you give us some figures to get an idea how big your project is, please ?

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

...

[andre....@gmail.com]

And by the way yes, I have used default settings for sonar.web.javaOpts
I only added -server option

[Sebastien Lesaint]

Hello,

From the Analysis reports logs you gave, I get a pretty sense of how big your project are (1Gb report is a lot), presumably with a lot of files (since compression rate of the report is pretty good, there must quite some source code).

Could you give us the size measures of your project (files, directories, ncloc, ...), the language(s) and how big are the biggest files you have?

Regarding the OOM you faced at startup, I've opened ticket SONAR-7125 to fix it and the workaround is indeed to delete the temp folder manually.

Let us know how much you increased the max heap of the WebServer JVM to get your project processed.

Thanks

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

--

You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8d94e2c2-bd26-48b8-8150-b04270f443cb%40googlegroups.com.

[andre....@gmail.com]

Hi,

new update: At First, the analysis is still not running to an end, currently I have -Xmx2048, next try will be with -Xmx4096

the biggest issue I see is, that currently it is running 270min and then it I get an out of memory exception. In 5.1.2 it definitely did not take so long, the analysis was much faster, on client side, we are now ~10-15mins faster, but on server Side it did not take 4 hours!
Also what I saw is, that at least for 170mins the website were not reachable anymore, the server seems to be down.
On server side I observed 95% CPU usage on the 4 Core machine, over that time!

 

2015.12.10 15:03:38 INFO  web[o.e.client.transport] [sonar-1449737149903] failed to get node info for [#transport#-1][][inet[/127.0.0.1:9022]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/127.0.0.1:9022]][cluster:monitor/nodes/info] request_id [3492] timed out after [19126ms]
    at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529) ~[elasticsearch-1.7.2.jar:na]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

2015.12.10 15:01:04 ERROR web[o.a.t.u.n.NioEndpoint]
java.lang.OutOfMemoryError: Java heap space

 

Regarding the other Data, I will collect it, and see which information I can give you and how.

What I can say now, we are currently just/mainly analyzing C# and XML in this project, the C++ parts are yet not in the analysis, which would make it the report larger...

[Sebastien Lesaint]

Hello,

Thank you for the update.

Regarding the C++ parts, are they excluded from the analysis in your project configuration?

You can have a look at the documentation if you don't know how this can be achieved: http://docs.sonarqube.org/display/SONAR/Narrowing+the+Focus.

We will definitely have a go at the memory usage of the Compute Engine for SQ 5.4. It won't be for 5.3 as the release is already ongoing.

The more information you can give us on your project, the better we will be able to handle it in the CE. So, thanks in advance for any help you can provide.

If you don't want some information to be public on the mailing list, you can send them to me directly.

I don't want to sound like a sales man, but since you seem to have a very demanding project to analyse, an option is to seek commercial support by getting in touch with our sales team. There is only limited things we can do on the mailing list while respecting confidentiality and limited time we dedicate to community support.

Cheers,

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/cbc360d1-c6ab-4ba2-a477-6a88a1d793db%40googlegroups.com.

[andre....@gmail.com]

Hi,

with Xmx4096 it now finally finished, after 2 hours of processing on Sonar Server.
I have started a second run, I will see the results tomorrow.
Project Details I can send you via mail, tomorrow.

Currently there is no C++ Plugin Installed, and I am quite familiar with the mentioned topic.
I have a well defined Module/Component Structure reflecting the Source Control structure.

I don't want to sound like a sales man, but since you seem to have a very demanding project to analyse, an option is to seek commercial support by getting in touch with our sales team. There is only limited things we can do on the mailing list while respecting confidentiality and limited time we dedicate to community support.

I know, up to now it was not necessary and the issues are comming now with SQ5.2 ;)
I am running my analysis since quite a long time without any trouble or issues, for all my projects
It all runs quite smooth, but currently I also have not closed the Dev cycle with QualityGates and "BuildBreaker"
waiting here also for https://jira.sonarsource.com/browse/SONAR-6763

Nevertheless I am missing here more connection to the DevEnv (directly open the Code in VisualStudio for fixing issues, or showing the issues in VisualStudio,yeah we are Microsoft ;) )
Also there is currently no easy connection to our Project Planning tool, for creating bugs or connect Actions plans with our Backlog.

[Sebastien Lesaint]

Hello Andre,

Thanks for keeping us informed on this and glad the processing went through. Indeed, it's a lot of heap and a quite a duration. I can't tell if it's unexpected without informations about the project.

Would you mind sharing (privately if you want) the logs of the background task? You can access them through the "background tasks" page. It contains only generic informations and timings. These timings would be valuable information.

Cheers,

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/52a646b2-7e18-405b-bf5a-84003b0a9879%40googlegroups.com.