After upgrade not possible to contact server

[mikael petterson]

Hi,

We are using SonarQube 5.6.6 and when I upgraded from:

http://<myip>/sonar/updatecenter/installed

I think it was java  and javascript plugins that I selected and upgraded to latest available.
This required restart of server and I did so using gui then nothing happened ( waited for 5-6 min and nothing).
So I logged into the server ( ssh ) and I did:

sudo /etc/init.d/sonar restartStopping SonarQube...
Waiting for SonarQube to exit...
Waiting for SonarQube to exit...
Stopped SonarQube.
Starting SonarQube...
Started SonarQube.
<userid>@seliius01837:/opt/sonar/bin/linux-x86-32$ sudo /etc/init.d/sonar status
SonarQube is running (4298).

I still cannot access server.

What should I do? No errors in log, sonar.log

2017.08.11 08:17:56 INFO   es[o.s.p.ProcessEntryPoint]  Starting es
2017.08.11 08:17:56 INFO   es[o.s.s.EsSettings]  Elasticsearch listening on 127.0.0.1:9001
2017.08.11 08:17:57 INFO   es[o.elasticsearch.node]  [sonar-1502432273683] version[1.7.5], pid[1227], build[00f95f4/2016-02-02T09:55:30Z]
2017.08.11 08:17:57 INFO   es[o.elasticsearch.node]  [sonar-1502432273683] initializing ...
2017.08.11 08:17:58 INFO   es[o.e.plugins]  [sonar-1502432273683] loaded [], sites []
2017.08.11 08:17:58 INFO   es[o.elasticsearch.env]  [sonar-1502432273683] using [1] data paths, mounts [[/ (/dev/mapper/seliius01837--vg-root)]], net usable_space [16.2gb], net total_space [24.9gb], types [ext4]
2017.08.11 08:18:01 WARN   es[o.e.bootstrap]  JNA not found. native methods will be disabled.
2017.08.11 08:18:03 INFO   es[o.elasticsearch.node]  [sonar-1502432273683] initialized
2017.08.11 08:18:03 INFO   es[o.elasticsearch.node]  [sonar-1502432273683] starting ...
2017.08.11 08:18:03 INFO   es[o.e.transport]  [sonar-1502432273683] bound_address {inet[/127.0.0.1:9001]}, publish_address {inet[/127.0.0.1:9001]}
2017.08.11 08:18:03 INFO   es[o.e.discovery]  [sonar-1502432273683] sonarqube/i5Df6uFSQt6-3YsrKkLPIQ
2017.08.11 08:18:06 INFO   es[o.e.cluster.service]  [sonar-1502432273683] new_master [sonar-1502432273683][i5Df6uFSQt6-3YsrKkLPIQ][seliius01837.seli.gic.ericsson.se][inet[/127.0.0.1:9001]]{rack_id=sonar-1502432273683}, reason: zen-disco-join (elected_as_master)
2017.08.11 08:18:06 INFO   es[o.elasticsearch.node]  [sonar-1502432273683] started
2017.08.11 08:18:06 INFO   es[o.e.gateway]  [sonar-1502432273683] recovered [6] indices into cluster_state
2017.08.11 08:18:35 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][young][29][15] duration [741ms], collections [1]/[1s], total [741ms]/[3.7s], memory [251.6mb]->[202.2mb]/[1007.3mb], all_pools {[young] [86.8mb]->[23.7mb]/[133.1mb]}{[survivor] [14.8mb]->[16.6mb]/[16.6mb]}{[old] [149.8mb]->[161.9mb]/[857.6mb]}
2017.08.11 08:19:44 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][young][94][52] duration [793ms], collections [1]/[1s], total [793ms]/[12.7s], memory [643.1mb]->[574.7mb]/[1007.3mb], all_pools {[young] [82.7mb]->[3.3mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [543.8mb]->[554.7mb]/[857.6mb]}
2017.08.11 08:21:09 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][old][154][13] duration [6s], collections [1]/[6s], total [6s]/[20s], memory [1006.7mb]->[887mb]/[1007.3mb], all_pools {[young] [133.1mb]->[29.3mb]/[133.1mb]}{[survivor] [15.9mb]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
2017.08.11 08:21:21 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][old][158][15] duration [5.7s], collections [1]/[6.2s], total [5.7s]/[29.5s], memory [964.7mb]->[900.7mb]/[1007.3mb], all_pools {[young] [107mb]->[43.1mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
2017.08.11 08:21:34 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][old][162][17] duration [5.5s], collections [1]/[6.5s], total [5.5s]/[38.4s], memory [980.9mb]->[917.7mb]/[1007.3mb], all_pools {[young] [123.2mb]->[60.1mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
2017.08.11 08:21:45 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][old][166][19] duration [5.5s], collections [1]/[5.7s], total [5.5s]/[47.4s], memory [989.7mb]->[930.7mb]/[1007.3mb], all_pools {[young] [132.1mb]->[73.1mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
2017.08.11 08:21:56 INFO   es[o.e.monitor.jvm]  [sonar-1502432273683] [gc][old][170][21] duration [5.2s], collections [1]/[5.6s], total [5.2s]/[56.2s], memory [1004.7mb]->[942.5mb]/[1007.3mb], all_pools {[young] [133.1mb]->[84.9mb]/[133.1mb]}{[survivor] [14mb]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}

[mikael petterson]

I checked the log after a while:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1227.hprof ...
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807
Heap dump file created [1382941435 bytes in 98.650 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

How can I increase HeapSpace?

//mikael

[mikael petterson]

Strange when I restart sonar I don't see web server being started and this time no error in sonar.log
I am confused I really need some help here :-)

//mikael

[Duarte Meneses]

Hi Mikael,

FYI, sonar.log aggregates logs from the three processes: web, elastic search (es) and compute engine (ce). You can check the logs for each process in [sonarqube]/logs/ (es.log, ce.log and web.log).

It seems that the elastic search process is running out of memory. To increase the heap size, you need to change the property sonar.search.javaOpts in [sonarqube]/conf/sonar.properties.

Example to have a heap of 3gb:

sonar.search.javaOpts=-Xmx3g

[mikael petterson]

Hi Duarte,

Yes I increased the elastic search memory. I started with 2G and that did the trick!

Thx for your assistance.

//mikale