web listener doesn't start anymore

[nico.mo...@gmail.com]

Hey guys,

I have this strange problem. I restarted our test SonarQube instance (uptime of several months), but it seems only the elastic search is getting started. I cannot acces the webui anymore.

I can see the these processes running in Linux:

sonar    18764  0.0  0.0  19128   744 ?        Sl   17:15   0:00 /data/sonar/sonarqube-5.6.3/bin/linux-x86-64/./wrapper /data/sonar/sonarqube-5.6.3/bin/linux-x86-64/../../conf/wrapper.conf wrapper.syslog.ident=SonarQube wrapper.pidfile=/data/sonar/sonarqube-5.6.3/bin/linux-x86-64/./SonarQube.pid wrapper.daemonize=TRUE
sonar    18766  0.2  1.8 2004596 36372 ?       Sl   17:15   0:03 java -Djava.awt.headless=true -Xms3m -Xmx3m -Djava.library.path=./lib -classpath ../../lib/jsw/wrapper-3.2.3.jar:../../lib/sonar-application-5.6.3.jar -Dwrapper.key=1B9qF9HRIQ6R45Ua -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=18764 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.sonar.application.App
sonar    18790 98.3 61.5 3144104 1183368 ?     Sl   17:16  26:30 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/data/sonar/sonarqube-5.6.3/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /data/sonar/sonarqube-5.6.3/temp/sq-process5342876113168521904properties

but the logging only shows elastic starting up:

--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
2017.02.03 17:16:01 INFO  app[o.s.a.AppFileSystem] Cleaning or creating temp directory /data/sonar/sonarqube-5.6.3/temp
2017.02.03 17:16:01 INFO  app[o.s.p.m.JavaProcessLauncher] Launch process[es]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/data/sonar/sonarqube-5.6.3/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /data/sonar/sonarqube-5.6.3/temp/sq-process5342876113168521904properties
2017.02.03 17:16:02 INFO   es[o.s.p.ProcessEntryPoint]  Starting es
2017.02.03 17:16:02 INFO   es[o.s.s.EsSettings]  Elasticsearch listening on 127.0.0.1:9001
2017.02.03 17:16:03 INFO   es[o.elasticsearch.node]  [sonar-1486138560565] version[1.7.5], pid[18790], build[00f95f4/2016-02-02T09:55:30Z]
2017.02.03 17:16:03 INFO   es[o.elasticsearch.node]  [sonar-1486138560565] initializing ...
2017.02.03 17:16:03 INFO   es[o.e.plugins]  [sonar-1486138560565] loaded [], sites []
2017.02.03 17:16:03 INFO   es[o.elasticsearch.env]  [sonar-1486138560565] using [1] data paths, mounts [[/data (/dev/mapper/datavg-data)]], net usable_space [40.6gb], net total_space [127.9gb], types [ext3]
2017.02.03 17:16:06 WARN   es[o.e.bootstrap]  JNA not found. native methods will be disabled.
2017.02.03 17:16:08 INFO   es[o.elasticsearch.node]  [sonar-1486138560565] initialized
2017.02.03 17:16:08 INFO   es[o.elasticsearch.node]  [sonar-1486138560565] starting ...
2017.02.03 17:16:08 INFO   es[o.e.transport]  [sonar-1486138560565] bound_address {inet[/127.0.0.1:9001]}, publish_address {inet[/127.0.0.1:9001]}
2017.02.03 17:16:08 INFO   es[o.e.discovery]  [sonar-1486138560565] sonarqube/owhaNU8GRxmLe8-W_ikm0g
2017.02.03 17:16:11 INFO   es[o.e.cluster.service]  [sonar-1486138560565] new_master [sonar-1486138560565][owhaNU8GRxmLe8-W_ikm0g][sv-arg-nexus-d1][inet[/127.0.0.1:9001]]{rack_id=sonar-1486138560565}, reason: zen-disco-join (elected_as_master)
2017.02.03 17:16:11 INFO   es[o.elasticsearch.node]  [sonar-1486138560565] started
2017.02.03 17:16:11 INFO   es[o.e.gateway]  [sonar-1486138560565] recovered [6] indices into cluster_state

 

Logging ends there. Putting the loglevel on DEBUG doesn't change anything. Nothing changed in the config, I just restarted SonarQube (version 5.6.3).

Our production system is still running and has additional processes compared to our test environment: 

sonar     7525  0.2 14.0 3222252 552128 ?      Sl    2016 250:39 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.management.enabled=false -Djruby.compile.invokedynamic=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/data/sonar/sonarqube-5.6.3/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/server/*:/data/sonar/sonarqube-5.6.3/extensions/jdbc-driver/oracle/ojdbc6.jar org.sonar.server.app.WebServer /data/sonar/sonarqube-5.6.3/temp/sq-process1510845463473826913properties
sonar     7671  0.6 12.7 3089360 502092 ?      Sl    2016 759:59 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/data/sonar/sonarqube-5.6.3/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/server/*:./lib/ce/*:/data/sonar/sonarqube-5.6.3/extensions/jdbc-driver/oracle/ojdbc6.jar org.sonar.ce.app.CeServer /data/sonar/sonarqube-5.6.3/temp/sq-process7558880842775461817properties

Any ideas what might be wrong? For some reason the webserver doesn't start but I see no errors in the logging...

[Sebastien Lesaint]

Hello,

You can't access the web ui because the web process is not started until the ES process is done starting.

It seems the problem is that the ES process is not done starting or that the main process doesn't detect the ES process is done starting.

If it wasn't one of those, you would see the following log line:

app[o.s.p.m.Monitor] Process[es] is up

I see several options for you:

* ensure the ES process is still there

** don't know why it wouldn't be, but just to be sure

* ensure the ES process is idling (eg. check the CPU or memory usage)

* if so, the main process is just not detecting ES startup

** startup detection is based on inter process communication via a shared file, ensure there is no other SQ process that then one you just started accessing it (eg. some orphan SQ process) and that would interfere

** you could try cleaning SQ's temp directory manually (where the shared file is) after stopping all SQ processes

* otherwise, ES is just taking forever to start, so :

** you can wait longer

** or you can drop the existing indexes (delete directory data/es) to force ES re-indexing from scratch at the next SQ startup (warning: could be very time and cpu consuming depending on how much data you have)

Cheers,

Sébastien LESAINT | SonarSource
Platform Developer

http://sonarsource.com

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/fbe77bd1-9079-45f6-8074-fe89fb6f97db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nico Mommaerts]

Hey,

had to give ES more memory and delete all the indices like you said. After that the web thread started. Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/fbe77bd1-9079-45f6-8074-fe89fb6f97db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/ZyxPI0y_bZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAAiZAQxLkgpoEf%3DJfrWTioqv7HmJUrH-_iKcPbRg9n_8cVbvwA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

-- 

Nico Mommaerts
Hansei BVBA
BE 0810.495.871 
Priester Poppestraat 59
2640 Mortsel
GSM: 0498/85.23.59
Email: nico.mo...@hansei.be
LinkedIn: http://www.linkedin.com/in/mommaerts