Post 6.7.1 LTS upgrade warnings in es.log

61 views
Skip to first unread message

Robert Patrick

unread,
Jan 21, 2018, 12:41:31 PM1/21/18
to SonarQube
Hi,

I just upgraded my Sonarqube installation from 6.2 to 6.7.1 LTS by carefully following using the Upgrade documentation.

Now, I am seeing the following warning repeating itself many times in es.log:

2018.01.21 09:37:03 WARN  es[][o.e.t.n.Netty4Transport] exception caught on transport layer [[id: 0x6684861f, L:/127.0.0.1:9001 - R:/127.0.0.1:37456]], closing connection
java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1373) ~[elasticsearch-5.6.3.jar:5.6.3]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-5.6.3.jar:5.6.3]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

What do I need to do to resolve this issue?

Thanks,
Robert

Robert Patrick

unread,
Jan 21, 2018, 1:15:31 PM1/21/18
to SonarQube
This is also having a side-effect of the server's Background Analysis task taking a really long time (e.g., 10 minutes) before it completes.  Once complete, it shows the analysis took only 13 seconds...

Sebastien Lesaint

unread,
Jan 22, 2018, 3:44:27 AM1/22/18
to Robert Patrick, SonarQube
Hello,

The log seem to indicate the Elastic server embedded in SonarQube (lets call it SQ1) has received a message over the network from a ES client (or another server, who knows) in an old unsupported version (2.0.0).

Double check you're not having another instance of an older version of SonarQube somewhere configured to talk to the ES of SQ1.

Cheers,

Sébastien LESAINT | SonarSource
Platform Developer

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/df9bc917-c066-4369-b832-3c445d57850a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Robert Patrick

unread,
Jan 22, 2018, 8:57:31 AM1/22/18
to Sebastien Lesaint, SonarQube
This is definitely not the case.  How can I debut this?

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

Sebastien Lesaint

unread,
Jan 22, 2018, 9:01:23 AM1/22/18
to SonarQube
If this is not it, then I don't know.

Could you share how you configure sonarqube (and Elastic)? Describe your setup?



Sébastien LESAINT | SonarSource
Platform Developer

On 22 January 2018 at 14:57, Robert Patrick <rhpa...@gmail.com> wrote:
This is definitely not the case.  How can I debut this?
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.

Simon Brandhof

unread,
Jan 22, 2018, 9:06:43 AM1/22/18
to Sebastien Lesaint, SonarQube
Or did you unzip version 6.7.1 over a 6.2 installation so that classpath contains both versions of Elasticsearch?

On Mon, 22 Jan 2018 at 15:01 Sebastien Lesaint <sebastie...@sonarsource.com> wrote:
If this is not it, then I don't know.

Could you share how you configure sonarqube (and Elastic)? Describe your setup?



Sébastien LESAINT | SonarSource
Platform Developer

On 22 January 2018 at 14:57, Robert Patrick <rhpa...@gmail.com> wrote:
This is definitely not the case.  How can I debut this?
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAAiZAQzMa4O3wy5AW-4_HyAL5hKzrA3cD6Dpuo4sC-wLGU04tg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com

Robert Patrick

unread,
Jan 22, 2018, 9:08:00 AM1/22/18
to Simon Brandhof, Sebastien Lesaint, SonarQube
Definitely not...will send my configuration whe I get to the office.

You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/j3ErXcxWDvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CA%2BZZQ37_MRQuDo1N-oLLTgYViA05-n4i7qZ3PBBJGrAZnrEYWQ%40mail.gmail.com.

Robert Patrick

unread,
Jan 22, 2018, 12:40:58 PM1/22/18
to Simon Brandhof, Sebastien Lesaint, SonarQube
I changed the sonar.search.port and stack traces in the es.log went away so clearly, something was trying to talk to it on 9001...

Unfortunately, the really slow processing of the scanner results is still happening.  Here is my sonar.properties (with names changed to protect the innocent), I have not changed wrapper.conf from the default:

onar.jdbc.username=sonar
sonar.jdbc.password=mypassword

sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.web.host=0.0.0.0
sonar.web.port=9000

sonar.ce.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true

sonar.search.javaOpts=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.search.port=8123
sonar.search.host=127.0.0.1

http.proxyHost=my-proxy.mycompany.com
http.proxyPort=80
http.nonProxyHosts=127.0.0.1|localhost|*.mycompany.com

This is becoming a huge problem...it looks like it tries to run for 10 minutes and then times out and restarts indefinitely (see attached...note the Submitted time since the Started time is reset every time it times out).  The only reliable way I have found to stop it is to restart the sonarqube server.


On Mon, Jan 22, 2018 at 8:07 AM, Robert Patrick <rhpa...@gmail.com> wrote:
Definitely not...will send my configuration whe I get to the office.
On Mon, Jan 22, 2018 at 8:06 AM Simon Brandhof <simon.brandhof@sonarsource.com> wrote:
Or did you unzip version 6.7.1 over a 6.2 installation so that classpath contains both versions of Elasticsearch?

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/j3ErXcxWDvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.
SonarBackgroundTasks.png

Simon Brandhof

unread,
Jan 22, 2018, 12:50:41 PM1/22/18
to Robert Patrick, Sebastien Lesaint, SonarQube
Some orphan processes of an older SonarQube probably lock the port 9001. They may also interact with database and introduce weird behaviors. For this reason I recommend to check that only the SonarQube 6.7.1 installation connects to database.
I hope it helps.

On Mon, 22 Jan 2018 at 18:40 Robert Patrick <rhpa...@gmail.com> wrote:
I changed the sonar.search.port and stack traces in the es.log went away so clearly, something was trying to talk to it on 9001...

Unfortunately, the really slow processing of the scanner results is still happening.  Here is my sonar.properties (with names changed to protect the innocent), I have not changed wrapper.conf from the default:

onar.jdbc.username=sonar
sonar.jdbc.password=mypassword

sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.web.host=0.0.0.0
sonar.web.port=9000

sonar.ce.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true

sonar.search.javaOpts=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.search.port=8123
sonar.search.host=127.0.0.1

http.proxyHost=my-proxy.mycompany.com
http.proxyPort=80
http.nonProxyHosts=127.0.0.1|localhost|*.mycompany.com

This is becoming a huge problem...it looks like it tries to run for 10 minutes and then times out and restarts indefinitely (see attached...note the Submitted time since the Started time is reset every time it times out).  The only reliable way I have found to stop it is to restart the sonarqube server.

On Mon, Jan 22, 2018 at 8:07 AM, Robert Patrick <rhpa...@gmail.com> wrote:
Definitely not...will send my configuration whe I get to the office.
On Mon, Jan 22, 2018 at 8:06 AM Simon Brandhof <simon.b...@sonarsource.com> wrote:
Or did you unzip version 6.7.1 over a 6.2 installation so that classpath contains both versions of Elasticsearch?

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/j3ErXcxWDvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

Robert Patrick

unread,
Jan 22, 2018, 12:51:19 PM1/22/18
to Simon Brandhof, Sebastien Lesaint, SonarQube
So I let the analysis task run and it finally finished at 11:42.  Unfortunately, the pending task that had been waiting for the previous analysis to finish failed with the following error:

Validation of project failed:
  o Date of analysis cannot be older than the date of the last known analysis on this project. Value: "2018-01-22T09:11:43-0800". Latest analysis: "2018-01-22T09:11:44-0800". It's only possible to rebuild the past in a chronological order.

On Mon, Jan 22, 2018 at 11:40 AM, Robert Patrick <rhpa...@gmail.com> wrote:
I changed the sonar.search.port and stack traces in the es.log went away so clearly, something was trying to talk to it on 9001...

Unfortunately, the really slow processing of the scanner results is still happening.  Here is my sonar.properties (with names changed to protect the innocent), I have not changed wrapper.conf from the default:

onar.jdbc.username=sonar
sonar.jdbc.password=mypassword

sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.web.host=0.0.0.0
sonar.web.port=9000

sonar.ce.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true

sonar.search.javaOpts=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.search.port=8123
sonar.search.host=127.0.0.1

http.proxyHost=my-proxy.mycompany.com
http.proxyPort=80
http.nonProxyHosts=127.0.0.1|localhost|*.mycompany.com

This is becoming a huge problem...it looks like it tries to run for 10 minutes and then times out and restarts indefinitely (see attached...note the Submitted time since the Started time is reset every time it times out).  The only reliable way I have found to stop it is to restart the sonarqube server.

On Mon, Jan 22, 2018 at 8:07 AM, Robert Patrick <rhpa...@gmail.com> wrote:
Definitely not...will send my configuration whe I get to the office.

Robert Patrick

unread,
Jan 22, 2018, 1:47:58 PM1/22/18
to Simon Brandhof, Sebastien Lesaint, SonarQube
That was it!  Thanks and sorry for the noise.

On Mon, Jan 22, 2018 at 11:50 AM, Simon Brandhof <simon.b...@sonarsource.com> wrote:
Some orphan processes of an older SonarQube probably lock the port 9001. They may also interact with database and introduce weird behaviors. For this reason I recommend to check that only the SonarQube 6.7.1 installation connects to database.
I hope it helps.
On Mon, 22 Jan 2018 at 18:40 Robert Patrick <rhpa...@gmail.com> wrote:
I changed the sonar.search.port and stack traces in the es.log went away so clearly, something was trying to talk to it on 9001...

Unfortunately, the really slow processing of the scanner results is still happening.  Here is my sonar.properties (with names changed to protect the innocent), I have not changed wrapper.conf from the default:

onar.jdbc.username=sonar
sonar.jdbc.password=mypassword

sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.web.host=0.0.0.0
sonar.web.port=9000

sonar.ce.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true

sonar.search.javaOpts=-Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.search.port=8123
sonar.search.host=127.0.0.1

http.proxyHost=my-proxy.mycompany.com
http.proxyPort=80
http.nonProxyHosts=127.0.0.1|localhost|*.mycompany.com

This is becoming a huge problem...it looks like it tries to run for 10 minutes and then times out and restarts indefinitely (see attached...note the Submitted time since the Started time is reset every time it times out).  The only reliable way I have found to stop it is to restart the sonarqube server.

On Mon, Jan 22, 2018 at 8:07 AM, Robert Patrick <rhpa...@gmail.com> wrote:
Definitely not...will send my configuration whe I get to the office.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/j3ErXcxWDvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.

Simon Brandhof

unread,
Jan 22, 2018, 2:10:00 PM1/22/18
to Robert Patrick, Sebastien Lesaint, SonarQube
Good news. Thanks Robert for the follow-up.

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/j3ErXcxWDvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.


--

Simon Brandhof | SonarSource

Co-Founder & Tech Lead

@SimonBrandhof

http://sonarsource.com
Reply all
Reply to author
Forward
0 new messages