[JIRA] (JENKINS-56993) Influx not responsive after error in log

Issue Type:	Bug
Assignee:	Aleksi Simell
Components:	influxdb-plugin
Created:	2019-04-12 15:55
Environment:	kubernetes, POD1: Jenkins, POD2:grafana with influxdb
Priority:	Major
Reporter:	Fred'O

Randomly the plugin bombs out and blocks any further access to the influx db. Influx pod needs to be restarted.

The build log never complains and always shows a successful message:

03:30:45 [InfluxDB Plugin] Collecting data for publication in InfluxDB...03:30:45 [InfluxDB Plugin] Publishing data to: url=[http://localhost:808603:30:55 [InfluxDB Plugin] Completed.

Note above the 10s before the Completed log, this is a sign that the plugin bombs out.

But on the Jenkins Admin logs consols you can see:

Jenkins Admin Logs

Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
Data source empty: Custom Data
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
Data source empty: Custom Data Map
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
Plugin skipped: SonarQube
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
Data source empty: Change Log
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
Plugin skipped: Performance Publisher
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService
[InfluxDB Plugin] Publishing data to: [url=http://localhost:8086, description=jenkins, username=admin, password=*****, database=jenkins]
Apr 12, 2019 3:30:55 PM WARNING jenkinsci.plugins.influxdb.InfluxDbPublicationService writeToInflux
Could not report to InfluxDB. Ignoring Exception. java.net.SocketTimeoutException: timeout at okio.Okio$4.newTimeoutException(Okio.java:232) at okio.AsyncTimeout.exit(AsyncTimeout.java:275) at okio.AsyncTimeout$2.read(AsyncTimeout.java:243) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:144) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:788) Caused: org.influxdb.InfluxDBIOException at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:800) at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:455) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.writeToInflux(InfluxDbPublicationService.java:299) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.perform(InfluxDbPublicationService.java:273) at jenkinsci.plugins.influxdb.InfluxDbPublisher.perform(InfluxDbPublisher.java:357) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Reproducible randomly with a pipeline like:

 
                                                                node {
    try {
        stage('Clone') {
            sh 'dosomething'
        }
    } catch (Exception e) {
        if(currentBuild.result == null) {
            currentBuild.result = "FAILURE"
        }
    } finally {
        step([$class: 'InfluxDbPublisher', jenkinsEnvParameterTag: 'EXEC=' + "dummy", target: 'jenkins'])
    }
}
 
                                                            

Could you suggest a workaround or a way to increase the log level to see where it fails?

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

frederic.odin@sap.com (JIRA)

unread,

Apr 12, 2019, 11:58:02 AM4/12/19

to jenkinsc...@googlegroups.com

Fred'O updated an issue

Jenkins /

Change By:	Fred'O

Randomly the plugin bombs out and blocks any further access to the influx db. Influx pod needs to be restarted.

The build log never complains and always shows a successful message:

{ quote code:java } *
03:30:45 *   [InfluxDB Plugin] Collecting data for publication in InfluxDB... *
03:30:45 *   [InfluxDB Plugin] Publishing data to: [ url=[http://localhost:8086 |http://localhost:8086/], description=jenkins, username=admin, password=*****, database=jenkins]*
03:30:55 *   [InfluxDB Plugin] Completed.
{ quote code }

Note above the 10s before the Completed log, this is a sign that the plugin bombs out.

But on the Jenkins Admin logs consols you can see:

{panel:title=Jenkins Admin Logs}

{panel}

Reproducible randomly with a pipeline like:

{code:java}

node {
    try {
        stage('Clone') {
            sh 'dosomething'
        }
    } catch (Exception e) {
        if(currentBuild.result == null) {
            currentBuild.result = "FAILURE"
        }
    } finally {
        step([$class: 'InfluxDbPublisher', jenkinsEnvParameterTag: 'EXEC=' + "dummy", target: 'jenkins'])
    }
}

{code}

Could you suggest a workaround or a way to increase the log level to see where it fails?

frederic.odin@sap.com (JIRA)

unread,

Apr 12, 2019, 11:59:03 AM4/12/19

to jenkinsc...@googlegroups.com

Fred'O updated an issue

Jenkins /

Change By:	Fred'O

Randomly the plugin bombs out and blocks any further access to the influx db. Influx pod needs to be restarted.

The build log never complains and always shows a successful message:

{code:java}
03:30:45 [InfluxDB Plugin] Collecting data for publication in InfluxDB...
03:30:45 [InfluxDB Plugin] Publishing data to: [ url= [ http://localhost:8086 , description=jenkins, username=admin, password=*****, database=jenkins]

03:30:55 [InfluxDB Plugin] Completed.

{code}

Note above the 10s before the Completed log, this is a sign that the plugin bombs out.

But on the Jenkins Admin logs consols console you can see:
{ panel code : title=Jenkins Admin Logs java }

{ panel code }

Reproducible randomly with a pipeline like:
{code:java}
node {
    try {
        stage('Clone') {
            sh 'dosomething'
        }
    } catch (Exception e) {
        if(currentBuild.result == null) {
            currentBuild.result = "FAILURE"
        }
    } finally {
        step([$class: 'InfluxDbPublisher', jenkinsEnvParameterTag: 'EXEC=' + "dummy", target: 'jenkins'])
    }
}
{code}

Could you suggest a workaround or a way to increase the log level to see where it fails?

frederic.odin@sap.com (JIRA)

unread,

Apr 15, 2019, 3:44:01 AM4/15/19

to jenkinsc...@googlegroups.com

Fred'O updated an issue

Jenkins /

Change By:	Fred'O
Environment:	Version 1.20.4, Jenkins 2.150.3

kubernetes, POD1: Jenkins, POD2:grafana with influxdb

frederic.odin@sap.com (JIRA)

unread,

Apr 15, 2019, 6:59:04 AM4/15/19

to jenkinsc...@googlegroups.com

Fred'O updated an issue

Jenkins /

Change By:	Fred'O

Randomly the plugin bombs out and blocks any further access to the influx db. Influx pod needs to be restarted.

The build log never complains and always shows a successful message:
{code:java}
03:30:45 [InfluxDB Plugin] Collecting data for publication in InfluxDB...

03:30:45 [InfluxDB Plugin] Publishing data to: [url=http://localhost:8086, description=jenkins, username=admin, password=*****, database=jenkins]
03:30:55 [ InfluxDB Plugin Pipeline ] Completed. }
{code [Pipeline] // timestamps
[Pipeline] }
Note above the 10s before the Completed log, this is a sign that the plugin bombs out [Pipeline] // node
[Pipeline] End of Pipeline
java . net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204)
But on the at java.net.SocketInputStream.read(SocketInputStream.java:141)
at okio.Okio$2.read(Okio.java:140)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
Caused: java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:232)
at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)

Caused: jenkinsci.plugins.influxdb.InfluxReportException
at jenkinsci.plugins.influxdb.InfluxDbPublicationService.writeToInflux(InfluxDbPublicationService.java:302)

at jenkinsci.plugins.influxdb.InfluxDbPublicationService.perform(InfluxDbPublicationService.java:273)
at jenkinsci.plugins.influxdb.InfluxDbPublisher.perform(InfluxDbPublisher.java:357)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

{code}
The Jenkins Admin logs console you can see:
{code:java}

{code}

aleksi.simell@eficode.com (JIRA)

unread,

Jun 27, 2019, 3:26:02 AM6/27/19

to jenkinsc...@googlegroups.com

Aleksi Simell commented on

Re: Influx not responsive after error in log

Are you still facing this issue? Unfortunately I've never seen this error before. I've just used Docker on my local stuff without Kubernetes and I believe this is an error related more to Kubernetes connectivity.

It could be possible to add some sort of connection listening to the plugin to ensure we have a connection.

frederic.odin@sap.com (JIRA)

unread,

Jul 1, 2019, 5:23:03 AM7/1/19

to jenkinsc...@googlegroups.com

Fred'O commented on