Why does my Jenkins server crash when running `wrangler publish`? (error 502)

[Mattias Fjellvang]

So I am trying to set up a Jenkins deploy pipeline. Almost everything seems to be working fine, except for the last part of the job, which is to run `wrangler publish` (to publish the site to Cloudflare Workers).

I have tried running it twice now, and both times, it will fail during the job, and I will receive an error code "502" when I try to access my jenkins server. The only thing that works is a full reboot of the server.

I have tried checking logs, but nothing much shows up. In jenkins.error.log I see this:

    2020/09/23 21:12:00 [error] 1098#1098: *498 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.94.165, server: jenkins.mydomain.com, request: "POST /job/my-project-staging/15/logText/progressiveHtml HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/logText/progressiveHtml", host: "jenkins.mydomain.com", referrer: "https://jenkins.mydomain.com/job/my-project-staging/15/console"

    2020/09/23 21:12:02 [error] 1098#1098: *500 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.93.72, server: jenkins.mydomain.com, request: "HEAD /job/my-project-staging/15/statusIcon HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/statusIcon", host: "jenkins.mydomain.com"

    2020/09/23 21:12:03 [error] 1098#1098: *502 connect() failed (111: Connection refused) while connecting to upstream, client: 172.69.34.207, server: jenkins.mydomain.com, request: "GET /job/my-project-staging/15/statusIcon HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/statusIcon", host: "jenkins.mydomain.com"

    2020/09/23 21:12:03 [error] 1098#1098: *504 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.91.152, server: jenkins.mydomain.com, request: "HEAD /job/my-project-staging/15/console HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/console", host: "jenkins.mydomain.com"

    2020/09/23 21:12:06 [error] 1098#1098: *506 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.91.146, server: jenkins.mydomain.com, request: "GET /job/my-project-staging/15/console HTTP/1.1", upstream: "http://127.0.0.1:8080/job/my-project-staging/15/console", host: "jenkins.mydomain.com"

    2020/09/23 22:15:09 [error] 1098#1098: *1773 connect() failed (111: Connection refused) while connecting to upstream, client: 74.120.14.35, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123:443"

    2020/09/23 22:15:10 [error] 1098#1098: *1775 connect() failed (111: Connection refused) while connecting to upstream, client: 74.120.14.35, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123"

    2020/09/23 23:08:18 [error] 1098#1098: *2378 connect() failed (111: Connection refused) while connecting to upstream, client: 193.118.53.210, server: jenkins.mydomain.com, request: "GET /solr/ HTTP/1.1", upstream: "http://127.0.0.1:8080/solr/", host: "123.123.123.123"

    2020/09/24 01:36:08 [error] 1098#1098: *7943 connect() failed (111: Connection refused) while connecting to upstream, client: 51.158.24.203, server: jenkins.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "123.123.123.123"

Any ideas?

I have tried restarting nginx, that doesn't prevent the 502 error. However, when I run `sudo /etc/init.d/jenkins restart` the jenkins server will reboot and be online again.

When I go into the failed job, I see this at the very end, before the server crashed:

    memory allocation of 240904 bytes failedCreating placeholder flownodes because failed loading originals.

    java.io.IOException: Tried to load head FlowNodes for execution Owner[jwr-nuxt-staging/15:jwr-nuxt-staging #15] but FlowNode was not found in storage for head id:FlowNodeId 1:26

    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:689)

    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:726)

    at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:691)

    at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:550)

    at hudson.model.RunMap.retrieve(RunMap.java:225)

    at hudson.model.RunMap.retrieve(RunMap.java:57)

    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)

    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)

    at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)

    at hudson.model.RunMap.getById(RunMap.java:205)

    at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:940)

    at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:951)

    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)

    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)

    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)

    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)

    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)

    at jenkins.model.Jenkins.<init>(Jenkins.java:1017)

    at hudson.model.Hudson.<init>(Hudson.java:85)

    at hudson.model.Hudson.<init>(Hudson.java:81)

    at hudson.WebAppMain$3.run(WebAppMain.java:282)

    Finished: FAILURE

Thinking this could be a memory related issue I went into `/etc/default/jenkins` and updated `JAVA_ARGS` to `JAVA_ARGS="-Xmx4g -XX:MaxPermSize=512m -Djava.awt.headless=true"` - but that didn't change anything. I have verified I can run `wrangler publish` via terminal on the server, without issues.