Jenkins crashes with "Illegal character 0x0 for HttpChannelOverHttp" - Why?

[daniel....@gmail.com]

Hi,

- Ubuntu 16.04.1

- Jenkins 2.32.2

Since a few days I see crashes on Jenkins: (/var/log/jenkins/jenkins.log)

Feb 10, 2017 12:15:09 AM hudson.security.csrf.CrumbFilter doFilter

WARNING: No valid crumb was included in request for /azenv.php. Returning 403.

Feb 10, 2017 2:06:33 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: Illegal character 0x1 in state=START for buffer HeapByteBuffer@6de2eef2[p=1,l=256,c=16384,r=255]={\x01<<<\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01...\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01>>>.com/\r\nContent-Ty...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}

Feb 10, 2017 2:06:33 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: badMessage: 400 Illegal character 0x1 for HttpChannelOverHttp@10d1bddb{r=0,c=false,a=IDLE,uri=}

Feb 10, 2017 4:35:15 AM hudson.security.csrf.CrumbFilter doFilter

WARNING: No valid crumb was included in request for /azenv.php. Returning 403.

Feb 10, 2017 5:30:24 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: Illegal character 0x0 in state=START for buffer HeapByteBuffer@5e57ec7c[p=1,l=1,c=16384,r=0]={\x00<<<>>>ET /login HTTP/1....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}

Feb 10, 2017 5:30:24 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: badMessage: 400 Illegal character 0x0 for HttpChannelOverHttp@1910d1bc{r=0,c=false,a=IDLE,uri=}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: Illegal character 0x4 in state=START for buffer HeapByteBuffer@53c389d4[p=1,l=10,c=16384,r=9]={\x04<<<\x01\x00P\xC0c\xF660\x00>>>/m.sogou.com/?ran...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@3e824b3e{r=0,c=false,a=IDLE,uri=}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: Illegal character 0x4 in state=START for buffer HeapByteBuffer@64380bcf[p=1,l=10,c=16384,r=9]={\x04<<<\x01\x00P\xC0c\xF660\x00>>> HTTP/1.1\r\nUser-A...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@605ff7ac{r=0,c=false,a=IDLE,uri=}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: Illegal character 0x5 in state=START for buffer HeapByteBuffer@53c389d4[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>P\xC0c\xF660\x00/m.sogou.c...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}

Feb 10, 2017 5:30:25 AM org.eclipse.jetty.util.log.JavaUtilLog warn

WARNING: badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@59989f5a{r=0,c=false,a=IDLE,uri=}

Anyone can help or have ideas how to solve this? 

[R. Tyler Croy]

(replies inline)

On Fri, 10 Feb 2017, daniel....@gmail.com wrote:

> Hi,
>
> - Ubuntu 16.04.1
> - Jenkins 2.32.2
>

> Since a few days I see crashes on Jenkins: (/var/log/jenkins/jenkins.log*)*

>
> Feb 10, 2017 12:15:09 AM hudson.security.csrf.CrumbFilter doFilter
> WARNING: No valid crumb was included in request for /azenv.php. Returning
> 403.

For what it's worth, I have been seeing this a bit lately as well. As far as I
can tell (https://bugs.eclipse.org/bugs/show_bug.cgi?id=471081) it's due to
something attempting to speak HTTPs to an HTTP only service.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9aacc222-8133-4ca6-bb75-2555554ba142%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


- R. Tyler Croy

------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>
xmpp: rty...@jabber.org

% gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------

[Daniel Beck]

> On 10. Feb 2017, at 09:50, daniel....@gmail.com wrote:
>
> Since a few days I see crashes on Jenkins: (/var/log/jenkins/jenkins.log)

To clarify, when you write "crash" you mean "logs a warning"? Or does it actually terminate the Jenkins process?

> Feb 10, 2017 4:35:15 AM hudson.security.csrf.CrumbFilter doFilter
> WARNING: No valid crumb was included in request for /azenv.php. Returning 403.

Looks like someone's sending `POST /azenv.php`, an URL that doesn't exist in Jenkins. Probably an automated (exploit?) script unrelated to Jenkins.

[daniel....@gmail.com]

• "Crash" means it terminates the Jenkins process.
  
• We also assume it was a kind of attack. We use additional webserver authentication with Jenkins since then and have not seen the error again.

[Daniel Beck]

> On 25. Aug 2017, at 10:32, daniel....@gmail.com wrote:
>
> • "Crash" means it terminates the Jenkins process.
>

Given that you're using a version of Jenkins with a critical security vulnerability[1][2], which is known to be exploited in a way that terminate the master process, on what is presumably a public network, do you have any information showing that it's these specific requests terminating Jenkins?

Otherwise, I'd just assume the master has been compromised and there's a cryptocurrency miner running, which keeps shutting down your Jenkins process.

1: https://jenkins.io/security/advisory/2017-04-26/#cli-unauthenticated-remote-code-execution
2: https://groups.google.com/d/msg/jenkinsci-advisories/sN9S0x78kMU/8ZqS3ASiAwAJ