Re: GitHub webhooks, where to put smee client if Jenkins is in a container?

242 views
Skip to first unread message

Michael Neale

unread,
Oct 24, 2019, 4:01:53 AM10/24/19
to Craig Rodrigues, Jenkins Users
Hi Craig, glad that post is getting mileage!

So in kubernetes, I guess that would be adding to the pod that is running your Jenkins container: there would be a pod definition (not sure if you wrote it) somewhere, and you could cook up an image with smee running and have it as a "sidecar" next to the Jenkins container, as pods share a network and anything running in the pod could access the /github-webhook/ endpoint 


So then it depends how you deployed that image into a pod (but to start with would need to cook up an image with smee in it ready to go - I am not sure if one exists yet). 



On Thu, Oct 24, 2019 at 5:39 PM Craig Rodrigues <rod...@crodrigues.org> wrote:
Michael,

In your blog post:

"Triggering builds with webhooks behind a secure firewall"

You gave a good overview of how someone can use webhooks
invoked from GitHub in the cloud, to a Jenkins server which exists behind
a firewall, using https://smee.io .

In your post, you mention:

"you should install the smee client next to where you have the Jenkins server running:"

In my case, I am running the jenkins/jenkins:lts docker image
( https://hub.docker.com/r/jenkins/jenkins/ ), which is deployed by Kubernetes 1.14.
My Jenkins setup is behind a firewall.
However, my source code exists on GitHub which exists in the public cloud.

Since I do not want to modify the jenkins/jenkins:lts docker image,
where can I run the smee client, so that I can still use it
with my setup?

I'd like to get webhooks from the public GitHub triggering builds on
my Jenkins server running behind a firewall.

Thanks.

--
Craig


Craig Rodrigues

unread,
Oct 24, 2019, 9:18:19 AM10/24/19
to Michael Neale, Jenkins Users

Craig Rodrigues

unread,
Oct 24, 2019, 11:00:45 AM10/24/19
to Michael Neale, Jenkins Users
This seems unnecessarily complicated.
Why does the smee client need to be next to the Jenkins server at all?

If I was not using Kubernetes, and if I had two separate physical machines,
one running Jenkins and one running smee,
would it be possible for smee to interact with Jenkins via the REST API?

--
Craig

Michael Neale

unread,
Oct 24, 2019, 5:40:03 PM10/24/19
to Craig Rodrigues, Jenkins Users
yeah fair point it doesn't have to be right next to it - but it has to be somewhere it can reach /github-webhook endpoint - so could be a totally separate app? (as long as your Jenkins master instance is discoverable and accessible from elsewhere from the cluster - which I guess it would be right?). It only uses the REST api, and only the /github-webhook endpoint, nothing else. 

Craig Rodrigues

unread,
Oct 24, 2019, 8:10:48 PM10/24/19
to Michael Neale, Jenkins Users
Awesome!  So I could run the smee client on a separate physical host, but with network connectivity to my Jenkins server.
Both the smee client and the Jenkins server would be behind the firewall.

To be cool, I could Dockerize the smee client, and deploy that in my Kubernetes cluster.  However, I want to understand
how the pieces fit together before I do that.

One other question, which organization is behind https://smee.io/ ?
If I do a whois lookup, I see: Registrant Organization: GitHub, Inc.

Is this a fully supported service of GitHub, or a side project?

I don't want to try using a service which may disappear.....
--
Craig

Michael Neale

unread,
Oct 24, 2019, 8:44:10 PM10/24/19
to Craig Rodrigues, Jenkins Users
yeah - as long as it can post the webhook, it can run anywhere it needs to. 

It is run by github, but not sure of its officially supported status (they also open source the code to it)

Craig Rodrigues

unread,
Nov 8, 2019, 3:07:44 PM11/8/19
to Michael Neale, Jenkins Users
Michael,

It looks like the smee client has a --target option to specify where to send notifications to:

Usage: smee [options]

Options:
  -v, --version          output the version number
  -u, --url <url>        URL of the webhook proxy service. Default: https://smee.io/new
  -t, --target <target>  Full URL (including protocol and path) of the target service the events will forwarded to. Default: http://127.0.0.1:PORT/PATH
  -p, --port <n>         Local HTTP server port (default: 3000)
  -P, --path <path>      URL path to post proxied requests to` (default: "/")
  -h, --help             output usage information


and set up a smee endpoint, then I started the smee client with:

./node_modules/smee-client/bin/smee.js  -u https://smee.io/mysmeeendpoint -t https://jenkins.example.com/github-webhook
Forwarding https://smee.io/mysmeeendpoint to https://jenkins.example.com/github-webhook
Connected https://smee.io/mysmeeendpoint


I then did a push to my repository in github.  However, the smee client showed an error:

   error: Error: cannot POST /github-webhook (404)
        at Response.toError (/Users/craigrodrigues/smee/
node_modules/superagent/lib/node/response.js:94:15)
        at ResponseBase._
setStatusProperties (/Users/craigrodrigues/smee/node_modules/superagent/lib/response-base.js:123:16)
        at new Response (/Users/craigrodrigues/smee/
node_modules/superagent/lib/node/response.js:41:8)
        at Request._emitResponse (/Users/craigrodrigues/smee/
node_modules/superagent/lib/node/index.js:752:20)
        at IncomingMessage.<anonymous> (/Users/craigrodrigues/smee/
node_modules/superagent/lib/node/index.js:916:38)
        at IncomingMessage.emit (events.js:208:15)
        at endReadableNT (_stream_readable.js:1154:12)
        at processTicksAndRejections (internal/process/task_queues.
js:77:11) {
      status: 404,
      text: '404 page not found\n',
      method: 'POST',
      path: '/github-webhook'
    },

Is there some other setup that is required on the Jenkins server to get this to work?
I have a Jenkins LTS 2.190.2 server, with Blue Ocean, Pipeline, and many other plugins installed.

--
Craig
 

Michael Neale

unread,
Nov 10, 2019, 5:35:10 PM11/10/19
to Craig Rodrigues, Jenkins Users
Hi Craig - I think that you need the trailing slash (as per the blog - I seem to recall it was very fickle about that, probably stapler!)
--
Regards, 

Michael Neale
twitter: @michaelneale, skype: michael_d_neale
Cell: +61 423175597 (Australia)
Cofounder @ CloudBees

Craig Rodrigues

unread,
Nov 11, 2019, 2:43:09 PM11/11/19
to Michael Neale, Jenkins Users
I tried with the trailing slash, and still got a 404 error.
I think your blog post is missing details of how to configure the github-webhook URL.
I think this is configured via this plugin: https://github.com/jenkinsci/github-plugin/

--
Craig

Craig Rodrigues

unread,
Nov 13, 2019, 5:03:00 PM11/13/19
to Michael Neale, Jenkins Users
Michael,

I did the following additional steps not in your blog post using a Jenkins 2.190.2 server:

  1. I configured the smee endpoint, https://smee.io/myendpoint
  2. I ran the smee client, which just sits there, and waits for notifications on https://smee.io/myendpoint , and then forwards these notifications to http://localhost:8080/github-webhook/
     using this script:

    #!/bin/sh
    URL=https://smee.io/myendpoint
    SMEE=/Users/craigrodrigues/smee/node_modules/smee-client/bin/smee.js
    TARGET=http://localhost:8080/github-webook/

    $SMEE --url $URL  --target $TARGET



  3. I configured a webhook on my GitHub repository to connect to https://smee.io/myendpoint
  4. I do a git push to my github repo
  5. GitHub does a POST on https://smee.io/myendpoint
  6. Smee client forwards that post to http://localhost:8080/github-webhook/

On my Jenkins server, I had to configure this:
image.png

After doing a git push, if I look in the logs of the smee client, I see:


     status: 403,
      text: '<html>\n' +
        '<head>\n' +
        '<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n' +
        '<title>Error 403 No valid crumb was included in the request</title>\n' +
        '</head>\n' +
        '<body><h2>HTTP ERROR 403</h2>\n' +
        '<p>Problem accessing /github-webook/. Reason:\n' +
        '<pre>    No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>\n' +
        '\n' +
        '</body>\n' +
        '</html>\n',
      method: 'POST',
      path: '/github-webook/'
    },


So it looks like in order to access http://localhost:8080/github-webhook/ , I need to pass some sort of authentication
in order to access that URL.

How did you configure this in your setup?

--
Craig

Michael Neale

unread,
Nov 13, 2019, 5:07:40 PM11/13/19
to Craig Rodrigues, Jenkins Users
Hrm - I didn't need to do anything like that, no. Unless something has changed since I tried it, I was using a very stock setup with the github plugin etc, and it worked fine if I setup the pipeline as a github source type. 
Reply all
Reply to author
Forward
0 new messages