RFC - GoCD - Github PR Integration

[Ashwanth Kumar]

Hey,

There have been couple of posts[1][2] on Github PR integration in GoCD. I came up with a workflow and would love to hear what you guys think about it. 

Topics on go-cd

[1] - https://groups.google.com/forum/#!searchin/go-cd/pull$20request/go-cd/i0ZXbBL48Vk/qCjboHjslYcJ

[2] - https://groups.google.com/forum/#!searchin/go-cd/pull$20request/go-cd/E1mKTHalRKU/G5Nh6_fIP10J

[3] - https://github.com/ashwanthkumar/gocd-github-pull-request/blob/master/README.md

--

Ashwanth Kumar / ashwanthkumar.in

[Derek Hammer]

Nice!

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken Mugrage]

Yes, very nice!

[Alexandre Conrad]

I think there should also be a way to trigger builds using a Github webhooks as well.

Also, technically you don't need to do the merging part "manually" since Github already does that for you under a "merge" branch.

+refs/pull/<id>/merge:refs/remotes/origin/pr/<id>/merge

This branch will not exist if the pull request has a merge conflict though. I believe this is how Github determines whether the "Merge pull request" is enabled or disabled (greyed out). But the /head will always be there, which is essentially represent the state of your PR at its latest commit.

Also, because Go builds pipelines and not just a single job, I think it's critical that the build status can be customized and updated per job to leverage Github's combined commit status API.

The green checkbox is called the "state" (which can be red is case of build failure), the green bold text next to it is called the "context" (which could be the name of the pipeline/stage/job), the text in grey is called the "description" and the link Details is called the "target_url" which is essentially a link to the build log of the job or to a generated test artifact.

If each of these fields could be customized using environment variables, that would make it super flexible so each job can have a different message (we currently do this to show a quick summary of why a build failed or succeeded, as you can see in the screenshot).

Also, a single job should be able to submit multiple commit statuses, for example, it's common to have the same job running the tests and measure coverage at once, then it should generate 2 commit statuses. The first test status could have its description showing how many tests passed / failed and have its target_url point to an HTML junit coverage report -- and the other one, the coverage status, would show the number of uncovered lines and a link to the coverage report.

If we're going to build Github PR integration for GoCD, I think this is a must-have feature... a half-baked PR integration would only be frustrating.

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex

[srinivas upadhya]

There is a LOT of interest in Pull Request / Feature Branch support. Here are some more threads discussing that: 1, 

​2 & 3​

​AFAIK

​

 someone on hackathon organized by Kurt was working on similar lines.

​ Would be great if he could pitch in.​

I & Aravind were discussing this approach once. The only downside we thought was that currently Go will batch multiple commits into a pipeline instance (to save time) which might not be what you would want. Ideally you would want a separate build for eac

​h​

PR

​​

.​ Aravind has been thinking of providing some way to force Go to schedule a separate build for each change.

​​

Finally, I am currently working on SCM extension point. Backend is done, working on UI bits. (dev docs & sample plugin). Hopefully 

​will be available in next release.​

 See if this is a better fit than Package Repository end-point

​ for the polling bit. Also, the build status update could be a notification plugin rather than task plugin?

[Aravind SV]

On Mon, Feb 2, 2015 at 9:20 AM, Ashwanth Kumar <ashwan...@googlemail.com> wrote:

Hey,

There have been couple of posts[1][2] on Github PR integration in GoCD. I came up with a workflow and would love to hear what you guys think about it. 

Nice! I like it.

You might not need the build task, actually. The SCM endpoint was accepted just a couple of days ago. Its documentation is in progress. What it provides is a "Checkout" message, which is used to tell the agent and server to checkout, using the plugin, to a specific revision of the material. I feel this removes the need for a task, if you want. Assuming, we can somehow differentiate between agent-side plugin and server-side plugin usage.

When Srinivas and I talked about GitHub a couple of days ago, and an approach like this, the small bump we hit was that multiple PRs, seen by this plugin, will be bunched together and run. That's the wrong behavior, isn't it? This needs a bit of a change in Go's scheduling, right?

I mean:

PR 1 is created or changed

Go starts building it

PR 2 is created or changed

PR 3 is created or changed

Go finishes building PR1

Next build scheduled will be with both PR 2 and PR 3.

This is because, in Go, multiple "commits" which show up while building can be bunched together. This is true of SCM commits, but is not true for PRs, since PR2 and PR3 are not necessarily related. PRs are not really commit, but whole branches, which is why we run up against this.

[srinivas upadhya]

Ok Aravind's reply is saying the same thing as mine, only his is better, so you can ignore mine ;) Wish there was a merge conflict feature for emails :P

[Aravind SV]

On Mon, Feb 2, 2015 at 1:50 PM, srinivas upadhya <srinivas...@gmail.com> wrote:

Ok Aravind's reply is saying the same thing as mine, only his is better, so you can ignore mine ;) Wish there was a merge conflict feature for emails :P

That's Google Buzz/Wave territory. I'm still trying to purge my mind of the bad memories from that time. Don't bring it back.

[Alexandre Conrad]

PRs (pull requests) are somewhat a black sheep when it comes to CD pipelines since most developers think of them as working drafts and not meant to be deployed anywhere until they are merged to master (although nothing should prevent one from doing this). Basically, what runs as part of a PR should only be a subset of the whole pipeline, the CI part of it.

Also, a PR might change the code in such was that the pipeline no longer "fits". For example, say that I have a PR that adds support for Python 3, well, I would like the pipeline to run Python 3 tests. But then if I add a Python 3 job to my pipeline, now my other PRs will break because they don't have Python 3 support yet. It's a chicken and egg problem.

This is also why I believe that the CI part of the pipelines should be defined in the code repository project itself. Travis CI did this beautifully by having on-demand, throw-away pipelines (yes, they are not really "pipelines" per se). The repo is cloned, the "travis.yml" file is found and created accordingly then the code it pushed through it. Once done, the pipeline is destroyed (although the visualization of the pipeline is archived for future reference), artifacts are saved and that's where the CD cycle ends for a PR, usually. But if you wanted to deploy the PR for some reason (to another testing environment) then the build artifact of the PR is used as the material to the next pipeline.

When the PR gets merged to master eventually, this is when the full CD pipeline should run so it eventually ends up to production if all tests passed.

--

You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex

[Ashwanth Kumar]

Thanks for the inputs, I didn't know about the Github's status APIs. It should be possible to submit multiple status messages just like how we do multiple artifacts out of a job. 

On Tue, Feb 3, 2015 at 12:58 AM, Alexandre Conrad <alexandr...@gmail.com> wrote:

PRs (pull requests) are somewhat a black sheep when it comes to CD pipelines since most developers think of them as working drafts and not meant to be deployed anywhere until they are merged to master (although nothing should prevent one from doing this). Basically, what runs as part of a PR should only be a subset of the whole pipeline, the CI part of it.

Also, a PR might change the code in such was that the pipeline no longer "fits". For example, say that I have a PR that adds support for Python 3, well, I would like the pipeline to run Python 3 tests. But then if I add a Python 3 job to my pipeline, now my other PRs will break because they don't have Python 3 support yet. It's a chicken and egg problem.

This is also why I believe that the CI part of the pipelines should be defined in the code repository project itself. Travis CI did this beautifully by having on-demand, throw-away pipelines (yes, they are not really "pipelines" per se). The repo is cloned, the "travis.yml" file is found and created accordingly then the code it pushed through it. Once done, the pipeline is destroyed (although the visualization of the pipeline is archived for future reference), artifacts are saved and that's where the CD cycle ends for a PR, usually. But if you wanted to deploy the PR for some reason (to another testing environment) then the build artifact of the PR is used as the material to the next pipeline.

This is the reason why I wanted a gocd-buid.conf in the repository. May be we could provide the ability to specify the commit status message against each script command that gets executed. Even if not part of this github integration, that would be a good task plugin on its own in the GoCD's CI world. Thoughts? 

 

When the PR gets merged to master eventually, this is when the full CD pipeline should run so it eventually ends up to production if all tests passed.

2015-02-02 10:47 GMT-08:00 Aravind SV <arv...@thoughtworks.com>:

On Mon, Feb 2, 2015 at 9:20 AM, Ashwanth Kumar <ashwan...@googlemail.com> wrote:

Hey,

There have been couple of posts[1][2] on Github PR integration in GoCD. I came up with a workflow and would love to hear what you guys think about it. 

Nice! I like it.

You might not need the build task, actually. The SCM endpoint was accepted just a couple of days ago. Its documentation is in progress. What it provides is a "Checkout" message, which is used to tell the agent and server to checkout, using the plugin, to a specific revision of the material. I feel this removes the need for a task, if you want. Assuming, we can somehow differentiate between agent-side plugin and server-side plugin usage.

When Srinivas and I talked about GitHub a couple of days ago, and an approach like this, the small bump we hit was that multiple PRs, seen by this plugin, will be bunched together and run. That's the wrong behavior, isn't it? This needs a bit of a change in Go's scheduling, right?

I mean:

PR 1 is created or changed

Go starts building it

PR 2 is created or changed

PR 3 is created or changed

Go finishes building PR1

Next build scheduled will be with both PR 2 and PR 3.

This is because, in Go, multiple "commits" which show up while building can be bunched together. This is true of SCM commits, but is not true for PRs, since PR2 and PR3 are not necessarily related. PRs are not really commit, but whole branches, which is why we run up against this.

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex

[Ashwanth Kumar]

Thanks for pointing me to SCP Endpoint. I read the documentation and was wondering, if there's a way to return a batch of latest revisions may be then we could trigger the pipeline (multiple times) with the corresponding checkout revision. In this case, it would one per PR. I guess this is what you mean by changes in Go's scheduling. 

Or like the way Srinivas put in the #298 if we can create pipeline clones on the fly that would also do the trick.

[srinivas upadhya]

On Tuesday, February 3, 2015 at 7:51:20 AM UTC+5:30, Ashwanth Kumar wrote:

Thanks for pointing me to SCP Endpoint. I read the documentation and was wondering, if there's a way to return a batch of latest revisions may be then we could trigger the pipeline (multiple times) with the corresponding checkout revision. In this case, it would one per PR. I guess this is what you mean by changes in Go's scheduling. 

Or like the way Srinivas put in the #298 if we can create pipeline clones on the fly that would also do the trick.

This was what i was trying to come up with. There are a lot of dependent features that need to be completed before this approach can be implemented. 

[Ashwanth Kumar]

Using the SCM Endpoint, I've built a plugin that can build PRs for a Github project on a single pipeline. Workflow is to create a clone of the existing build pipeline for your repository and change the Git material to Github material. Do check it out and share feedback. 

Ref - https://github.com/ashwanthkumar/gocd-build-github-pull-requests

[Alexandre Conrad]

Great stuff! Thanks for putting this together. I currently don't have a running instance of Go and could go through the code and figure things out but I'm not a Java expert and I might as well just ask you questions here.

- So I see you clone ’’/head", but do you end up merging it with the target branch? The merged version is what I'm really interested in since this is the whole point of testing a pull request: to make sure that once merged, everything is still going to work. Testing just the head of the branch doesn't provide much insight on whether the tests will properly run. You can also use the "/merge" branch which is only available when there are no merge conflicts (see my previous message in this thread).

- Do you inject useful environment variables of the PR payload? such as PR_ID=3; PR_BRANCH="myfeature"; TARGET_BRANCH="master"; PR_URL="https://...", etc. (the target branch may only be available via the Github hook payload). But these environment variables would be very useful for scripts in the pipeline to take extra actions based on the information from the PR.

- I see you have github.com hardcoded, but my company uses Github Enterprise only accessible from within our VPN. It would be nice if it could leverage any Github instances, not just the public github.

- Does it support Github webhooks or polling only? (I know Github has a builtin GoCD webhook, should we leverage this?)

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex

[Ashwanth Kumar]

See my answers inline

On Sat, Feb 21, 2015 at 4:16 AM, Alexandre Conrad <alexandr...@gmail.com> wrote:

- So I see you clone ’’/head", but do you end up merging it with the target branch? The merged version is what I'm really interested in since this is the whole point of testing a pull request: to make sure that once merged, everything is still going to work. Testing just the head of the branch doesn't provide much insight on whether the tests will properly run. You can also use the "/merge" branch which is only available when there are no merge conflicts (see my previous message in this thread).

I currently only checkout the revision that's the latest on the PR. I wasn't sure on how to handle the cases when /merge branch isn't available? What do you think is the right way to do it? 

 

- Do you inject useful environment variables of the PR payload? such as PR_ID=3; PR_BRANCH="myfeature"; TARGET_BRANCH="master"; PR_URL="https://...", etc. (the target branch may only be available via the Github hook payload). But these environment variables would be very useful for scripts in the pipeline to take extra actions based on the information from the PR.

I've only the PR_ID, I'll add the PR_BRANCH, TARGET_BRANCH and URL. Anything else to add? 

 

- I see you have github.com hardcoded, but my company uses Github Enterprise only accessible from within our VPN. It would be nice if it could leverage any Github instances, not just the public github.

Ah, thanks for pointing out. I'll add that to the list. 

 

- Does it support Github webhooks or polling only? (I know Github has a builtin GoCD webhook, should we leverage this?)

Currently it polls periodically (runs as part of Go's MDU). I don't have much insights on how Github's webhook work. Also, I'm not sure if that can be handled in the SCM endpoint. 

[Alexandre Conrad]

Hi Ashwanth,

2015-02-20 15:12 GMT-08:00 Ashwanth Kumar <ashwan...@googlemail.com>:

On Sat, Feb 21, 2015 at 4:16 AM, Alexandre Conrad <alexandr...@gmail.com> wrote:

- So I see you clone ’’/head", but do you end up merging it with the target branch? The merged version is what I'm really interested in since this is the whole point of testing a pull request: to make sure that once merged, everything is still going to work. Testing just the head of the branch doesn't provide much insight on whether the tests will properly run. You can also use the "/merge" branch which is only available when there are no merge conflicts (see my previous message in this thread).

I currently only checkout the revision that's the latest on the PR. I wasn't sure on how to handle the cases when /merge branch isn't available? What do you think is the right way to do it? 

Well, if the PR doesn't merge, then I wouldn't even bother burning resources to test something that doesn't even merge cleanly. So I would just not run the pipeline at all -- the user should expect the pipeline to run only if the code is in a mergeable state. Maybe you could provide better feedback once you support the commit status API.

Actually, the more I think about it, the more I think that the commit status API should be a whole separate plugin. As the API name indicates, the Commit Status API, is not meant to show the build status of a PR but to show the build status of a *commit*! Github leverages the status of the latest commit in a PR to show merge box as green or red (or grey), but technically the commit status is simply not directly tied to a PR per se -- if you push code directly to master without going through a PR, you can still trigger a build and report the build status on the commit SHA without going through a PR.

If you make it a separate plugin, it would allow pipelines, stages and jobs to report progress (pending, cloned, job X started, job X completed, stage Y failed, ...) on individual commits by leveraging Github's commit status API even if the build wasn't triggered by a PR. This would provide ultimate flexibility and I'm telling you: no other tools/plugins out there get it right.

- Do you inject useful environment variables of the PR payload? such as PR_ID=3; PR_BRANCH="myfeature"; TARGET_BRANCH="master"; PR_URL="https://...", etc. (the target branch may only be available via the Github hook payload). But these environment variables would be very useful for scripts in the pipeline to take extra actions based on the information from the PR.

I've only the PR_ID, I'll add the PR_BRANCH, TARGET_BRANCH and URL. Anything else to add?

Here is a screenshot of what is provided by the Github Pull Request Builder (ghprb) plugin of Jenkins. Similar information would be useful:

 

- Does it support Github webhooks or polling only? (I know Github has a builtin GoCD webhook, should we leverage this?)

Currently it polls periodically (runs as part of Go's MDU). I don't have much insights on how Github's webhook work. Also, I'm not sure if that can be handled in the SCM endpoint.

If you go to the Github settings of your repo, under webhooks, for each of the configured hooks, you can see delivery of the hook to the configured endpoint. It contains all the information about each payload sent out (which on your side you can leverage). Here's screenshot of a payload sent from one of my Github projects to Snap-ci:

--

Alex

[Ashwanth Kumar]

Sounds perfect. I was reading up on how travis does this and even they trigger a build only if the PR is mergeable. I'll make the changes for these. 

Regarding a separate plugin for Github commit status was actually the plan, but I didn't know how to get the status of the build in the stage via a task plugin. I could move to notification based approach, I don't have enough information on payload to the plugin to make a proper decision. I am waiting for what Srinivas suggested here. Once they add support for querying materials via plugins then it would be straight forward to do it. 

On the params to expose,  we should be having all these fields, I can expose them as ENV variables. 

Thanks for the note on the webhooks, let me check how the GoCDs webhooks work and a possibility of integration with it. 

[Ashwanth Kumar]

One quick update. I've integrated some of the feedback from Alexandre into the new release (v0.0.2). 

Features

1. Now supports repos from Github Enterprise - need volunteers to test this ;)
2. We now build merge between the PR and the destination instead of HEAD of the PR.
   
3. Exposing more attributes via environment variables during the build.
4. Reduced the number of Github API calls from 3 per MDU check to 2 per every build

[Pasha Sadikov]

Hello,

I have an interest in developing a pull request pipeline trigger but for BitBucket pull requests. I am currently trying to decide whether to go with an SCM plugin approach or to build an external tool that calls on the Go APIs to trigger, monitor and cancel pipeline builds. With the latter approach I would need to write a stateful application and custom logic for polling the feeds API.

Thank you Ashwanth for making great progress on the Github PR builder. I was looking through your code and I couldn't figure out how the plugin would deal with multiple PRs. Does it kick of simultaneous pipelines for each PR's branch (in a situation where two PRs are created at the same time before Go was able to poll)? Does it kick them off sequentially?

Is it even possible to have a single SCM plugin trigger multiple simultaneous pipeline builds (I'm assuming the SCM is not polled when the pipeline is running)?

Regards,

Pasha

[Ashwanth Kumar]

So the SCM's latestRevision and latestRevisionSince are called as part of the Go's MDU process. The plugin takes 1 new PR at a time and schedules it. To answer your question - we schedule multiple PRs sequentially. 

Ref - This line checks for any new PRs from the given current state of PRs (activePullRequests).

- If your build Pipeline has only 1 stage you can't run them in parallel, Go's design is that no 2 versions of the pipeline#stage combination runs in parallel. 

- One SCM Plugin can be used in multiple Pipelines, so my guess is MDU check is independent of whether a build is happening or not. If the plugin should return valid revisions on latestRevision or latestRevisionSince messages, my understanding is they would be scheduled automatically by Go. 

GoCD's core team guys can much provide better insight into these questions.

[Ashwanth Kumar]

Pasha, I was thinking if its not too much to ask. With little refactoring we can support the Bitbucket PRs (for git repos) also in same plugin. Let me know if you're interested on doing something like that.

[srinivas upadhya]

I would suggest moving the "core git" functionality out into a common module which everyone (github, bitbucket etc.)

I am working on that module. It will support both JGit & git command-line client hopefully.

What do you guys think?

@Ashwanth - I have written post-commit hook support for scm end-point. So you can turn off polling & add web hook on GitHub when a PR is added/updated/closed. The UI bit is ready to be merged. Will be in soon.

[Ashwanth Kumar]

On Fri, Feb 27, 2015 at 11:28 AM, srinivas upadhya <srinivas...@gmail.com> wrote:

I would suggest moving the "core git" functionality out into a common module which everyone (github, bitbucket etc.)

I am working on that module. It will support both JGit & git command-line client hopefully.

What do you guys think?

By that you mean, one SCM Plugin that supports regular Git and PR (from hosted services). Correct? If yes, I'm up for it. 

 

@Ashwanth - I have written post-commit hook support for scm end-point. So you can turn off polling & add web hook on GitHub when a PR is added/updated/closed. The UI bit is ready to be merged. Will be in soon.

Sweet, I'll make the changes accordingly. 

Quick question - When is the Release 15.1 happening? 

[srinivas upadhya]

On Friday, February 27, 2015 at 12:55:21 PM UTC+5:30, Ashwanth Kumar wrote:

On Fri, Feb 27, 2015 at 11:28 AM, srinivas upadhya <srinivas...@gmail.com> wrote:

I would suggest moving the "core git" functionality out into a common module which everyone (github, bitbucket etc.)

I am working on that module. It will support both JGit & git command-line client hopefully.

What do you guys think?

By that you mean, one SCM Plugin that supports regular Git and PR (from hosted services). Correct? If yes, I'm up for it. 

Im not sure of one plugin supporting everything. The code might become very convoluted. I was thinking of moving out code to clone, fetch updates, read commits etc. part could be a separate module that other "git" plugins could use.

 

@Ashwanth - I have written post-commit hook support for scm end-point. So you can turn off polling & add web hook on GitHub when a PR is added/updated/closed. The UI bit is ready to be merged. Will be in soon.

Sweet, I'll make the changes accordingly. 

The PR needs to be accepted :) 

Quick question - When is the Release 15.1 happening? 

 

When most of the issues for 15.1 is resolved. Its slated April 1st week for now. Lets hope we make it :)

[Pavel Sadikov]

Thanks for the feedback.

Ashwanth, the issue I see is that BitBucket does not create refs for pull requests the way GitHub does. For example, the alembic package on BitBucket has one outstanding pull request right now, and these are the remote refs that I see - there are no 'merge' refs. For a GitHub repo like this (which has 16 pull requests outstanding), these are the refs that I see.

The only way to get information about BitBucket pull requests is through their API. As such, I agree with Srinivas: I don't believe both providers should be merged into one plugin.

Pasha

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd-dev/Rt_Y5G2VkOc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd-dev+...@googlegroups.com.

[Alexandre Conrad]

+1 on NOT merging Github PRs and Bitbucket PRs into the same plugin. Even though the 2 workflows look the same they will only diverge over time since they these workflows are driven by two different companies and maintaining these divergences will cause headaches -- so we might as well just has separate plugins where one workflow doesn't affect the other.

--

Alex

[srinivas upadhya]

@Ashwanth & @Pasha - I have created the common module for all git plugins that I have talking about previously.

Note: Its definitely WIP. Im trying to improve test coverage. Also, there are some corner cases to be handled, Im working on it.

There are 2 implementations, one that relies on git to be installed on the box & run git commands to get things done while the other uses packaged jgit to do the same. There is a factory method that will give you an implementation based on wether git is installed or not (installed git is preferred over jgit). Let me know what you guys think about it. Try & use the module for your plugins (GitHub & BitBucket) and add git related code into this module. sample usage.

[srinivas upadhya]

On Friday, February 27, 2015 at 7:51:10 AM UTC+5:30, Ashwanth Kumar wrote:

So the SCM's latestRevision and latestRevisionSince are called as part of the Go's MDU process. The plugin takes 1 new PR at a time and schedules it. To answer your question - we schedule multiple PRs sequentially. 

Ref - This line checks for any new PRs from the given current state of PRs (activePullRequests).

- If your build Pipeline has only 1 stage you can't run them in parallel, Go's design is that no 2 versions of the pipeline#stage combination runs in parallel. 

- One SCM Plugin can be used in multiple Pipelines, so my guess is MDU check is independent of whether a build is happening or not. If the plugin should return valid revisions on latestRevision or latestRevisionSince messages, my understanding is they would be scheduled automatically by Go. 

GoCD's core team guys can much provide better insight into these questions.

Yes Ashwanth. That is correct. That is a short coming of this approach right now. Aravind is trying to add a feature that allow user to force one pipeline instance per change (i.e. not merging multiple changes into one pipeline instance that is the current design to save build time). The other major short coming being, if you make the "main" pipeline build PRs then the downstream pipelines will get triggered off of PR code & might accidentally get deployed. So the user of the plugin needs to make sure the VSM using these plugins NEVER do any deployments/anything that is not for production. The ideal approach for this (feature branch) would be to spin up new pipelines/VSMs for each branch/PR and delete them once done. But that will take considerable effort + might not be very usable (lot of pipelines showing up on your dashboard etc.). Still trying to think how we can make this explicit.

[srinivas upadhya]

@Ashwanth & @Pasha - The UI for add & edit SCM is in. Do try out & provide any feedback you have. It only contains 'add new global SCM' & 'edit global SCM'. 'select existing global SCM' needs to be finished yet. You should be able to test 'scm-view', 'validate-scm-configuration', 'check-scm-connection' from these views.

Also, it would be great if you guys use git-cmd module for your projects. Happy to add you guys as collaborators for git-cmd if you think thats required.

The post commit hook PR is yet to be merged.

[Jens Melgaard]

Great effort...

Has anyone by any chance looked into integrating stash into a similar plugin (based of @srinivas upadhya's git-cmd module maybe?)...

[srinivas upadhya]

On Wed, Mar 11, 2015 at 7:59 PM, Jens Melgaard <dotje...@gmail.com> wrote:

Great effort...

Has anyone by any chance looked into integrating stash into a similar plugin (based of @srinivas upadhya's git-cmd module maybe?)...

​Do you mean Stash PR support? Does the API for Stash & BitBucket match? If it does then may be @Pasha's BitBucket plugin could be used for Stash as well?

--

You received this message because you are subscribed to the Google Groups "go-cd-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.

[Jens Melgaard]

Yes, that would be rather nice. But as far as I can see the API's don't match up, not completely at least... (as far as I have been able to grog)...

[srinivas upadhya]

AFAIK apart from "git feature branch" there is scope to build plugins for 

• Github PR support - picked by Ashwanth
• BitBucket PR support - picked by Pasha
• Stash PR support
• Gerrit review support
• GitLab PR support
  

If would be great if someone working with these tools picked up implementation for these.

[srinivas upadhya]

Ashwanth & I refactored the Github plugin to use git-cmd.

Now, the code for feature branch support & Github pull request support looks very similar.

The refspec for fetch is:

Feature branch: "+refs/heads/*:refs/remotes/origin/*";

Github PR: "+refs/pull/*/merge:refs/gh-merge/remotes/origin/*";

Stash PR: "+refs/pull-requests/*/from:refs/remotes/origin/pull-request/*";

The branch selection pattern is:
Feature branch: "refs/remotes/origin/"

Github PR: "refs/gh-merge/remotes/origin/"

Stash PR: "refs/remotes/origin/pull-request/"

Looks like we can actually merge the 3 plugins cleanly with strategy pattern. We will also need to take care of populating correct data in databag like here. Not sure if it should be 1 plugin supporting all 3 or 3 plugins but one codebase (in which case we will need to make the build toggleable).

[srinivas upadhya]

Also Branch-to-revision map / PR-to-revision map is material data (material state) & not revision data. The problems it cause is - 1) it becomes an env. var to tasks which is not needed in this case, 2) since its store for every revision its duplicate data. Im planning to provide the ability for the plugin to provide material state data in get-latest-revision & get-latest-revisions-since call which will be available in the next call. The data will be persisted at material level (table) so the above mentioned problems dont occur.

[srinivas upadhya]

I added support for Stash Pull Request & Gerrit Change Set build.

Feature Branch RefSpec: "+refs/heads/*:refs/remotes/origin/*"

GitHub RefSpec: "+refs/pull/*/head:refs/remotes/origin/pull-request/*"

Stash RefSpec: "+refs/pull-requests/*/from:refs/remotes/origin/pull-request/*"

Gerrit RefSpec: "refs/changes/*:refs/changes/*"

Could not figure this for BitBucket & GitLab.

[Kerr Marin Miller]

This is great, but the documentation for the plugins is lacking a bit of clarity. Can we have an update to the docs? I've spent the past few hours trying to integrate this plugin with Stash, and then with Github, but I haven't had any luck so far. Would be great to get a set of clear steps to follow to get this working.

Great work so far!

[srinivas upadhya]

Taking this up here - https://twitter.com/kerrmarin/status/598513742792564736, If someone else is interested.

[Tony Wei]

Hi,

I used its version 1.2.3 (ref: https://github.com/ashwanthkumar/gocd-build-github-pull-requests/releases) and ran into 2 critical issues:

1. It seems only to allow one GitHub repository defined in a GoCD system. A warning message tells "This is a global copy. All pipelines using this SCM will be affected". So when I set up another pipeline to test PRs from the different GitHub repository, the new materials setting for GitHub repository URL will change the first pipeline's materials GitHub repository URL.
   
2. When I have 2 PR generated at the same time, the GoCD picks up the code with the first PR but the code change information shows about 2 PRs, and the second PR will never trigger the GoCD pipeline after the first PR build is done.

Any comments and suggestions? By the way, the same post is also posted athttps://groups.google.com/forum/#!topic/go-cd/RDL8qwO6ygs.

Thanks very much!

[Milindu Sanoj Kumarage]

Hey, I tried using this plugin with GitHub Enterprise.

 I tried setting the endpoint in ~/.github as endpoint=https://<my-enterprice-dns>/api/v3 but the plugin still checks the repo in api.github.com

By ~ I assumed this means the home of go the user, which the GoCD Server runs. For me, it is /var/gothus the file path is at /var/go/.github

On Sunday, February 22, 2015 at 12:04:30 PM UTC+5:30, Ashwanth Kumar wrote:

One quick update. I've integrated some of the feedback from Alexandre into the new release (v0.0.2). 

Features

1. Now supports repos from Github Enterprise - need volunteers to test this ;)
2. We now build merge between the PR and the destination instead of HEAD of the PR.
   
3. Exposing more attributes via environment variables during the build.
4. Reduced the number of Github API calls from 3 per MDU check to 2 per every build

On Sat, Feb 21, 2015 at 5:55 AM, Alexandre Conrad <alexand...@gmail.com> wrote:

Hi Ashwanth,

2015-02-20 15:12 GMT-08:00 Ashwanth Kumar <ashwan...@googlemail.com>:

On Sat, Feb 21, 2015 at 4:16 AM, Alexandre Conrad <alexand...@gmail.com> wrote:

- So I see you clone ’’/head", but do you end up merging it with the target branch? The merged version is what I'm really interested in since this is the whole point of testing a pull request: to make sure that once merged, everything is still going to work. Testing just the head of the branch doesn't provide much insight on whether the tests will properly run. You can also use the "/merge" branch which is only available when there are no merge conflicts (see my previous message in this thread).

I currently only checkout the revision that's the latest on the PR. I wasn't sure on how to handle the cases when /merge branch isn't available? What do you think is the right way to do it? 

Well, if the PR doesn't merge, then I wouldn't even bother burning resources to test something that doesn't even merge cleanly. So I would just not run the pipeline at all -- the user should expect the pipeline to run only if the code is in a mergeable state. Maybe you could provide better feedback once you support the commit status API.

Actually, the more I think about it, the more I think that the commit status API should be a whole separate plugin. As the API name indicates, the Commit Status API, is not meant to show the build status of a PR but to show the build status of a *commit*! Github leverages the status of the latest commit in a PR to show merge box as green or red (or grey), but technically the commit status is simply not directly tied to a PR per se -- if you push code directly to master without going through a PR, you can still trigger a build and report the build status on the commit SHA without going through a PR.

If you make it a separate plugin, it would allow pipelines, stages and jobs to report progress (pending, cloned, job X started, job X completed, stage Y failed, ...) on individual commits by leveraging Github's commit status API even if the build wasn't triggered by a PR. This would provide ultimate flexibility and I'm telling you: no other tools/plugins out there get it right.

- Do you inject useful environment variables of the PR payload? such as PR_ID=3; PR_BRANCH="myfeature"; TARGET_BRANCH="master"; PR_URL="https://...", etc. (the target branch may only be available via the Github hook payload). But these environment variables would be very useful for scripts in the pipeline to take extra actions based on the information from the PR.

I've only the PR_ID, I'll add the PR_BRANCH, TARGET_BRANCH and URL. Anything else to add?

Here is a screenshot of what is provided by the Github Pull Request Builder (ghprb) plugin of Jenkins. Similar information would be useful:

 

- Does it support Github webhooks or polling only? (I know Github has a builtin GoCD webhook, should we leverage this?)

Currently it polls periodically (runs as part of Go's MDU). I don't have much insights on how Github's webhook work. Also, I'm not sure if that can be handled in the SCM endpoint.

If you go to the Github settings of your repo, under webhooks, for each of the configured hooks, you can see delivery of the hook to the configured endpoint. It contains all the information about each payload sent out (which on your side you can leverage). Here's screenshot of a payload sent from one of my Github projects to Snap-ci:

--

Alex