GitHub and Bitbucket branch source UI refactoring

200 views
Skip to first unread message

Stephen Connolly

unread,
Jun 16, 2017, 2:19:12 PM6/16/17
to jenkin...@googlegroups.com, jenkins...@googlegroups.com
Just a quick status update.

In final stages of this work now. Bobby is being a superstar and reviewing my 13k LoC change on the Bitbucket branch source - brings lots of feature parity with GitHub and adds the configuration ability of the pure Git branch source

I am finalising the GitHub Branch Source changes... likely to be another big PR

Then there's a 5k LoC change in the Git plugin

Plan is to try and get all merged next week and cut a beta

I'll be looking for people to help test at that stage.

Please respond if you think you can help (lots of bugs fixed as a side effect of the refactoring - it makes things more easy to test => I found and fixed bugs)
--
Sent from my phone

Mark Waite

unread,
Jun 16, 2017, 2:35:47 PM6/16/17
to jenkin...@googlegroups.com, jenkins...@googlegroups.com
I'd like to be part of the beta test.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxfYrZphgYDXFD3i%2Bo_7eDn7mn2qVrzJz6wFaoVkNmc%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Stephen Connolly

unread,
Jun 16, 2017, 2:39:02 PM6/16/17
to jenkin...@googlegroups.com
Well I kind of though you were required- given the git plugin is part of the changes ;-)


For more options, visit https://groups.google.com/d/optout.

Dan Tran

unread,
Jun 16, 2017, 6:28:52 PM6/16/17
to Jenkins Users, jenkin...@googlegroups.com
I will give it a spin too.

Thanks

-Dan

On Friday, June 16, 2017 at 11:57:26 AM UTC-7, Kevin Burnett wrote:
we'd be down to try that, yes. thanks for making these changes in a way that will benefit the product long-term!

fingers are crossed that there's already a built-in way to pretend like pull requests don't exist! you're already building the branches; why also build the pull requests, eh? :)

thanks!
kb

Joseph P

unread,
Jun 16, 2017, 6:32:01 PM6/16/17
to Jenkins Developers, jenkins...@googlegroups.com
I'd love to get the bitbucket going, I really want merged PRs TODAY :D

Stephen Connolly

unread,
Jun 16, 2017, 6:52:07 PM6/16/17
to jenkin...@googlegroups.com
On Fri 16 Jun 2017 at 23:32, Joseph P <jose...@gmail.com> wrote:
I'd love to get the bitbucket going, I really want merged PRs TODAY :D

If you are on BB server you should be fine

If you are on BB cloud, unless you set permissions up when forking, only PRs from forks in the team account and PRs from the origin repo will be built.

You can choose merge, head or both

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Owen Mehegan

unread,
Jun 16, 2017, 6:57:41 PM6/16/17
to jenkin...@googlegroups.com
Does this mean we can move forward with working on GitLab branch source work after these changes are final?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxfYrZphgYDXFD3i%2Bo_7eDn7mn2qVrzJz6wFaoVkNmc%2Bw%40mail.gmail.com.

Stephen Connolly

unread,
Jun 17, 2017, 2:30:38 AM6/17/17
to jenkin...@googlegroups.com
On Fri 16 Jun 2017 at 23:57, Owen Mehegan <omeh...@gmail.com> wrote:
Does this mean we can move forward with working on GitLab branch source work after these changes are final?

Yes

(I think the changes are final now, but let's see what it takes to get the GitHub changes polishing finished)

On Fri, Jun 16, 2017 at 11:18 AM, Stephen Connolly <stephen.al...@gmail.com> wrote:
Just a quick status update.

In final stages of this work now. Bobby is being a superstar and reviewing my 13k LoC change on the Bitbucket branch source - brings lots of feature parity with GitHub and adds the configuration ability of the pure Git branch source

I am finalising the GitHub Branch Source changes... likely to be another big PR

Then there's a 5k LoC change in the Git plugin

Plan is to try and get all merged next week and cut a beta

I'll be looking for people to help test at that stage.

Please respond if you think you can help (lots of bugs fixed as a side effect of the refactoring - it makes things more easy to test => I found and fixed bugs)
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAHtcACF7FpgZ%3D_i1596JMuWRipfKhxVPNT3n%3D%3DzPt%2BwUOaOb0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Łukasz Zachulski

unread,
Jun 17, 2017, 2:43:29 AM6/17/17
to Jenkins Developers
Can you summarize what can be expected from those new versions
Is it really only UI refactoring or do changes introduce something more?

Stephen Connolly

unread,
Jun 17, 2017, 4:36:00 AM6/17/17
to jenkin...@googlegroups.com
So there is the UI refactoring that should make configuration easier to understand: instead of 6 checkboxes with 64 possible combinations that are hard to comprehend and identify which combinations are nonsensical... there are three "discovery" traits that control discovery of origin branches, origin PRs and fork PRs

New functionality is that we can now expose as traits configuration of the git extensions, so if you want to checkout to a local branch rather than a detached head, if you want to control submodule behaviour, etc that is all now available (we also remove the git extensions options that are incompatible with branch sources to stop people complaining when they don't work)

New functionality is that - by default - we only clone the minimal ref specs required (add the advanced clone behaviour trait and it will be back to as before)

New functionality is that we can use shared code for filtering repositories and branches by wild cards or by regex (and somebody can add Glob style wild cards via an extension plugin and *everyone* gets them!)

Then there are the subtle bugs that I fixed due to the better testability of the trait approach

Then Bitbucket gets merge commits (except mercurial repositories on Bitbucket cloud... but I hope to address that once I figure out how to get mercurial to do the merges)

I claim that the data migration is fully tested... so what I am looking for from testers is:

* any issues with data migration
* any branches / PRs that were present before and have now disappeared on first rescan after upgrade
* any branches / PRs that fall in a "rebuild" storm after upgrade / rescan

I believe there should be none of these issues but we need testing from other people's real configuration before we can push an non-beta release for a set of changes this big (40k LoC in the diffs... though a lot - at least half - of that is new tests)

On Sat 17 Jun 2017 at 07:43, Łukasz Zachulski <lzach...@gmail.com> wrote:
Can you summarize what can be expected from those new versions
Is it really only UI refactoring or do changes introduce something more?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Joseph P

unread,
Jun 17, 2017, 7:01:42 AM6/17/17
to Jenkins Developers
BB server :) so all good

Stephen Connolly

unread,
Jun 20, 2017, 9:29:33 AM6/20/17
to jenkins...@googlegroups.com, jenkin...@googlegroups.com
OK! Here we are... testing time!

These are the plugins that are being covered: (download links should be live in an hour or two)


Recommended testing procedure:

1. Set up a throw-away Jenkins running a version similar to your production environment with the pre-upgrade versions of the plugins you are using.
2. Set up ideally at least one organization folder and one standalone multibranch project building your source code - to a first order you do not care if the builds succeed or fail, only that the branches are found.
3. Trigger a scan / index of your organization folders and standalone multibranch projects.
4. Wait for the queue to complete
5. Run the script in the system script console: https://gist.github.com/stephenc/64ef58783b4438a126ad4e3f43062df1 and save the output to smoke-pre-upgrade.txt
6. Upgrade the relevant plugins, restart Jenkins.
7. Run the script in the system script console: https://gist.github.com/stephenc/64ef58783b4438a126ad4e3f43062df1 and save the output to smoke-post-upgrade.txt
8. Trigger a scan / index of your organization folders and standalone multibranch projects.
9. Wait for the queue to complete
10. Run the script in the system script console: https://gist.github.com/stephenc/64ef58783b4438a126ad4e3f43062df1 and save the output to smoke-post-rescan.txt

At this point, do a diff between smoke-pre-upgrade.txt and smoke-post-rescan.txt

You are looking for three classes of difference:

a. branch jobs that have been rebuilt for no reason (i.e. the revision is the same)
b. branch jobs that have disappeared for no good reason (i.e. the branch is still present in the backing scm)
c. branch jobs that have suddenly appeared for no good reason (i.e. the branch was there before but not found) [expecting some of these for BitBucket PRs from forks, but only after configuration updated, saved and another rescan performed]

My expectation is that nobody will have these kinds of issues.

Also try out the new UI to see what you think.

Please report back your testing results either way. Don't forget to report back your UI feedback too ;-)

After doing that test in a throw-away Jenkins, you can *optionally* repeat the test on a more production-like (emphasis on being production-like not production) instance... but this is code that has not yet completed code review (hence -alpha-1 not -beta-1) so it is at your own risk. There are additional issues to be aware when using more production-like environment:

a. You may have builds that were assuming branches were full clones, now the refspec is tightly reduced to minimize clone time. If you need a full clone you will need to add the "Advanced Clone" behaviour.
b. Mercurial repositories on Bitbucket Cloud do not support merge commits for PR building (yet)
c. Credential domains were not being correctly compared so as a result - if you are using credential domains to help sort credentials - there may be cases where the credentials are now searched for in a different domain than you had them in, so your domains may need reconfiguration to have the credentials found by the multibranch project / org folder.
d. The pipeline snippitizer is generating $class style for some of the GitHub and BitBucket specific behaviours, this is because my plan is to further consolidate the implementations and have a single shared implementation of each for these plugins, that way they can have a single @Symbol annotation... if that is too difficult then the @Symbol would need to be prefixed with gitHub / bitbucket respectively, e.g. gitHubBranches, bitbucketBranches for the discover branches behaviour.


Thanks in advance

-Stephen

On 18 June 2017 at 15:53, Michael Kobit <mko...@gmail.com> wrote:

I may be able to help with this as well.


You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5dd15ac2-b8a2-4ebd-bb4a-3bffa4815227%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALELY9GHbX4WuHdDKM8-bU1xR5voh-NsfHeQXNAxMjJpXkiwrw%40mail.gmail.com.

Stephen Connolly

unread,
Jun 20, 2017, 9:53:02 AM6/20/17
to jenkins...@googlegroups.com, jenkin...@googlegroups.com
If you are chomping at the bit, here are all the binaries:


SHA1 checksums:
d9c346ac8db497a35825c7dbbb934842a2bc429a  branch-api.hpi
16da429f09fb585fd1d744809ee22c8d612fb62c  cloudbees-bitbucket-branch-source.hpi
234fa8eb88dad3241d620bb0116dd12fb9decbba  git.hpi
a68be01144f3045f81a5cf3c0bc60ad12f39b643  github-branch-source.hpi
92237097815b45260bb8b272caa9be9f92eb5085  mercurial.hpi
04c321420b3752a8d8b3af89cae1bf5934607b1c  scm-api.hpi

SHA256 checksums:
858ce20992c3f179b850c512979999084b11fe7c4c173cf6d4d2e07bbfebf3e7  branch-api.hpi
8ebff7a3ec43df276d4b51d1e5bcb910bbe8eb4cd47a4be0e35f2f2ca1cd0e03  cloudbees-bitbucket-branch-source.hpi
46cbbf11395df4a085829094d5a36dee7328aeba00d33e34b44aa0dcf9898248  git.hpi
6495a60f1bf0733d807f412434c6c2e24b7bba53fd7ce348ca5319ef38571f20  github-branch-source.hpi
173d12042fe8582efdb52e740f4e939b9daa05f181c6aaff31824337d519a31c  mercurial.hpi
9b58e9e6d13ce90a91b73f38142bf0977f244df9c52b948988f9d5bdc3785481  scm-api.hpi

-Stephen

Stephen Connolly

unread,
Jun 21, 2017, 2:30:03 PM6/21/17
to jenkins...@googlegroups.com, jenkin...@googlegroups.com
How many people have been able to try this so far?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Mark Waite

unread,
Jun 21, 2017, 2:39:30 PM6/21/17
to jenkin...@googlegroups.com, jenkins...@googlegroups.com
I haven't tried it yet.  I'm configuring the "before" state today and will capture its state, then will deploy the new code tomorrow morning and capture the after state.  I won't do anything to compare those until tomorrow evening or this weekend.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMz1NimOSn1UYZyjwZ4KFEiDFoHf6L9Wy7Q3vyDb6959-Q%40mail.gmail.com.

Michael Neale

unread,
Jun 21, 2017, 11:29:53 PM6/21/17
to Jenkins Developers, jenkins...@googlegroups.com
I saw all jobs disappear in a github org folder that blue ocean created previously, when I triggered a scan. 

(it also doesn't seem to trigger scans if I add change the pattern, or run the wizard again). 

Stephen Connolly

unread,
Jun 22, 2017, 1:20:25 AM6/22/17
to jenkin...@googlegroups.com, jenkins...@googlegroups.com
On Thu 22 Jun 2017 at 04:29, Michael Neale <mne...@cloudbees.com> wrote:
I saw all jobs disappear in a github org folder that blue ocean created previously, when I triggered a scan. 

(it also doesn't seem to trigger scans if I add change the pattern, or run the wizard again). 

I need to investigate some of the hacks BO is using.

I suspect this is probably just one deprecated constructor that needs tweaking


For more options, visit https://groups.google.com/d/optout.

Stephen Connolly

unread,
Jun 22, 2017, 2:05:52 AM6/22/17
to jenkin...@googlegroups.com, jenkins...@googlegroups.com
I believe the issue is due to BlueOcean continually replacing the GitHubSCMNavigator and the legacy constructor not populating the discovery traits using the legacy defaults. Great that we caught this now.

This does point to an issue for anyone using BlueOcean and configuring traits on a BlueOcean-managed organization folder... namely any configuration they do in the classic UI will be blown away by the next change from the BlueOcean UI... this only applies to GitHub Org Folders

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

Joseph P

unread,
Jun 22, 2017, 2:10:41 AM6/22/17
to Jenkins Developers, jenkins...@googlegroups.com
I will try and get around to testing it today.

Michael Neale

unread,
Jun 22, 2017, 4:27:34 AM6/22/17
to Jenkins Users, jenkin...@googlegroups.com
It does look better now with the new version. I confirmed running a scan didn't make anything go away. 

Stephen Connolly

unread,
Jun 22, 2017, 6:56:39 AM6/22/17
to jenkin...@googlegroups.com, Jenkins Users
So yes, it's github-branch-source 2.2.0-alpha-3 that is required for as good as possible inter-op with BlueOcean.

Note that when using BlueOcean with GitHub Org Folders you will fall victim to JENKINS-45058 irrespective of the version of GitHub Branch Source that you are using... the migration to traits makes this issue even more evident as now it is no longer the includes / excludes / build branches / build branches that are PRs / build origin PR head / build origin PR merge / build fork PR head / build fork PR merge / pattern fields that are subject to being reset but also the new traits... so if you configure say a Git LFS checkout behaviour and then add a repository with Blue Ocean, that Git LFS behaviour will be removed by BlueOcean

https://github.com/jenkinsci/blueocean-plugin/pull/1188 or https://github.com/jenkinsci/blueocean-plugin/pull/1187 fix that bug in BlueOcean, but there are some questions that need resolving before those PRs can be finalized.

https://github.com/jenkinsci/blueocean-plugin/pull/1186 shows that the changes (as of 2.2.0-alpha-2 + ) do not introduce any regressions in the interaction with BlueOcean... though obviously as JENKINS-45058 was always present, it remains so.    

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0a9e9a1d-ac99-4acb-a9ca-8546c40b6377%40googlegroups.com.

Mark Waite

unread,
Jun 22, 2017, 8:41:36 AM6/22/17
to jenkin...@googlegroups.com, Jenkins Users
I used the experimental update center to update to the latest beta plugin versions.  I didn't have access from my Jenkins to the CloudBees BitBucket Source Branch plugin, so I'm ignoring it.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0a9e9a1d-ac99-4acb-a9ca-8546c40b6377%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwQU7%3D74BGP4grSWt4e00ct-RwkA8KZp3HBV02scHKtkg%40mail.gmail.com.

Stephen Connolly

unread,
Jun 22, 2017, 9:17:36 AM6/22/17
to jenkin...@googlegroups.com, Jenkins Users
It's now known as the BitBucket Branch Source plugin... but the plugin short-name is cloudbees-bitbucket-branch-source

On 22 June 2017 at 13:41, Mark Waite <mark.ea...@gmail.com> wrote:
I used the experimental update center to update to the latest beta plugin versions.  I didn't have access from my Jenkins to the CloudBees BitBucket Source Branch plugin, so I'm ignoring it.

Mark Waite

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwQU7%3D74BGP4grSWt4e00ct-RwkA8KZp3HBV02scHKtkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtEgQcGACu6RgfyjbQk%2BnSO3FUvdQOGM_Q09j1nejFVOJQ%40mail.gmail.com.

Stephen Connolly

unread,
Jun 22, 2017, 9:19:43 AM6/22/17
to jenkin...@googlegroups.com, Jenkins Users
In any case we are probably more interested in other Bitbucket users as at most you (Mark) would end up using the CloudBees testing Bitbucket Server instance which is the same one I used and we are more interested in testing a variety of configurations not multiple people testing the same configuration ;-)

Mark Waite

unread,
Jun 22, 2017, 9:29:54 AM6/22/17
to jenkin...@googlegroups.com, Jenkins Users
On Thu, Jun 22, 2017 at 7:19 AM Stephen Connolly <stephen.al...@gmail.com> wrote:
In any case we are probably more interested in other Bitbucket users as at most you (Mark) would end up using the CloudBees testing Bitbucket Server instance which is the same one I used and we are more interested in testing a variety of configurations not multiple people testing the same configuration ;-)


My bitbucket testing is typically limited to the bitbucket.org account that I've used for git plugin testing.  I'm sure that type of account (and server) will be exercised by others.

I hadn't added that plugin to my standard set of installed plugins.  I'll add it now.

Mark Waite
 
On 22 June 2017 at 14:17, Stephen Connolly <stephen.al...@gmail.com> wrote:
It's now known as the BitBucket Branch Source plugin... but the plugin short-name is cloudbees-bitbucket-branch-source
On 22 June 2017 at 13:41, Mark Waite <mark.ea...@gmail.com> wrote:
I used the experimental update center to update to the latest beta plugin versions.  I didn't have access from my Jenkins to the CloudBees BitBucket Source Branch plugin, so I'm ignoring it.

Mark Waite