freestyle-multibranch, multibranch, scm-api questions

Kanstantsin Shautsou

unread,

Oct 19, 2017, 8:42:56 PM10/19/17

to Jenkins Developers

Trying to make multibranch support with not messed buildhistory by reusing my flexible github triggering code and experience. Have some questions after first scm/branch/freestyle API reading impression:

- Why uncategorized view is primary jenkins.branch.MultiBranchProjectViewHolder#getViews and how to set own primary view?

- Why uncategorized DEFAULT view (category) is named "Branch" by default?

- Freestyle-multibranch uses observer jenkins.branch.MultiBranchProject.SCMHeadObserverImpl#observe is it possible to replace it with custom logic?

This observer is not flexible and not extendable, it can do only simple case comparison this vs that by revision. How to run the same build if it was already built? Revision+Head are not enough information.

(Probably this is an answer why there is still no smart messages for rebuilds.

if (rebuild) {

    listener.getLogger().format(

"%s reopened: %s (%s)%n",) < why API expects defined behavior? Again, what if i want rebuild by comment, date, other ifluence?

One more place of decision making is SCMSourceCriteria... Oh, and Trait(?)... And BranchPropertyStrategy... How much places are designed for decision making and influence on final result?

Can i decide in one (my) place and just return to multibranch my decision: "this head, branch, etc should be built, please create views by corresponding SCMHeadCategory, etc" Or just not call observe and bypass it?

- Is State.class expected to be used as persistent storage of data for plugins? If yes, then how to control exceptions during xmlread to exclude data clearing?

Stephen Connolly

unread,

Oct 20, 2017, 4:50:51 AM10/20/17

to jenkin...@googlegroups.com

On 19 October 2017 at 17:42, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:

Trying to make multibranch support with not messed buildhistory by reusing my flexible github triggering code and experience.

Unclear what you mean by "not messed build history"... with multibranch, each branch gets its own clean history... the only issue that I know about build history is that when a new branch is discovered, its first build will have an empty history (we'd like to have it link to the other branch that this new branch was forked from... though that may not always be possible)

Or are you referring to the crazy of having git build many branches of in a single job that drove me to develop multibranch in the first place.

Have some questions after first scm/branch/freestyle API reading impression:
- Why uncategorized view is primary jenkins.branch.MultiBranchProjectViewHolder#getViews and how to set own primary view?

Because the "standard" SCMHeads should not have a category.

- Why uncategorized DEFAULT view (category) is named "Branch" by default?

That is a decision for the SCMSource. If Accurev gets around to implementing SCM API then that would probably have its uncategorised category called `Stream`

If you then have a multibranch project with two sources, one github and the other accurev then you would have:

- `Branches / Streams` as the default view

- `Snapshots / Tags` if somebody has enabled tag discovery on both SCMSources

- `Pull Requests` if you enable pull request discovery for GitHub but there is no equivalent category for accurev... if accurev had an equivalent functionality and it was enabled at the same time then you'd have this tab called `Change Requests / Pull Requests` (in the event that Accurev uses the term `Change Requests`... if they use the term `Pull Requests` then this would be just `Pull Requests` as both sources would be agreeing on the terminology.

- Freestyle-multibranch uses observer jenkins.branch.MultiBranchProject.SCMHeadObserverImpl#observe is it possible to replace it with custom logic?

Why should it do custom logic? It's job is just to create the project and trigger a build if and only if:

1. The SCMRevision is different from the previous

2. The BranchBuildStrategy gives the OK to build (or if there are no branch build strategies then auto-build everything except tags)

This observer is not flexible and not extendable, it can do only simple case comparison this vs that by revision. How to run the same build if it was already built? Revision+Head are not enough information.

BranchBuildStrategy is the place for more complex decisions, multibranch is opinionated and aims to remove the effect of 100 jobs trying to poll for changes and replace that with (worst case) one multibranch polling for changes or (best case) one multibranch being pushed events

(Probably this is an answer why there is still no smart messages for rebuilds.
if (rebuild) {
    listener.getLogger().format(
"%s reopened: %s (%s)%n",) < why API expects defined behavior? Again, what if i want rebuild by comment, date, other ifluence?

Freestyle-multibranch was an early experiment, probably needs some updating to the latest code.

One more place of decision making is SCMSourceCriteria... Oh, and Trait(?)... And BranchPropertyStrategy... How much places are designed for decision making and influence on final result?

The SCMSourceCriteria is to decide *is this a branch/tag/pr that should have a project created*

The BranchPropertyStrategy is to decorate each child job

The BranchBuildStrategy is to decide when to build branches

Can i decide in one (my) place and just return to multibranch my decision: "this head, branch, etc should be built, please create views by corresponding SCMHeadCategory, etc" Or just not call observe and bypass it?

The Multibranch is supposed to create one view for each of the (consolidated) categories that its SCMSources report are enabled. If you don't like that then you can override `newFolderViewHolder` and return your own view holder. Keep in mind that in an Org Folder the user will not be able to configure the views for each repository, so the view holder must be automatic.

- Is State.class expected to be used as persistent storage of data for plugins? If yes, then how to control exceptions during xmlread to exclude data clearing?

It is intended to store the SCM-API state in order to allow the branch-api to decide when to trigger rebuilds, etc

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5e646943-4151-4084-9f92-89788467a5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kanstantsin Shautsou

unread,

Oct 20, 2017, 7:02:20 AM10/20/17

to Jenkins Developers

2017-10-20 11:50 GMT+03:00 Stephen Connolly <stephen.al...@gmail.com>:

On 19 October 2017 at 17:42, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:
Trying to make multibranch support with not messed buildhistory by reusing my flexible github triggering code and experience.

Unclear what you mean by "not messed build history"... with multibranch, each branch gets its own clean history... the only issue that I know about build history is that when a new branch is discovered, its first build will have an empty history (we'd like to have it link to the other branch that this new branch was forked from... though that may not always be possible)

Or are you referring to the crazy of having git build many branches of in a single job that drove me to develop multibranch in the first place.

Yes, i mean that it good to have separated history.

Have some questions after first scm/branch/freestyle API reading impression:
- Why uncategorized view is primary jenkins.branch.MultiBranchProjectViewHolder#getViews and how to set own primary view?

Because the "standard" SCMHeads should not have a category.

Why?

- Why uncategorized DEFAULT view (category) is named "Branch" by default?

That is a decision for the SCMSource. If Accurev gets around to implementing SCM API then that would probably have its uncategorised category called `Stream`

If you then have a multibranch project with two sources, one github and the other accurev then you would have:

- `Branches / Streams` as the default view
- `Snapshots / Tags` if somebody has enabled tag discovery on both SCMSources
- `Pull Requests` if you enable pull request discovery for GitHub but there is no equivalent category for accurev... if accurev had an equivalent functionality and it was enabled at the same time then you'd have this tab called `Change Requests / Pull Requests` (in the event that Accurev uses the term `Change Requests`... if they use the term `Pull Requests` then this would be just `Pull Requests` as both sources would be agreeing on the terminology.

AFAIR they will be in one view only when they will have one category that sounds weird to mix different sources in one view.

- Freestyle-multibranch uses observer jenkins.branch.MultiBranchProject.SCMHeadObserverImpl#observe is it possible to replace it with custom logic?

Why should it do custom logic? It's job is just to create the project and trigger a build if and only if:

1. The SCMRevision is different from the previous

That wrong assumption locked all APIs to only this one possible thing in triggering branch/pr builds.

Though i can try to hack it and control build triggering via it in deterministics and equals.

2. The BranchBuildStrategy gives the OK to build (or if there are no branch build strategies then auto-build everything except tags)

This observer is not flexible and not extendable, it can do only simple case comparison this vs that by revision. How to run the same build if it was already built? Revision+Head are not enough information.

BranchBuildStrategy is the place for more complex decisions, multibranch is opinionated and aims to remove the effect of 100 jobs trying to poll for changes and replace that with (worst case) one multibranch polling for changes or (best case) one multibranch being pushed events

Where did you saw 100 jobs targeted to one repository for 100 branches? Before people used one trigger that produced messed build history with different branches/prs in one job, but trigger was one, now the same.

(Probably this is an answer why there is still no smart messages for rebuilds.
if (rebuild) {
listener.getLogger().format(
"%s reopened: %s (%s)%n",) < why API expects defined behavior? Again, what if i want rebuild by comment, date, other ifluence?

Freestyle-multibranch was an early experiment, probably needs some updating to the latest code.

This code is from fresh branch api observer.

One more place of decision making is SCMSourceCriteria... Oh, and Trait(?)... And BranchPropertyStrategy... How much places are designed for decision making and influence on final result?

The SCMSourceCriteria is to decide *is this a branch/tag/pr that should have a project created*

The BranchPropertyStrategy is to decorate each child job

The BranchBuildStrategy is to decide when to build branches

Can i decide in one (my) place and just return to multibranch my decision: "this head, branch, etc should be built, please create views by corresponding SCMHeadCategory, etc" Or just not call observe and bypass it?

The Multibranch is supposed to create one view for each of the (consolidated) categories that its SCMSources report are enabled. If you don't like that then you can override `newFolderViewHolder` and return your own view holder. Keep in mind that in an Org Folder the user will not be able to configure the views for each repository, so the view holder must be automatic.

- Is State.class expected to be used as persistent storage of data for plugins? If yes, then how to control exceptions during xmlread to exclude data clearing?

It is intended to store the SCM-API state in order to allow the branch-api to decide when to trigger rebuilds, etc

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5e646943-4151-4084-9f92-89788467a5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/gEH1EdzMblM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwr6eTNZoEn6tiyi5peuZgWrQeH21s_%3DtYw-PO9qiKJOA%40mail.gmail.com.

Stephen Connolly

unread,

Oct 20, 2017, 7:16:41 AM10/20/17

to jenkin...@googlegroups.com

On 20 October 2017 at 04:02, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:

2017-10-20 11:50 GMT+03:00 Stephen Connolly <stephen.alan.connolly@gmail.com>:
On 19 October 2017 at 17:42, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:
Trying to make multibranch support with not messed buildhistory by reusing my flexible github triggering code and experience.

Unclear what you mean by "not messed build history"... with multibranch, each branch gets its own clean history... the only issue that I know about build history is that when a new branch is discovered, its first build will have an empty history (we'd like to have it link to the other branch that this new branch was forked from... though that may not always be possible)

Or are you referring to the crazy of having git build many branches of in a single job that drove me to develop multibranch in the first place.
Yes, i mean that it good to have separated history.

Have some questions after first scm/branch/freestyle API reading impression:
- Why uncategorized view is primary jenkins.branch.MultiBranchProjectViewHolder#getViews and how to set own primary view?

Because the "standard" SCMHeads should not have a category.
Why?

because categories are a mix-in. Some SCMs will not have different types of "branch like" things that can be discovered.

- Why uncategorized DEFAULT view (category) is named "Branch" by default?

That is a decision for the SCMSource. If Accurev gets around to implementing SCM API then that would probably have its uncategorised category called `Stream`

If you then have a multibranch project with two sources, one github and the other accurev then you would have:

- `Branches / Streams` as the default view
- `Snapshots / Tags` if somebody has enabled tag discovery on both SCMSources
- `Pull Requests` if you enable pull request discovery for GitHub but there is no equivalent category for accurev... if accurev had an equivalent functionality and it was enabled at the same time then you'd have this tab called `Change Requests / Pull Requests` (in the event that Accurev uses the term `Change Requests`... if they use the term `Pull Requests` then this would be just `Pull Requests` as both sources would be agreeing on the terminology.
AFAIR they will be in one view only when they will have one category that sounds weird to mix different sources in one view.

So the expected case is that you only ever have one SCMSource... but consider the case where you are migrating from one SCM to another for your "project"

In that case you might have both Accurev and GitHub with the maintenance branches on one and the new development on another.

Now when I need to go look at the status of a "branch" (and as a developer I don't give a f*ck whether the SCM called that a "Stream" or a "Branch") I want to go to one tab and see all of them together... same goes for tag/snapshot and pull request/change request.

So we group the same categories together... categories are a way of grouping outside of the class hierarchy

- Freestyle-multibranch uses observer jenkins.branch.MultiBranchProject.SCMHeadObserverImpl#observe is it possible to replace it with custom logic?

Why should it do custom logic? It's job is just to create the project and trigger a build if and only if:

1. The SCMRevision is different from the previous
That wrong assumption locked all APIs to only this one possible thing in triggering branch/pr builds.
Though i can try to hack it and control build triggering via it in deterministics and equals.

2. The BranchBuildStrategy gives the OK to build (or if there are no branch build strategies then auto-build everything except tags)

This observer is not flexible and not extendable, it can do only simple case comparison this vs that by revision. How to run the same build if it was already built? Revision+Head are not enough information.

BranchBuildStrategy is the place for more complex decisions, multibranch is opinionated and aims to remove the effect of 100 jobs trying to poll for changes and replace that with (worst case) one multibranch polling for changes or (best case) one multibranch being pushed events
Where did you saw 100 jobs targeted to one repository for 100 branches? Before people used one trigger that produced messed build history with different branches/prs in one job, but trigger was one, now the same.

If you were to have multibranch just creating the jobs, you'd have 100 jobs all doing polling for some SCMs... don't just think Git

(Probably this is an answer why there is still no smart messages for rebuilds.
if (rebuild) {
listener.getLogger().format(
"%s reopened: %s (%s)%n",) < why API expects defined behavior? Again, what if i want rebuild by comment, date, other ifluence?

Freestyle-multibranch was an early experiment, probably needs some updating to the latest code.
This code is from fresh branch api observer.

I had a multibranch freestyle impl that I pushed the code for... I didn't do any more on it because of the UI config problem.

One more place of decision making is SCMSourceCriteria... Oh, and Trait(?)... And BranchPropertyStrategy... How much places are designed for decision making and influence on final result?

The SCMSourceCriteria is to decide *is this a branch/tag/pr that should have a project created*

The BranchPropertyStrategy is to decorate each child job

The BranchBuildStrategy is to decide when to build branches

Can i decide in one (my) place and just return to multibranch my decision: "this head, branch, etc should be built, please create views by corresponding SCMHeadCategory, etc" Or just not call observe and bypass it?

The Multibranch is supposed to create one view for each of the (consolidated) categories that its SCMSources report are enabled. If you don't like that then you can override `newFolderViewHolder` and return your own view holder. Keep in mind that in an Org Folder the user will not be able to configure the views for each repository, so the view holder must be automatic.

- Is State.class expected to be used as persistent storage of data for plugins? If yes, then how to control exceptions during xmlread to exclude data clearing?

It is intended to store the SCM-API state in order to allow the branch-api to decide when to trigger rebuilds, etc

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5e646943-4151-4084-9f92-89788467a5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/gEH1EdzMblM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwr6eTNZoEn6tiyi5peuZgWrQeH21s_%3DtYw-PO9qiKJOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAM9nkw9u%3DzJBm%3DCuH-4m8jEyd%2BNO8mgf0H1%3D%3DsNr0Dun5vpvHw%40mail.gmail.com.

Kanstantsin Shautsou

unread,

Oct 20, 2017, 4:14:48 PM10/20/17

to Jenkins Developers

Could you remind what exactly? Something around descriptor and displayName?

One more place of decision making is SCMSourceCriteria... Oh, and Trait(?)... And BranchPropertyStrategy... How much places are designed for decision making and influence on final result?

The SCMSourceCriteria is to decide *is this a branch/tag/pr that should have a project created*

The BranchPropertyStrategy is to decorate each child job

The BranchBuildStrategy is to decide when to build branches

Can i decide in one (my) place and just return to multibranch my decision: "this head, branch, etc should be built, please create views by corresponding SCMHeadCategory, etc" Or just not call observe and bypass it?

The Multibranch is supposed to create one view for each of the (consolidated) categories that its SCMSources report are enabled. If you don't like that then you can override `newFolderViewHolder` and return your own view holder. Keep in mind that in an Org Folder the user will not be able to configure the views for each repository, so the view holder must be automatic.

- Is State.class expected to be used as persistent storage of data for plugins? If yes, then how to control exceptions during xmlread to exclude data clearing?

It is intended to store the SCM-API state in order to allow the branch-api to decide when to trigger rebuilds, etc

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5e646943-4151-4084-9f92-89788467a5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/gEH1EdzMblM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwr6eTNZoEn6tiyi5peuZgWrQeH21s_%3DtYw-PO9qiKJOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Stephen Connolly

unread,

Oct 20, 2017, 4:24:36 PM10/20/17

to jenkin...@googlegroups.com

https://github.com/jenkinsci/freestyle-multibranch-plugin has last commit from me... what code are you looking at?

I had a multibranch freestyle impl that I pushed the code for... I didn't do any more on it because of the UI config problem.
Could you remind what exactly? Something around descriptor and displayName?

Form validation.

All the descriptors assume they are in a FreeStyleProject... or if you are luck an AbstractProject... or perhaps if you are really lucky, a Job...

So form validation blows up when the only ancestor in path is an Item (technically a ComputedFolder)

Then there's even more advanced form validation that blows up when there is no workspace against which to validate builds, etc.

You can save the config alright... just very hard to know if the config is right

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3766d647-49a9-4e39-bdde-841d19355cb5%40googlegroups.com.

Jesse Glick

unread,

Oct 20, 2017, 6:14:48 PM10/20/17

to Jenkins Dev

Have not been following closely, but offhand:

On Fri, Oct 20, 2017 at 4:24 PM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
> Form validation.
>
> All the descriptors assume they are in a FreeStyleProject... or if you are
> luck an AbstractProject... or perhaps if you are really lucky, a Job...
>
> So form validation blows up when the only ancestor in path is an Item
> (technically a ComputedFolder)

IIRC I tried to work around this with

https://github.com/jenkinsci/freestyle-multibranch-plugin/commit/4081dcdf05bea9e6f47982e37099d176dcc2be45

> Then there's even more advanced form validation that blows up when there is
> no workspace against which to validate builds, etc.

There were a bunch of cases like this in various plugins that have
gotten fixed. Broadly speaking, any `@AncestorInPath` should also be
`@CheckForNull`.

Kanstantsin Shautsou

unread,

Nov 1, 2017, 6:07:01 PM11/1/17

to Jenkins Developers

So how to do the very easy feature: trigger PR/builds only when they match to some criteria unrelated to hash i.e. trigger branch only when commit message matches to XXX, PRs to some trigger message?

And the second case: trigger build only when PR is closed, branch deleted.

How bypass dumb SCMHeadObserverImpl ? Make revision nondetermenistic and guess they way in if cases to offload trigger decision logic to flexible implementation?

Just curious, have you ever looked at ghprb and github-integration-plugins long existing features when designed scm-api&branch-api?

Reply all

Reply to author

Forward