[JIRA] (JENKINS-59317) Spurious SCM change with a failed git call (e.g. network error)

0 views
Skip to first unread message

jimklimov@gmail.com (JIRA)

unread,
Sep 11, 2019, 10:08:04 AM9/11/19
to jenkinsc...@googlegroups.com
Jim Klimov created an issue
 
Jenkins / Improvement JENKINS-59317
Spurious SCM change with a failed git call (e.g. network error)
Issue Type: Improvement Improvement
Assignee: Mark Waite
Components: git-plugin
Created: 2019-09-11 14:07
Priority: Minor Minor
Reporter: Jim Klimov

Our internal build farm's connection to the internet has its flaws, ups and downs, and sometimes during the downs we see gems like this: an effectively failed SCM poll ("Couldn't get remote head revision") is interpreted as a "Changes found" situation, and causes a build "Started by an SCM change", as the (failed) build's Polling Log says:

````
Polling Log

This page captures the polling log that triggered this build.

Started on Sep 7, 2019 6:15:15 AM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
Unexpected ls-remote output line ''
[poll] Couldn't get remote head revision
Done. Took 23 sec
Changes found
````

In this particular situation (outage) as a result, this build fails quickly during checkout of sources with
```
...
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/42ity/fty-outage.git +refs/heads/release/IPM_Infra-1.4:refs/remotes/origin/release/IPM_Infra-1.4" returned status code 128:
stdout:
stderr: fatal: unable to access 'https://github.com/42ity/fty-outage.git/': The requested URL returned error: 503
...
```
which is a reasonable outcome of a network outage (our proxy failed to fetch => 503)... however when the network comes back, the last-tried (green, long ago) revision is the same as the current HEAD, so no need to rebuild, as the branch's Git polling log says:
````
Git Polling Log

Started on Sep 11, 2019 1:59:13 PM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
````

So we end up with with a latest-known state of a branch being "failed", more prominently so on still-tracked old releases that are in fact untouched for months and years (and won't be, unless a need for some hotfix arises).

As a workaround, I have to go to such jobs and manually rebuild them, so they do not pollute our logs and job lists with red lights, and do not trigger alarms we've made.

I believe the undetermined HEAD revision should either not cause a build, or if as a failsafe we want to try one - be remembered as such (null?) and when the network comes back, cause a re-rebuild with a definitive commit, even if same as tried before, so the latest-build status converges to a valid up-to-date verdict.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 11, 2019, 11:23:03 AM9/11/19
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite
Reply all
Reply to author
Forward
0 new messages