Environment variable "sometimes" not expanded as "Repository URL"

[mj1414...@gmail.com]

Hello,

we’re running Jenkins 2.277.4 on RH7 using OpenJDK11 (and no container) + Git plugin 4.7.1

In “Global properties” of the Jenkins configuration, we’ve defined a shell variable:

GerritSlave

ssh://hy@gerrit-....com:29418/repo/path

 

Later on, in the job configuration we use this in

Source Code Management

  Git

    Repository URL

       ${GerritSlave}

 

Having the job triggered by polling.

 Now we had to find, that we see jobs not starting and in “Git polling log” we find:

--------

Started on Jun 8, 2021, 5:26:00 PM

Polling SCM changes on master

Using strategy: Default

[poll] Last Built Revision: Revision 5cb005655e96b87a21df36a5c1dad1ebdb925854 (refs/remotes/origin/branches/maintenance/PSint)

The recommended git tool is: NONE

using credential 50da578b-178c-4cba-b103-91c74f6522ed

> git rev-parse --resolve-git-dir /var/fpwork/WS/PHY_TX--LF--PSint--buildTrigger/PHY_TX/.git # timeout=10

Fetching changes from the remote Git repositories

> git config remote.origin.url ${GerritSlave} # timeout=10

Cleaning workspace

> git rev-parse --verify HEAD # timeout=10

Resetting working tree

> git reset --hard # timeout=10

> git clean -fdx # timeout=10

Fetching upstream changes from ${GerritSlave}

> git --version # timeout=10

> git --version # 'git version 2.30.0'

> git config --get remote.${GerritSlave}.url # timeout=10

FATAL: hudson.plugins.git.GitException: Failed to fetch from ${GerritSlave}

hudson.plugins.git.GitException: Command "git config --get remote.${GerritSlave}.url" returned status code 1:

stdout:

stderr:

  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2661)

 -------

So it looks like the variable didn’t get expanded and then the polling fails.

If this happens, it seems the polling gets stuck in this situation.

 

If I now trigger the job once manually (“Build with parameters”), the GIT polling will work: The variable gets expanded as expected, the job gets triggered if needed.

But it seems eventually, it gets stuck in the same way again.

I’m not sure, but it seems the problem started with updating the Jenkins core (from 2.263.4 to 2.277.4) and the GIT plugin.

Any ideas how to debug – or even better solve – this problem ?

Anyone seeing similar issues, maybe since updating ?

Regards

Martin

[Björn Pedersen]

> git config --get remote.${GerritSlave}.url # timeout=10

FATAL: hudson.plugins.git.GitException: Failed to fetch from ${GerritSlave}

This  seems to indicate the ${GerritSlave} is used in the wrong field (the 'name' field) . It should ever be used in the repo url field.

Björn

[mj1414...@gmail.com]

H Björn,

thanks for the hint. I had assumed, my good and bad cases behave the same way, except the expanding of the var and the error message.
But they behave differently, although I didn't touch the job configuration in between:

1. If I now trigger the job once manually (“Build with parameters”), the GIT polling will start to work: The variable gets expanded as expected, the job gets triggered if needed.
   No configuration change. So the configuration seems to be correct, but it seems something gets "confused". (See also below)

2. We download the job configuration from Jenkins for all jobs once every day (by a CI job) and commit the changes to SVN.
   Between 18. May and 7. June there where no changes besides due to plugin updates (like "plugin="parameteri...@2.40")
   Between 7. June and today, this got added (by a plugin update I assume)
         <hudson.plugins.git.extensions.impl.CleanBeforeCheckout>
           <deleteUntrackedNestedRepositories>false</deleteUntrackedNestedRepositories>
         </hudson.plugins.git.extensions.impl.CleanBeforeCheckout>
   
3. Sooner or later the polling will get blocked again, with symptoms as described
   If it blocks, it seems it stays that way. I have jobs hanging in this state for days (Fortunately no updates on these branches any more)
   
4. The start of the log for good and bad cases is the same, up to and including "> git --version # 'git version 2.30.0'"
   Then the good case does:
    > git fetch --no-tags --force --progress -- ssh://ca_l...y@gerrit-....com:29418/.....LF/phy-tx-lf +refs/heads/*:refs/remotes/origin/* # timeout=10
   
   While the bad case does at the position instead:

1. 
    > git config --get remote.${GerritSlave}.url # timeout=10
   FATAL: hudson.plugins.git.GitException: Failed to fetch from ${GerritSlave}
   
   

1. So it looks, like the actions change - I don't understand why.,
2. ${gerritSlave} is " Repository URL" as described above
   In XML:
     <scm class="hudson.plugins.git.GitSCM" plugin="g...@4.7.1">
       <configVersion>2</configVersion>
       <userRemoteConfigs>
         <hudson.plugins.git.UserRemoteConfig>
           <url>${GerritSlave}</url>
   I just cross-checked (in the XML-file): The var is used  there and only there.

Here is the good case for comparison:

Started on Jun 10, 2021, 5:51:00 PM

Polling SCM changes on master
Using strategy: Default

[poll] Last Built Revision: Revision e23c3fd40dc0790b7cedebefd779a156d581efcb (refs/remotes/origin/branches/maintenance/PSint)

The recommended git tool is: NONE
using credential 50da578b-178c-4cba-b103-91c74f6522ed
 > git rev-parse --resolve-git-dir /var/fpwork/WS/PHY_TX--LF--PSint--buildTrigger/PHY_TX/.git # timeout=10
Fetching changes from the remote Git repositories

 > git config remote.origin.url ssh://ca_lty@gerrit......com:29418/.....phy-tx-lf # timeout=10

Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10

Fetching upstream changes from ssh://ca...hy@gerri.....m:29418//TX-LF/phy-tx-lf

 > git --version # timeout=10
 > git --version # 'git version 2.30.0'

using GIT_ASKPASS to set credentials Jenkins user via 'ca_' (Common account - basically the same as ltedlphy)
 > git fetch --no-tags --force --progress -- ssh://ca_l....@gerrit,,,,,,com:29418/M......y-tx-lf +refs/heads/*:refs/remotes/origin/* # timeout=10
Polling for changes in
 > git rev-parse refs/remotes/origin/branches/maintenance/PSint^{commit} # timeout=10
 > git rev-parse refs/heads/branches/maintenance/PSint^{commit} # timeout=10
Done. Took 0.3 sec
No changes

All ideas welcome

Martin