[JIRA] (JENKINS-60948) Branch indexing is ~5x -> 20x slower in v0.4.1 vs v0.3.3

[johannes.carlsson@acconeer.com (JIRA)]

Johannes Carlsson created an issue

Jenkins / JENKINS-60948 Branch indexing is ~5x -> 20x slower in v0.4.1 vs v0.3.3 Issue Type: Bug Assignee: Luca Domenico Milanesio Attachments: 0.3.3-gerrit-httpd_log, 0.4.1-gerrit-httpd_log Components: gerrit-code-review-plugin Created: 2020-02-03 10:25 Environment: Jenkins ver. 2.204.2 Gerrit Code Review (3.1.2) Gerrit Code Review Plugin v0.4.1 Priority: Major Reporter: Johannes Carlsson When upgrading from v0.3.3 the branch indexing takes much longer. In our production environment the branch indexing went from ~20 s to ~60 s, on my local test environment (Gerrit & Jenkins running locally on same laptop) the branch indexing goes from ~10 s to ~ 2 min 30 s. All installed plugins are of the latest version. Only changing the version of the "Gerrit Code Review" plugin between 0.3.3 and 0.4.1 will cause the difference in indexing time (I tried locally several times). The project have ~70 open changes and ~35 branches visible to Jenkins. The "checks" plugin is not used or installed on Gerrit. When looking into the gerrit httpd log it is clear that version 0.4.1 (see 0.4.1-gerrit-httpd_log) creates a lot more traffic during branch indexing than version 0.3.3 (see 0.3.3-gerrit-httpd_log). This can also be seen in "Scan Multibranch Pipeline Log" from Jenkins (I have not attached these logs as they contain sensitive information). In v0.4.1 an extra "git fetch" is performed for each change: v0.3.3: ... No changes detected: 95/2795/5 (still at 37b3f66b4fda4cb5c764d89d997ea7fe9fd9c784) Skipping branches refs/notes/review Checking change 34/2834/1 ‘Jenkinsfile’ found Met criteria ... v0.4.1: ... > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://jcdell:8443/a/sw-main > git config remote.origin.url https://jcdell:8443/a/sw-main # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials Jenkins@Gerrit > git fetch --tags --progress -- origin refs/changes/95/2795/5:refs/remotes/origin/95/2795/5 # timeout=10 No changes detected: 95/2795/5 (still at 37b3f66b4fda4cb5c764d89d997ea7fe9fd9c784) ‘Jenkinsfile’ found Met criteria ...   Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[luca.milanesio@gmail.com (JIRA)]

Luca Domenico Milanesio commented on JENKINS-60948

Re: Branch indexing is ~5x -> 20x slower in v0.4.1 vs v0.3.3 Johannes Carlsson thanks for reporting the issue. In a nutshell, the changes with 0.4.1 were composed of: Fetch of individual refs instead of refs/changes/* Use of the Checks API to filter out the changes that do not need validation With the combination of 1. and 2. the branch reindexing time would be a lot faster in 0.4.1 compared to 0.3.3. However, in your case, you do not have 2. because you are not using the Checks plugin. Also, by looking at your Gerrit httpd_log, it looks like the repository doesn't have a lot of refs and thus you do not see a benefit but, instead, a performance regression. When the repository has lots of refs (hundreds of thousands), fetching refs/changes/* (what the v0.3.3 was doing) may take tens of minutes and, in extreme cases, also up to 1h. That is why I changed the approach and did the fetch of the individual refs, which is slower in your case. Let me see if I can "convince" Jenkins that the subsequent "fetch" isn't needed  and thus cut the tail of git-upload-pack after the query changes API.

Add Comment