Blue Ocean: Does not show all steps during a stage with parallel

[Sverre Moe]

I wonder if I have discovered a bug in Blue Ocean or a known limitation:

On a verification stage we loop through all downstream dependencies and rebuild them. To verify that they compile against this new change.

Previously we called build on their respective Jenkins multibranch pipeline project. This took more time than we cared for.

So I rewrote it. Instead of calling build, we now checkout each downstream dependency and simply build them. We do this in parallel on two build nodes.

In Blue Ocean on the Verification stage it only lists the steps for under half of the ~40 downstream dependencies. There is 6 steps per downstream dependency, so the total number of steps on the parallel execution is 240.

The entire output from each parallel execution is massive, ~70000 lines and ~5MB. Perhaps Blue Ocean cannot parse it all. Either to many lines or too many steps.

By rewriting the verification stage like this we cut the execution time in half, from 1h30m to ~45min.

My test pipeline with over 200 steps was all shown in Blue Ocean. So it is not a limitation on number of steps. Could it be that BO didn't manage to parse the massive output? Though it is not that massive when you think about it. 4MB of text should pose no problem parsing.

My shared pipeline script for verification builds:

    void buildDownstream() {
        def stepsForParallel = [:]
        for (def buildHost : parallellRunBuildHosts) {
            final def nodeName = buildEnvironment.toLowerCase() + "-" + buildHost
            stepsForParallel[nodeName] = transformIntoStep(nodeName)
        }

        try {
            buildIn.parallel stepsForParallel
        } catch (e) {
            throw e
        }
    }

    def transformIntoStep(nodeName) {
        return {
            buildIn.node(nodeName) {
                try {
                    final def repositoryResourceLock = nodeName
                    buildIn.lock(repositoryResourceLock) {
                        compileAndPackage(projectName)
                    }
                } catch (e) {
                    throw e
                } finally {
                    buildIn.deleteDir()
                }
            }
        }
    }

    void compileAndPackage(projectName) {
        for (def dependency : downstreamDependencies) {
            buildIn.println "Building " + dependency // STEP 1

            try {
                final def URI = "ssh://g...@git.company.com/projects/${dependency}.git"
                buildIn.checkout(  // STEP 2
                    changelog: false,
                    poll: false,
                    scm: [$class: 'GitSCM',
                        branches: [
                            [name: "origin/" + buildEnvironment]
                        ],
                        doGenerateSubmoduleConfigurations: false,
                        extensions: [
                            [$class: 'CloneOption', depth: 0, noTags: false, reference: '', shallow: false],
                            [$class: 'CleanBeforeCheckout'],
                            [$class: 'RelativeTargetDirectory', relativeTargetDir: dependency]
                        ],
                        submoduleCfg: [],
                        userRemoteConfigs: [
                            [url: URI]
                        ]
                    ]
                )

                buildIn.dir(dependency) {
                    final def tagName = buildIn.sh(returnStdout: true, script: "git describe --tags --abbrev=0 origin/" + buildEnvironment).trim() // STEP 3

                    /* Find and parse RPM spec file for package names. */
                    final def files = buildIn.findFiles(glob: "${dependency}.spec*") // STEP 4
                    final def dependencySpecFile = files.first().name
                    final def dependencySpecFileContent = buildIn.readFile(dependencySpecFile) // STEP 5
                    final def packages = buildIn.utility.parsePackageNames(dependencySpecFileContent, dependency)

                    buildIn.sh """
                        git checkout ${tagName}
                        our-script-archive-dist.sh --output ${dependency}*.tar.gz
                        our-script-build.sh ${dependency}*.tar.gz
                        sudo zypper --no-gpg-checks --non-interactive install *.rpm
                        sudo zypper --non-interactive remove ${packages}
                    """ // STEP 6
                }
            }
        }
    }

[Sverre Moe]

This is still an issue with Blue Ocean v1.8

I am running a verification build of 40 projects in a loop.

The Blue Ocean list of steps cuts off after half of the steps in loop 17.

The Verification stage first finds all the projects running on node("master"), then loops with parallel executions

The classic Stage list shows only the steps in the master execution, but none of the parallel.

The Blue Ocean shows only the steps in the parallel execution, but not the steps running prior on master.