| We have a pipeline which in Test stage executes a lot of new jobs by using build step (we cannot move the jobs logic into this pipeline due to JENKINS-52391 - we don't want to rebuild 120 jobs because 1 has randomly failed):
stage('Test') {
steps {
script {
def parallelTests = [:]
def recipes = env.TESTED_RECIPES.split(',')
def testSplit = env.TEST_SPLIT != null ? env.TEST_SPLIT.toInteger() : 1
def testWebSplit = env.WEBTEST_SPLIT != null ? env.WEBTEST_SPLIT.toInteger() : 1
for (def recipe in recipes) {
for (def index = 1; index <= testSplit; ++index) {
parallelTests["Test JUnit $recipe $index/$testSplit"] = createTestExecutionStage('generic-test-junit', [
string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
string(name: 'RECIPE', value: recipe),
string(name: 'TEST_SPLIT_IDX', value: "${index}"),
string(name: 'TEST_SPLIT_MAX', value: "${testSplit}"),
string(name: 'PRIORITY', value: params.PRIORITY)
])
}
for (def index = 1; index <= testWebSplit; ++index) {
parallelTests["Test Web JUnit $recipe $index/$testWebSplit"] = createTestExecutionStage('generic-test-junit', [
string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
string(name: 'RECIPE', value: recipe),
booleanParam(name: 'WEB_TESTS', value: true),
string(name: 'TEST_SPLIT_IDX', value: "${index}"),
string(name: 'TEST_SPLIT_MAX', value: "${testWebSplit}"),
string(name: 'PRIORITY', value: params.PRIORITY)
], false)
}
parallelTests["Test Initialization $recipe"] = createTestExecutionStage('generic-test-initialization', [
string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
string(name: 'RECIPE', value: recipe),
string(name: 'PRIORITY', value: params.PRIORITY)
])
parallelTests["Test Server $recipe"] = createTestExecutionStage('generic-test-server', [
string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
string(name: 'RECIPE', value: recipe),
string(name: 'PRIORITY', value: params.PRIORITY)
])
}
parallel parallelTests
}
}
}
def createTestExecutionStage(name, parameters, propagate = true) {
return {
build(job: name, parameters: parameters, propagate: propagate)
}
}
Unfortunately, very rarely we see the following entries in logs:
Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
Skipped parameter `PIPELINE_ID` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
Skipped parameter `RECIPE` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
Skipped parameter `PRIORITY` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
It of course breaks the build, because parameters are missing. This is how generic-test-junit looks after I clicked Build with Parameters:
 In most cases it looks correct:
 But sometimes it is broken:
 The generic-junit-test job is generated by Job DSL plugin:
pipelineJob('generic-test-junit') {
displayName('Generic JUnit Test')
description('''Multi-line description
is here''')
logRotator {
daysToKeep(90)
}
parameters {
stringParam('PIPELINE_ID', '', 'Identifier of the pipeline which will be built.')
stringParam('RECIPE', '', 'Recipe which will be tested.')
booleanParam('WEB_TESTS', false, 'Defines whether web tests should be executed instead of the non-web tests.')
stringParam('TEST_SPLIT_IDX', '', 'Index of the test package to be performed.')
stringParam('TEST_SPLIT_MAX', '', 'Number of the test packages.')
booleanParam('DEBUG', false, 'Defines whether test target should be executed in debug mode.')
stringParam('PRIORITY', '', 'Pipeline priority (number from 1 to 5, where 1 is the highest).')
}
definition {
cps {
script(readFileFromWorkspace('jobs/genericTestJunit.jenkinsfile'))
sandbox()
}
}
}
This is the definition of generic-junit-test job (genericTestJunit.jenkinsfile:
pipeline {
agent {
label 'swarm'
}
options {
ansiColor('xterm')
}
stages {
stage('Test') {
steps {
cleanWs()
sh """#!/bin/bash
rsync -az jkmaster:build-scripts .
rsync -az jkmaster:resources/${params.PIPELINE_ID}/pipeline.properties .
"""
script {
def environmentVariables = ""
readProperties(file: 'pipeline.properties')
.findAll { !params.containsKey(it.key) }
.each { k, v -> environmentVariables = "${environmentVariables}env.${k}='${v}'\n" }
writeFile file: 'env.groovy', text: environmentVariables
}
load 'env.groovy'
script {
currentBuild.displayName = "#${env.BUILD_NUMBER}-${env.PRODUCT_NAME}-${params.PIPELINE_ID}-${params.RECIPE}-${params.WEB_TESTS ? 'WEB-' : ''}${params.TEST_SPLIT_IDX}-${params.TEST_SPLIT_MAX}"
}
timestamps {
lock(resource: null, label: 'rsync', quantity: 1) {
sh './build-scripts/synchronize-with-jkmaster.sh'
}
}
wrap([$class: 'LogfilesizecheckerWrapper', failBuild: true, maxLogSize: 0, setOwn: false]) {
wrap([$class: 'Xvfb', displayNameOffset: 2]) {
timestamps {
timeout(time: 6, unit: 'HOURS') {
timeout(time: 15, activity: true) {
sh "./build-scripts/jobs/test-junit.sh ${params.WEB_TESTS ? 'web' : ''}"
}
}
}
}
}
junit 'junit/**/*.xml'
archiveArtifacts artifacts: 'junit/**/*.xml'
}
}
}
post {
always {
cleanWs()
}
}
}
(I know the code is not very pretty (rsync etc - we are migrating very old Jenkins to newer technologies). At this moment we are forced to use this flag -Dhudson.model.ParametersAction.safeParameters=true. |
