Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56875

Missing parameters due to SECURITY-170

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a pipeline which in Test stage executes a lot of new jobs by using build step (we cannot move the jobs logic into this pipeline due to JENKINS-52391 - we don't want to rebuild 120 jobs because 1 has randomly failed):

      stage('Test') {
          steps {
              script {
                  def parallelTests = [:]
                  def recipes = env.TESTED_RECIPES.split(',')
                  def testSplit = env.TEST_SPLIT != null ? env.TEST_SPLIT.toInteger() : 1
                  def testWebSplit = env.WEBTEST_SPLIT != null ? env.WEBTEST_SPLIT.toInteger() : 1
                  for (def recipe in recipes) {
                      for (def index = 1; index <= testSplit; ++index) {
                          parallelTests["Test JUnit $recipe $index/$testSplit"] = createTestExecutionStage('generic-test-junit', [
                              string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
                              string(name: 'RECIPE', value: recipe),
                              string(name: 'TEST_SPLIT_IDX', value: "${index}"),
                              string(name: 'TEST_SPLIT_MAX', value: "${testSplit}"),
                              string(name: 'PRIORITY', value: params.PRIORITY)
                          ])
                      }
                      for (def index = 1; index <= testWebSplit; ++index) {
                          parallelTests["Test Web JUnit $recipe $index/$testWebSplit"] = createTestExecutionStage('generic-test-junit', [
                              string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
                              string(name: 'RECIPE', value: recipe),
                              booleanParam(name: 'WEB_TESTS', value: true),
                              string(name: 'TEST_SPLIT_IDX', value: "${index}"),
                              string(name: 'TEST_SPLIT_MAX', value: "${testWebSplit}"),
                              string(name: 'PRIORITY', value: params.PRIORITY)
                          ], false)
                      }
                      parallelTests["Test Initialization $recipe"] = createTestExecutionStage('generic-test-initialization', [
                          string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
                          string(name: 'RECIPE', value: recipe),
                          string(name: 'PRIORITY', value: params.PRIORITY)
                      ])
                      parallelTests["Test Server $recipe"] = createTestExecutionStage('generic-test-server', [
                          string(name: 'PIPELINE_ID', value: params.PIPELINE_ID),
                          string(name: 'RECIPE', value: recipe),
                          string(name: 'PRIORITY', value: params.PRIORITY)
                      ])
                  }
                  parallel parallelTests
              }
          }
      }
      def createTestExecutionStage(name, parameters, propagate = true) {
          return {
              build(job: name, parameters: parameters, propagate: propagate)
          }
      }

      Unfortunately, very rarely we see the following entries in logs:

      Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
      Skipped parameter `PIPELINE_ID` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
      Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
      Skipped parameter `RECIPE` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
      Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter
      Skipped parameter `PRIORITY` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.

      It of course breaks the build because parameters are missing. This is how generic-test-junit looks after I clicked Build with Parameters:

      In most cases it looks correct:

      But sometimes it is broken:

      The generic-junit-test job is generated by Job DSL plugin:

      pipelineJob('generic-test-junit') {
          displayName('Generic JUnit Test')
          description('''Multi-line description
      is here''')
      
          logRotator {
              daysToKeep(90)
          }
      
          parameters {
              stringParam('PIPELINE_ID', '', 'Identifier of the pipeline which will be built.')
              stringParam('RECIPE', '', 'Recipe which will be tested.')
              booleanParam('WEB_TESTS', false, 'Defines whether web tests should be executed instead of the non-web tests.')
              stringParam('TEST_SPLIT_IDX', '', 'Index of the test package to be performed.')
              stringParam('TEST_SPLIT_MAX', '', 'Number of the test packages.')
              booleanParam('DEBUG', false, 'Defines whether test target should be executed in debug mode.')
              stringParam('PRIORITY', '', 'Pipeline priority (number from 1 to 5, where 1 is the highest).')
          }
      
          definition {
              cps {
                  script(readFileFromWorkspace('jobs/genericTestJunit.jenkinsfile'))
                  sandbox()
              }
          }
      }
      

      This is the definition of generic-junit-test job (jobs/genericTestJunit.jenkinsfile):

      pipeline {
          agent {
              label 'swarm'
          }
          options {
              ansiColor('xterm')
          }
          stages {
              stage('Test') {
                  steps {
                      cleanWs()
                      sh """#!/bin/bash
                          rsync -az jkmaster:build-scripts .
                          rsync -az jkmaster:resources/${params.PIPELINE_ID}/pipeline.properties .
                      """
                      script {
                          def environmentVariables = ""
                          readProperties(file: 'pipeline.properties')
                              .findAll { !params.containsKey(it.key) }
                              .each { k, v -> environmentVariables = "${environmentVariables}env.${k}='${v}'\n" }
                          writeFile file: 'env.groovy', text: environmentVariables
                      }
                      load 'env.groovy'
                      script {
                          currentBuild.displayName = "#${env.BUILD_NUMBER}-${env.PRODUCT_NAME}-${params.PIPELINE_ID}-${params.RECIPE}-${params.WEB_TESTS ? 'WEB-' : ''}${params.TEST_SPLIT_IDX}-${params.TEST_SPLIT_MAX}"
                      }
      
                      timestamps {
                          lock(resource: null, label: 'rsync', quantity: 1) {
                              sh './build-scripts/synchronize-with-jkmaster.sh'
                          }
                      }
                      wrap([$class: 'LogfilesizecheckerWrapper', failBuild: true, maxLogSize: 0, setOwn: false]) {
                          wrap([$class: 'Xvfb', displayNameOffset: 2]) {
                              timestamps {
                                  timeout(time: 6, unit: 'HOURS') {
                                      timeout(time: 15, activity: true) {
                                          sh "./build-scripts/jobs/test-junit.sh ${params.WEB_TESTS ? 'web' : ''}"
                                      }
                                  }
                              }
                          }
                      }
                      junit 'junit/**/*.xml'
                      archiveArtifacts artifacts: 'junit/**/*.xml'
                  }
              }
          }
          post {
              always {
                  cleanWs()
              }
          }
      }
      

      (I know the code is not very pretty (rsync etc.) - we are migrating very old Jenkins to newer technologies ).

      At this moment we are forced to use this flag -Dhudson.model.ParametersAction.keepUndefinedParameters=true.

        Attachments

          Activity

          Hide
          agabrys Adam Gabryś added a comment -

          We've hit it a few times since April. I compared build.xml of two builds of the same jobs (parameters values are very similar):

          I see only important differences in hudson.model.ParametersAction node.

          Show
          agabrys Adam Gabryś added a comment - We've hit it a few times since April. I compared build.xml of two builds of the same jobs (parameters values are very similar): build-missing.txt build-ok.txt I see only important differences in hudson.model.ParametersAction node.

            People

            • Assignee:
              Unassigned
              Reporter:
              agabrys Adam Gabryś
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: