Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44142

Step jobDsl can be used at most once in pipeline with DELETE

    Details

    • Similar Issues:

      Description

      The example for the jobDsl pipeline step shows usage that cannot actually be used:

      https://github.com/jenkinsci/job-dsl-plugin/wiki/User-Power-Moves#use-job-dsl-in-pipeline-scripts

      node {
          jobDsl scriptText: 'job("example-2")'
      
          jobDsl targets: ['jobs/projectA/*.groovy', 'jobs/common.groovy'].join('\n'),
                 removedJobAction: 'DELETE',
                 removedViewAction: 'DELETE',
                 lookupStrategy: 'SEED_JOB',
                 additionalClasspath: ['libA.jar', 'libB.jar'].join('\n')
      }
      

      This will always delete the job "example-2" because the second step DELETEs all unfreferenced items, unless of course one of the scripts also happens to create a job named "example-2".

       

      From my brief look at the source, I don't think this would be easy to implement. However, it would still be incredibly useful to be able to call jobDsl multiple times with different arguments.

       

        Attachments

          Issue Links

            Activity

            Hide
            vqrs Christian V added a comment -

            I've taken another look at this.

            When multiple job-dsl scripts are run, job-dsl will attach multiple "actions" to the build. The current code seems to assume that there is only a single one.

             

            To find the last generated objects, it's probably a simple change, maybe something like this:

             

             

            Set<T> findLastGeneratedObjects() {
                for (Run run = job.lastBuild; run != null; run = run.previousBuild) {
                    // We need to fetch all actions because job-dsl could habe been run multiple times in a single build.
            
                    // Previously, it would use the first build, whose first buildAction's modifiedObjects was set
                    // Now, it chooses the first build, where at least one buildAction's modifiedObjects is set
                    List<B> actions = run.getActions(buildActionClass)
                    Set<T> result = []
                    boolean useThisResult = false
                    for (B action : actions) {
                        if (action.modifiedObjects != null) {
                            useThisResult = true
                            result += action.modifiedObjects
                        }
                    }
            
                    if (useThisResult) {
                        return result
                    }
                }
                []
            }
            

            Further, job-dsl needs to be made aware of previous invocations during the same build run.

             

            Currently, it compares the "new items" against the items items from the first suitable previous run. Here's a simple idea:

            GeneratedItems generatedItems = dslScriptLoader.runScripts(scriptRequests);
            mergeWithCurrentRun(generatedItems, run); // <-- this would be the only change
            Set<GeneratedJob> freshJobs = generatedItems.getJobs();
            Set<GeneratedView> freshViews = generatedItems.getViews();
            Set<GeneratedConfigFile> freshConfigFiles = generatedItems.getConfigFiles();
            Set<GeneratedUserContent> freshUserContents = generatedItems.getUserContents();

             

            One could simply pretend that a subsequent invocation also "generated" the items that were generated previously in the same run.

             

            The last invocation's log output would then summarize everything that job-dsl did.

             

            Finally, one could check whether a build action is already appened to the current run and replace it it instead, instead of unconditionally doing run.addAction:

             

            // Save onto Builder, which belongs to a Project.
            run.addAction(new GeneratedJobsBuildAction(freshJobs, getLookupStrategy()));
            run.addAction(new GeneratedViewsBuildAction(freshViews, getLookupStrategy()));
            run.addAction(new GeneratedConfigFilesBuildAction(freshConfigFiles));
            run.addAction(new GeneratedUserContentsBuildAction(freshUserContents));

             

            This might affect JENKINS-29784

             

            Daniel Spilker Thoughts?

             

            Show
            vqrs Christian V added a comment - I've taken another look at this. When multiple job-dsl scripts are run, job-dsl will attach multiple "actions" to the build. The current code seems to assume that there is only a single one.   To find the last generated objects, it's probably a simple change, maybe something like this:     Set<T> findLastGeneratedObjects() { for (Run run = job.lastBuild; run != null ; run = run.previousBuild) { // We need to fetch all actions because job-dsl could habe been run multiple times in a single build. // Previously, it would use the first build, whose first buildAction's modifiedObjects was set // Now, it chooses the first build, where at least one buildAction's modifiedObjects is set List<B> actions = run.getActions(buildActionClass) Set<T> result = [] boolean useThisResult = false for (B action : actions) { if (action.modifiedObjects != null ) { useThisResult = true result += action.modifiedObjects } } if (useThisResult) { return result } } [] } Further, job-dsl needs to be made aware of previous invocations during the same build run.   Currently, it compares the "new items" against the items items from the first suitable previous run. Here's a simple idea: GeneratedItems generatedItems = dslScriptLoader.runScripts(scriptRequests); mergeWithCurrentRun(generatedItems, run); // <-- this would be the only change Set<GeneratedJob> freshJobs = generatedItems.getJobs(); Set<GeneratedView> freshViews = generatedItems.getViews(); Set<GeneratedConfigFile> freshConfigFiles = generatedItems.getConfigFiles(); Set<GeneratedUserContent> freshUserContents = generatedItems.getUserContents();   One could simply pretend that a subsequent invocation also "generated" the items that were generated previously in the same run.   The last invocation's log output would then summarize everything that job-dsl did.   Finally, one could check whether a build action is already appened to the current run and replace it it instead, instead of unconditionally doing run.addAction:   // Save onto Builder, which belongs to a Project. run.addAction( new GeneratedJobsBuildAction(freshJobs, getLookupStrategy())); run.addAction( new GeneratedViewsBuildAction(freshViews, getLookupStrategy())); run.addAction( new GeneratedConfigFilesBuildAction(freshConfigFiles)); run.addAction( new GeneratedUserContentsBuildAction(freshUserContents));   This might affect JENKINS-29784   Daniel Spilker Thoughts?  
            Hide
            vqrs Christian V added a comment - - edited

            Hmm, Generated*BuildActions also contains the lookup strategy. One would have to have up to two actions per type, one per possible strategy.

             

            Then, JENKINS-29784 should also be solved, hopefully.

            Show
            vqrs Christian V added a comment - - edited Hmm, Generated*BuildActions also contains the lookup strategy. One would have to have up to two actions per type, one per possible strategy.   Then, JENKINS-29784 should also be solved, hopefully.
            Hide
            markus_b_b Markus Baur added a comment -

            This problem took me by surprise and removed history of many jobs. It would be great if it gets addressed soon. It should at least be documented with the DELETE action.

            Show
            markus_b_b Markus Baur added a comment - This problem took me by surprise and removed history of many jobs. It would be great if it gets addressed soon. It should at least be documented with the DELETE action.
            Hide
            lvotypkova Lucie Votypkova added a comment -

            Hello,

            it is connected with problem with multiple job dsl action - action with generated objects is added for every job dsl build step, so there is the same number of actions as job-dsl steps. If option DELETE is checked, it compares generated items of current build (if there is unreferenced job/view/config - jobs/views ... which were created by this seed job, but are not included in generated objects by job dsl step). But this comparison contains only  generated items from one action not from the all actions from all job dsl steps because getAction(Class<T> returns only the first hit.  Including them into one action (so do not create action for every job dsl step, but include generated objects into existing action) should fix the problem, but only in case that you use DELETE option only in your last build job dsl step (because generation of all jobs/views/config is not finished if there any job dsl step reminds to execute).

            my pull request is there https://github.com/jenkinsci/job-dsl-plugin/pull/1190, you can check it or comment.

             

             

            Show
            lvotypkova Lucie Votypkova added a comment - Hello, it is connected with problem with multiple job dsl action - action with generated objects is added for every job dsl build step, so there is the same number of actions as job-dsl steps. If option DELETE is checked, it compares generated items of current build (if there is unreferenced job/view/config - jobs/views ... which were created by this seed job, but are not included in generated objects by job dsl step). But this comparison contains only  generated items from one action not from the all actions from all job dsl steps because getAction(Class<T> returns only the first hit.  Including them into one action (so do not create action for every job dsl step, but include generated objects into existing action) should fix the problem, but only in case that you use DELETE option only in your last build job dsl step (because generation of all jobs/views/config is not finished if there any job dsl step reminds to execute). my pull request is there https://github.com/jenkinsci/job-dsl-plugin/pull/1190,  you can check it or comment.    
            Hide
            daniel_c_686 Daniel Carrington added a comment -

            I have used the Job DSL plugin in production for over a year with the DISABLE action in place and I hit a similar issue when using multiple job dsl build steps in a single pipeline. The fix for this issue changed the behavior of the job dsl plugin when using multiple job dsl build steps in parallel. Is that (parallel job dsl build steps) a use case which is interesting to anyone else?

            (The new behavior of the Job DSL plugin ended up disabling many of my jobs. I spent a day with a mostly broken environment because I had to track this down.)

             Based on the discussion here, it seems like the correct thing to do is to use the IGNORE behavior on all uses of the job dsl build step except the very last one, which should not run in parallel with any other job dsl build step. Looking at the available hook points in hudson.model.Run and hudson.model.Job, I think that there isn't a great way to solve this. We could register a new hudson.model.listeners.RunListener that disables or deletes jobs after a successful job completes (and runs Job DSL build steps that DISABLE or DELETE jobs). Has that idea already been considered?

            Show
            daniel_c_686 Daniel Carrington added a comment - I have used the Job DSL plugin in production for over a year with the DISABLE action in place and I hit a similar issue when using multiple job dsl build steps in a single pipeline. The fix for this issue changed the behavior of the job dsl plugin when using multiple job dsl build steps in parallel. Is that (parallel job dsl build steps) a use case which is interesting to anyone else? (The new behavior of the Job DSL plugin ended up disabling many of my jobs. I spent a day with a mostly broken environment because I had to track this down.)  Based on the discussion here, it seems like the correct thing to do is to use the IGNORE behavior on all uses of the job dsl build step except the very last one, which should not run in parallel with any other job dsl build step. Looking at the available hook points in hudson.model.Run and hudson.model.Job, I think that there isn't a great way to solve this. We could register a new hudson.model.listeners.RunListener that disables or deletes jobs after a successful job completes (and runs Job DSL build steps that DISABLE or DELETE jobs). Has that idea already been considered?
            Hide
            daspilker Daniel Spilker added a comment -

            I reopened this issue because I agree that it's still too easy to misconfigure the Job DSL build step.

            I added some hints to the documentation, see https://github.com/jenkinsci/job-dsl-plugin/pull/1193.

            Using a RunListener has the drawback that the build status can not be changed anymore. When deleting or disabling a job fails (e.g. missing permissions), the error can be logged to the console output, but the job status will be "successful". So it's very likely that the problem will not be detected.

            Can someone explain the use case for running multiple Job DSL build steps? And that is the use case for running scripts in parallel?

            Show
            daspilker Daniel Spilker added a comment - I reopened this issue because I agree that it's still too easy to misconfigure the Job DSL build step. I added some hints to the documentation, see https://github.com/jenkinsci/job-dsl-plugin/pull/1193 . Using a RunListener has the drawback that the build status can not be changed anymore. When deleting or disabling a job fails (e.g. missing permissions), the error can be logged to the console output, but the job status will be "successful". So it's very likely that the problem will not be detected. Can someone explain the use case for running multiple Job DSL build steps? And that is the use case for running scripts in parallel?
            Hide
            daniel_c_686 Daniel Carrington added a comment -

            One use case for running multiple Job DSL build steps would be to accumulate jobs defined in multiple repositories. Consider a pipeline that does:

              stage ("repo a") {
                  checkout "our/repo-a"
                  jobDsl "jobs/**/*.groovy"
              }
              stage ("repo b") {
                  checkout "our/repo-b"
                  jobDsl "jobs/**/*.groovy"
              }
            

            I run multiple Job DSL build steps in order to speed up the Job DSL configuration seed job. I have about 520 active jobs on my Jenkins instance, and they are all managed in a single repository. They are, thankfully, somewhat organized by project codename. My Job DSL pipeline reads something like:

            List<String> folders = ['project-a', 'project-b', 'project-c', ...] // roughly 10-20 projects
            
            def startupTasks = [:]
            
            startupTasks["Folder Setup"] = {
              node("job_dsl") { stage("Folder Setup) {
                checkout scm
                sh "generate_some_files_read_by_job_dsl_code"
                jobDsl targets: "jobs/folders.groovy"
              } }
            }
            
            startupTasks["Self-Check"] = {
              // Checks that all .groovy files in jobs/** are consumed by exactly one jobDsl build step
            }
            
            parallel startupTasks
            
            def allProjectJobs = [:]
            for (int i = 0; i < folders.length; i++) {
                String projname = folders[i]
                allProjectJobs["Project $projname"] = {
                    node("job_dsl") { stage("Project $projname") {
                        checkout scm
                        // Other versions of this have used stash&unstash, see commentary below
                        sh "generate_some_files_read_by_job_dsl_code"
                        // Classes in src/ are used to implement templates that set up each project similarly.
                        jobDsl targets: "jobs/$projname/*.groovy", additionalClasspath: "src/"
                    } }
                }
            }
            parallel allProjectJobs
            

            I use parallel Job DSL execution because I see slow Job DSL performance on running a lot of Job DSL code to generate this many jobs.
            I have a total of 81 files in my jobs/ folder, and some representative runtimes that I see for each group of files, in the Job DSL build step only, is:

            N files in project total filesize (bytes) runtime
            16 54863 2m 55s
            14 28715 1m 55s
            10 21716 2m 5s
            9 13768 1m 55s
            6 12051 1m 45s
            5 7262 1m 15s
            4 20754 1m 0s
            4 6394 1m 20s
            3 2425 1m 15s
            3 1592 1m 0s
            3 1569 1m 0s
            2 1871 1m 0s
            1 607 0m 45s
            1 275 0m 45s

            Running these in parallel instead of in series or all in one build step provides us with better visibility and ability to debug issues in the Job DSL (can hit an exception in each of these branches and resolve them all in one change), and it reduces the execution time from ~20minutes to ~5minutes. I have repeatedly told my team that we need to set up a unit test environment for Job DSL, but as we are not Java developers, this is a bit of a task. It will probably fall to me as I have some familiarity with Java, but little familiarity with modern Java tooling. I am pretty much the only developer on my team who has used Java (because of some college class experience on my part!), but as we are a hardware R&D lab working on ASIC design, most of my colleagues don't know how to use modern Java tooling at all.

            I suspect that my Job DSL runtimes would be faster if I used a node with a ping of less than 100ms to the Jenkins master for running Job DSL, but for *various and sundry reasons*, it is the way it is. (Does the Job DSL fetch a lot of classes via the remoting channel to run? I am not sure how to debug this performance issue.)

            In any case, I had to separate each project into its own freestyle project to run the Job DSL in parallel like this with the unreferenced job action set to 'DISABLE'. The changes that I made to the pipeline were to create a downstream freestyle job for each sub-folder of jobs/ and to build that job instead of running the jobDsl step directly. As a result, I cannot use stash/unstash to generate the files that my Job DSL templates read with e.g. readFileFromWorkspace and I have to pay the time cost of generating them in each workspace used by the pipeline.

            Show
            daniel_c_686 Daniel Carrington added a comment - One use case for running multiple Job DSL build steps would be to accumulate jobs defined in multiple repositories. Consider a pipeline that does: stage ( "repo a" ) { checkout "our/repo-a" jobDsl "jobs /**/ *.groovy" } stage ( "repo b" ) { checkout "our/repo-b" jobDsl "jobs /**/ *.groovy" } I run multiple Job DSL build steps in order to speed up the Job DSL configuration seed job. I have about 520 active jobs on my Jenkins instance, and they are all managed in a single repository. They are, thankfully, somewhat organized by project codename. My Job DSL pipeline reads something like: List< String > folders = [ 'project-a' , 'project-b' , 'project-c' , ...] // roughly 10-20 projects def startupTasks = [:] startupTasks[ "Folder Setup" ] = { node( "job_dsl" ) { stage("Folder Setup) { checkout scm sh "generate_some_files_read_by_job_dsl_code" jobDsl targets: "jobs/folders.groovy" } } } startupTasks[ "Self-Check" ] = { // Checks that all .groovy files in jobs/** are consumed by exactly one jobDsl build step } parallel startupTasks def allProjectJobs = [:] for ( int i = 0; i < folders.length; i++) { String projname = folders[i] allProjectJobs[ "Project $projname" ] = { node( "job_dsl" ) { stage( "Project $projname" ) { checkout scm // Other versions of this have used stash&unstash, see commentary below sh "generate_some_files_read_by_job_dsl_code" // Classes in src/ are used to implement templates that set up each project similarly. jobDsl targets: "jobs/$projname/*.groovy" , additionalClasspath: "src/" } } } } parallel allProjectJobs I use parallel Job DSL execution because I see slow Job DSL performance on running a lot of Job DSL code to generate this many jobs. I have a total of 81 files in my jobs/ folder, and some representative runtimes that I see for each group of files, in the Job DSL build step only, is: N files in project total filesize (bytes) runtime 16 54863 2m 55s 14 28715 1m 55s 10 21716 2m 5s 9 13768 1m 55s 6 12051 1m 45s 5 7262 1m 15s 4 20754 1m 0s 4 6394 1m 20s 3 2425 1m 15s 3 1592 1m 0s 3 1569 1m 0s 2 1871 1m 0s 1 607 0m 45s 1 275 0m 45s Running these in parallel instead of in series or all in one build step provides us with better visibility and ability to debug issues in the Job DSL (can hit an exception in each of these branches and resolve them all in one change), and it reduces the execution time from ~20minutes to ~5minutes. I have repeatedly told my team that we need to set up a unit test environment for Job DSL, but as we are not Java developers, this is a bit of a task. It will probably fall to me as I have some familiarity with Java, but little familiarity with modern Java tooling. I am pretty much the only developer on my team who has used Java (because of some college class experience on my part!), but as we are a hardware R&D lab working on ASIC design, most of my colleagues don't know how to use modern Java tooling at all. I suspect that my Job DSL runtimes would be faster if I used a node with a ping of less than 100ms to the Jenkins master for running Job DSL, but for * various and sundry reasons *, it is the way it is. (Does the Job DSL fetch a lot of classes via the remoting channel to run? I am not sure how to debug this performance issue.) In any case, I had to separate each project into its own freestyle project to run the Job DSL in parallel like this with the unreferenced job action set to 'DISABLE'. The changes that I made to the pipeline were to create a downstream freestyle job for each sub-folder of jobs/ and to build that job instead of running the jobDsl step directly. As a result, I cannot use stash/unstash to generate the files that my Job DSL templates read with e.g. readFileFromWorkspace and I have to pay the time cost of generating them in each workspace used by the pipeline.
            Hide
            daspilker Daniel Spilker added a comment -

            Daniel Carrington: Please open a new feature request for running Job DSL scripts in parallel. Use your last comment as description. Running multiple Job DSL steps in currently not supported. Let's use that feature request to find out what's feasible and there the Job DSL plugin can help.

            Show
            daspilker Daniel Spilker added a comment - Daniel Carrington : Please open a new feature request for running Job DSL scripts in parallel. Use your last comment as description. Running multiple Job DSL steps in currently not supported. Let's use that feature request to find out what's feasible and there the Job DSL plugin can help.

              People

              • Assignee:
                daspilker Daniel Spilker
                Reporter:
                vqrs Christian V
              • Votes:
                5 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: