Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43693

multibranchPipelineJob overrides old branch indexing sources

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: job-dsl-plugin
    • Labels:
      None
    • Environment:
      Debian 8
      Jenkins 2.46.1 (as service)

      Plugins (all newest stable):
      Job DSL 1.61
      Pipeline: Multibranch 2.14
    • Similar Issues:
    • Released As:
      job-dsl 1.75

      Description

      When running the multibranchPipelineJob step it seems like it overrides the sources of a Multibranch Pipeline Job instead of just updating it. After running the step, even though nothing has changed the output of the branch indexing will become:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
      Branch reopened: master (b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      Scheduled build for branch: master
      

       

      To test this, I have installed a brand new Jenkins with the Pipeline plugins and Job DSL plugin (all as the stable version) and made a freestyle job, with a Job DSL step that has the following script:

      multibranchPipelineJob('job-dsl-plugin_multibranch') {
        branchSources {
          git { 
            remote('https://github.com/jenkinsci/job-dsl-plugin.git')
          }
        }
      }
      

       

      After running the Job DSL step and afterwards running branch indexing a "master" job is created with the following log message from branch indexing:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Scheduled build for branch: master
      

       

      When running branch indexing again, following log message are shown as expected:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: master (still at b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      

       

      However, if I run the Job DSL again and run branch indexing, the first log message (with takeover and so on) are shown. This is not what I would have expected, I would have expected something like a "no changes detected" message.

       

        Attachments

          Issue Links

            Activity

            anigif Anders Kielsholm created issue -
            anigif Anders Kielsholm made changes -
            Field Original Value New Value
            Description When running the multibranchPipelineJob step it seems like it overrides the sources of a Multibranch Pipeline Job instead of just updating it. After running the step, even though nothing has changed the output of the branch indexing will become:

             
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
            Branch reopened: master (b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
            Scheduled build for branch: master
            {code}
             

             

            To test this, I have installed a brand new Jenkins with the Pipeline plugins and Job DSL plugin (all as the stable version) and made a freestyle job, with a Job DSL step that has the following script:

             
            {code:java}
            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                git {
                  remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                }
              }
            }
            {code}
             

             

            After running the Job DSL step and afterwards running branch indexing a "master" job is created with the following log message from branch indexing:
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            Scheduled build for branch: master
            {code}
            When running branch indexing again, following log message are shown as expected:

             

             
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: master (still at b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
            {code}
            However, if I run the Job DSL again and run branch indexing, the first log message (with takeover and so on) are shown. This is not what I would have expected, I would have expected something like a "no changes detected" message.

             
            When running the multibranchPipelineJob step it seems like it overrides the sources of a Multibranch Pipeline Job instead of just updating it. After running the step, even though nothing has changed the output of the branch indexing will become:
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
            Branch reopened: master (b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
            Scheduled build for branch: master
            {code}
             

            To test this, I have installed a brand new Jenkins with the Pipeline plugins and Job DSL plugin (all as the stable version) and made a freestyle job, with a Job DSL step that has the following script:
            {code:java}
            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                git {
                  remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                }
              }
            }
            {code}
             

            After running the Job DSL step and afterwards running branch indexing a "master" job is created with the following log message from branch indexing:
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            Scheduled build for branch: master
            {code}
             

            When running branch indexing again, following log message are shown as expected:
            {code:java}
            Checking branch master
                  ‘Jenkinsfile’ found
            Met criteria
            No changes detected: master (still at b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
            {code}
             

            However, if I run the Job DSL again and run branch indexing, the first log message (with takeover and so on) are shown. This is not what I would have expected, I would have expected something like a "no changes detected" message.

             
            Hide
            daspilker Daniel Spilker added a comment - - edited

            The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job.

            You can use the Automatically Generated DSL to set the ID:

            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                branchSource {
                  source {
                    git {
                      id('must-be-unique')
                      remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                      credentialsId(null)
                      remoteName('origin')
                      rawRefSpecs('+refs/heads/*:refs/remotes/origin/*')
                      includes('*')
                      excludes('')
                      ignoreOnPushNotifications(false)
                    }
                  }
                }
              }
            }
            
            Show
            daspilker Daniel Spilker added a comment - - edited The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job. You can use the Automatically Generated DSL to set the ID: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { branchSource { source { git { id( 'must-be-unique' ) remote( 'https: //github.com/jenkinsci/job-dsl-plugin.git' ) credentialsId( null ) remoteName( 'origin' ) rawRefSpecs( '+refs/heads/*:refs/remotes/origin/*' ) includes( '*' ) excludes('') ignoreOnPushNotifications( false ) } } } } }
            Hide
            rocktavious kyle rockman added a comment -

            HUZZAH!  I've been fighting with this bug for a month or two now at work. Daniel Spilker Thank you that fixed the issue!

            Show
            rocktavious kyle rockman added a comment - HUZZAH!  I've been fighting with this bug for a month or two now at work. Daniel Spilker Thank you that fixed the issue!
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Spilker
            Path:
            docs/Home.md
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
            job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
            http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca
            Log:
            enhanced support for the Pipeline Multibranch Plugin

            [FIXES JENKINS-43693]

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca Log: enhanced support for the Pipeline Multibranch Plugin [FIXES JENKINS-43693]
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Spilker
            Path:
            docs/Home.md
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
            job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
            http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130
            Log:
            Merge branch 'JENKINS-43693'

            Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130 Log: Merge branch ' JENKINS-43693 ' Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c
            daspilker Daniel Spilker made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Hide
            chancez Chance Zibolski added a comment -

            I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?

            Show
            chancez Chance Zibolski added a comment - I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?
            Hide
            widgetpl Mike Dziedziela added a comment - - edited

            I face the same issue.

            Jenkins - v2.60.1

            Job DSL - v1.64

            Pipeline Multibranch - v2.16

            I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it.

            Checking branch develop
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » develop by source #1 from source that no longer exists
            Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471)
            Scheduled build for branch: develop
            
            Checking branch test
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » test by source #1 from source that no longer exists
            Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e)
            Scheduled build for branch: test
            
            Checking branch master
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » master by source #1 from source that no longer exists
            Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5)
            Scheduled build for branch: master

            let me know if you need more information.

            EDIT:

            Jenkins is integrated with GitLab v. 10.0.3 CE

            Show
            widgetpl Mike Dziedziela added a comment - - edited I face the same issue. Jenkins - v2.60.1 Job DSL - v1.64 Pipeline Multibranch - v2.16 I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it. Checking branch develop ‘Jenkinsfile’ found Met criteria Takeover for example_project » develop by source #1 from source that no longer exists Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471) Scheduled build for branch: develop Checking branch test ‘Jenkinsfile’ found Met criteria Takeover for example_project » test by source #1 from source that no longer exists Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e) Scheduled build for branch: test Checking branch master ‘Jenkinsfile’ found Met criteria Takeover for example_project » master by source #1 from source that no longer exists Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5) Scheduled build for branch: master let me know if you need more information. EDIT: Jenkins is integrated with GitLab v. 10.0.3 CE
            Hide
            kizzie Kat added a comment -

            So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6

            (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)

            Show
            kizzie Kat added a comment - So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6 (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)
            Hide
            duemir Denys Digtiar added a comment -

            I have created a fresh environment from 2.121 and installing:

            • Pipeline: Multibranch 2.18
            • Github Branch Source Plugin 2.3.4
            • Job DSL 1.69

            I was able to reproduce the issue using the steps in the description. I can still see

            Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
            Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020)
            Scheduled build for branch: master
            

            after I re-run the Seed Job.

            Show
            duemir Denys Digtiar added a comment - I have created a fresh environment from 2.121 and installing: Pipeline: Multibranch 2.18 Github Branch Source Plugin 2.3.4 Job DSL 1.69 I was able to reproduce the issue using the steps in the description. I can still see Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020) Scheduled build for branch: master after I re-run the Seed Job.
            duemir Denys Digtiar made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            Hide
            duemir Denys Digtiar added a comment - - edited

            Sorry, Daniel Spilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this:

            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                def repo = 'https://github.com/jenkinsci/job-dsl-plugin.git'
                git {
                  id = UUID.nameUUIDFromBytes(repo.getBytes())
                  remote(repo)
                }
              }
            }
            Show
            duemir Denys Digtiar added a comment - - edited Sorry, Daniel Spilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { def repo = 'https: //github.com/jenkinsci/job-dsl-plugin.git' git { id = UUID.nameUUIDFromBytes(repo.getBytes()) remote(repo) } } }
            Hide
            widgetpl Mike Dziedziela added a comment -

            Denys Digtiar yes, it worked for me and I use repository name for id.

            Show
            widgetpl Mike Dziedziela added a comment - Denys Digtiar yes, it worked for me and I use repository name for id.
            duemir Denys Digtiar made changes -
            Status Reopened [ 4 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Hide
            namelessspear Vyacheslav Spiridonov added a comment -

            This is how to do it for bitbucket-branch-source:

              branchSources {
              def repo = "https://bitbucket.com/..."
                branchSource {
                  source { 
                    bitbucket {
                      id(UUID.nameUUIDFromBytes(repo.getBytes()).toString())
                      serverUrl("...")
                      repository("...")
                      repoOwner("...")
                     ...
                    }
                  }
                }
              }
            Show
            namelessspear Vyacheslav Spiridonov added a comment - This is how to do it for bitbucket-branch-source: branchSources { def repo = "https: //bitbucket.com/..." branchSource { source { bitbucket { id(UUID.nameUUIDFromBytes(repo.getBytes()).toString()) serverUrl( "..." ) repository( "..." ) repoOwner( "..." ) ... } } } }
            Hide
            rkivisto Ray Kivisto added a comment -

            Should this documentation be updated to include setting the id?

            https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob

            Show
            rkivisto Ray Kivisto added a comment - Should this documentation be updated to include setting the id ? https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob
            Hide
            jknurek J Knurek added a comment -

            I'm with Ray Kivisto on this.

            I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected. 

            The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 

            Show
            jknurek J Knurek added a comment - I'm with Ray Kivisto on this. I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected.  The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 
            Hide
            rkivisto Ray Kivisto added a comment -

            Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.

            Show
            rkivisto Ray Kivisto added a comment - Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.
            rkivisto Ray Kivisto made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            Hide
            daspilker Daniel Spilker added a comment -

            PR for improving the documentation and removing support for randomly generated identifiers:

            https://github.com/jenkinsci/job-dsl-plugin/pull/1187

            To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.

            Show
            daspilker Daniel Spilker added a comment - PR for improving the documentation and removing support for randomly generated identifiers: https://github.com/jenkinsci/job-dsl-plugin/pull/1187 To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.
            daspilker Daniel Spilker made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            daspilker Daniel Spilker made changes -
            Link This issue is duplicated by JENKINS-57235 [ JENKINS-57235 ]
            Hide
            renescheibe René Scheibe added a comment - - edited

            Daniel Spilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?

            Show
            renescheibe René Scheibe added a comment - - edited Daniel Spilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?
            Hide
            daspilker Daniel Spilker added a comment -

            René Scheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems.

            I'll close this issue since there no more known problems.

            Show
            daspilker Daniel Spilker added a comment - René Scheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems. I'll close this issue since there no more known problems.
            daspilker Daniel Spilker made changes -
            Status In Progress [ 3 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Released As job-dsl 1.75
            Hide
            renescheibe René Scheibe added a comment -

            Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced.

            That's for example all the sources under branchSources - branchSource which are for example:

            • bitbucket
            • git
            • github
            • mercurialSCMSource
            • subversionSCMSource

            Do you think these can also be covered somehow?

            Show
            renescheibe René Scheibe added a comment - Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced. That's for example all the sources under branchSources - branchSource which are for example: bitbucket git github mercurialSCMSource subversionSCMSource Do you think these can also be covered somehow?

              People

              • Assignee:
                daspilker Daniel Spilker
                Reporter:
                anigif Anders Kielsholm
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: