Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29317

Trigger only autogenerated jobs, not all the jobs for given git url

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Currently gitlab-hook-plugin searches all the jobs that match given repository url.

      In fact, I use multijob configuration like the following:

      MainJob <-- uses git configuration so gitlab-hook-plugin can find & clone & start it
        * Phase 1: compile job <-- uses git to fetch project sources
        * Phase 2: deploy
        * Phase 3: fast tests <-- uses git to fetch tests
      ...
      

      Current gitlab-hook-plugin triggers all the jobs, not just the main one.

      In my installation I've modified plugins\gitlab-hook\WEB-INF\classes\models\use_cases\process_commit.rb }} as follows (added {{ && (project.description.match /#{settings.description}/)) so it triggers only jobs with autogenerated description:

          def get_projects_to_process(details)
            projects = @get_jenkins_projects.matching_uri(details)
            if projects.any?
              if settings.automatic_project_creation?
                projects.select! do |project|
                  project.matches?(details, details.branch, true) && (project.description.match /#{settings.description}/)
                end
                projects << @create_project_for_branch.with(details) if projects.empty?
      

      However, it could make sense to use another approach: get the job by name (hook knows naming convention, so it can get the job) check if that matches. In case the job does not match fallback to the current implementation.

        Attachments

          Issue Links

            Activity

            Hide
            javiplx Javier Palacios added a comment -

            I've no experience with multijob projects, but the plugin does search projects that match the repository url, so if your phase 2 & 3 are triggered, it is because they also clone your git repository.
            Your suggestion is not bad, but it breaks everything else, as every job created manually will stop building on pushes. A configuration item could be added to enable this "only autocreated" behaviour, but I would like to understand your use case before attempting it.

            Regarding your second suggestion, if I understand it well, has the same problem of your current modification, and forces to name the project as the repository, and although it's probably happening with most cases, nothing guarantees that.

            Show
            javiplx Javier Palacios added a comment - I've no experience with multijob projects, but the plugin does search projects that match the repository url, so if your phase 2 & 3 are triggered, it is because they also clone your git repository. Your suggestion is not bad, but it breaks everything else, as every job created manually will stop building on pushes. A configuration item could be added to enable this "only autocreated" behaviour, but I would like to understand your use case before attempting it. Regarding your second suggestion, if I understand it well, has the same problem of your current modification, and forces to name the project as the repository, and although it's probably happening with most cases, nothing guarantees that.
            Hide
            vladimirsitnikov Vladimir Sitnikov added a comment -

            Javier Palacios, my case is as follows:
            1) I have a project at Gitlab. Suppose it is named "ProjectFoo".
            2) I want each branch to spawn a dedicated Jenkins job.
            The intention is as follows: each developer works in his/her own branch, thus having separate Jenkins jobs helps a lot.

            However, the build process itself is not that simple.
            Basically it includes several stages:
            S1) Fetch & compile project
            S2) Prepare database (each test is to be executed in a clean environment, thus each job execution gets its own database schema)
            S3) Deploy the code (run installation SQL scripts, etc, etc)
            S4) Execute fast tests (if they fail, no need to run full suite)
            S5) Execute full suite of tests (here additional DB instances are provisioned to test against different DB versions, etc)
            S6) Uninstall (to test uninstall)

            I think it is somewhat typical scenario for integration tests.

            In order to simplify management of those integration jobs, I use Job DSL plugin.
            In other words, I have a "seed job" that generates all those S1, S2, ..., S3 given the input parameters like "branch name", "DB versions to test", etc.

            The "seed job" has just a single step: "invoke groovy script".
            Job management via groovy scripts turned out to be very efficient.

            So my end-to-end is:

            E1) A developer pushes commit to Gitlab
            E2) Gitlab notifies Jenkins. Here only seed job should be triggered.
            E3) Seed job fetches project sources, figures out the proper set of integration jobs, creates/updates as required, then triggers the actual integration testing.

            The problem is at step E2 my Jenkins server has lots of jobs with given GIT project URL.
            For each project I have exactly one seed job, so I do not want to trigger all of them.

            Does that make sense?

            Show
            vladimirsitnikov Vladimir Sitnikov added a comment - Javier Palacios , my case is as follows: 1) I have a project at Gitlab. Suppose it is named "ProjectFoo". 2) I want each branch to spawn a dedicated Jenkins job. The intention is as follows: each developer works in his/her own branch, thus having separate Jenkins jobs helps a lot. However, the build process itself is not that simple. Basically it includes several stages: S1) Fetch & compile project S2) Prepare database (each test is to be executed in a clean environment, thus each job execution gets its own database schema) S3) Deploy the code (run installation SQL scripts, etc, etc) S4) Execute fast tests (if they fail, no need to run full suite) S5) Execute full suite of tests (here additional DB instances are provisioned to test against different DB versions, etc) S6) Uninstall (to test uninstall) I think it is somewhat typical scenario for integration tests. In order to simplify management of those integration jobs, I use Job DSL plugin. In other words, I have a "seed job" that generates all those S1, S2, ..., S3 given the input parameters like "branch name", "DB versions to test", etc. The "seed job" has just a single step: "invoke groovy script". Job management via groovy scripts turned out to be very efficient. So my end-to-end is: E1) A developer pushes commit to Gitlab E2) Gitlab notifies Jenkins. Here only seed job should be triggered. E3) Seed job fetches project sources, figures out the proper set of integration jobs, creates/updates as required, then triggers the actual integration testing. The problem is at step E2 my Jenkins server has lots of jobs with given GIT project URL. For each project I have exactly one seed job, so I do not want to trigger all of them. Does that make sense?
            Hide
            javiplx Javier Palacios added a comment -

            Yes, it does make a lot of sense, but I'm not sure if it's doable in the context of this plugin. It basically depends on how non-build phases are done. I'll tell you the flow I used for this same scenario, although the config.xml of the jobs is still useful to fully understand your flow.

            In my case, I have a single environment where to deploy artifacts and run the tests. Everything was built as debian packages, so I got an additional element, but I don't believe it is important. The build job was the only one that performed a git clone, and also to deploy the package to a repository which was the same for all the jobs. The deploy & test part was a bit complex, but there was a single set of them, and the connection was done with the standard jenkins triggering. This was a bit o a mesh, as concurrent builds are very dangerous, and in any case the flow graph was a bit ugly, but fortunately every repository used to have only one developer working on it (although there were plenty of repositories, libraries triggering other jobs and stuff like that). And tracking which builds introduced which error was far from being straightforward.
            But I had separated (multiple) phase 1, and single unified later phases.

            If your deploy & test phases must be multiple ones (maybe a full environment is deployed for them), I would suggest to write all those parts into the phase 1 job, to get a single job that could be easily cloned. But when I get a bit of time I will look to multijob, because there should be some link among them within the project configuration.

            In the long term, I've made some attempts towards a real trigger, that should be configured not on global jenkins configuration, but enabled on every job. And at some point in that (not short) path there is probably a solution for your problem, as you could just select your seed project as the only one to be considered.

            Show
            javiplx Javier Palacios added a comment - Yes, it does make a lot of sense, but I'm not sure if it's doable in the context of this plugin. It basically depends on how non-build phases are done. I'll tell you the flow I used for this same scenario, although the config.xml of the jobs is still useful to fully understand your flow. In my case, I have a single environment where to deploy artifacts and run the tests. Everything was built as debian packages, so I got an additional element, but I don't believe it is important. The build job was the only one that performed a git clone, and also to deploy the package to a repository which was the same for all the jobs. The deploy & test part was a bit complex, but there was a single set of them, and the connection was done with the standard jenkins triggering. This was a bit o a mesh, as concurrent builds are very dangerous, and in any case the flow graph was a bit ugly, but fortunately every repository used to have only one developer working on it (although there were plenty of repositories, libraries triggering other jobs and stuff like that). And tracking which builds introduced which error was far from being straightforward. But I had separated (multiple) phase 1, and single unified later phases. If your deploy & test phases must be multiple ones (maybe a full environment is deployed for them), I would suggest to write all those parts into the phase 1 job, to get a single job that could be easily cloned. But when I get a bit of time I will look to multijob, because there should be some link among them within the project configuration. In the long term, I've made some attempts towards a real trigger, that should be configured not on global jenkins configuration, but enabled on every job. And at some point in that (not short) path there is probably a solution for your problem, as you could just select your seed project as the only one to be considered.
            Hide
            vladimirsitnikov Vladimir Sitnikov added a comment - - edited

            I have lots of jobs that access git for different reasons:
            0) "seed job" to get "integration configuration"
            1) compile job to compile the project
            2) test job to get test sources
            etc, etc.

            In other words, what I want from Gitlab-Jenkins integration is that the plugin triggers ONLY the seed job.
            I'm fine to set "trigger job in case of gitlab push" kind of flag on a seed job.

            connection was done with the standard jenkins triggering

            Multijob connection enables to have:
            1) Single status view of each step
            2) Single "stop all" button
            3) Nice "cleanup if anything went wrong" steps
            4) etc, etc.

            However multijob stuff is completely irrelevant for the particular issue. The issue is to have an ability to limit the set of triggered jobs. Why trigger all the jobs with given url?

            Show
            vladimirsitnikov Vladimir Sitnikov added a comment - - edited I have lots of jobs that access git for different reasons: 0) "seed job" to get "integration configuration" 1) compile job to compile the project 2) test job to get test sources etc, etc. In other words, what I want from Gitlab-Jenkins integration is that the plugin triggers ONLY the seed job. I'm fine to set "trigger job in case of gitlab push" kind of flag on a seed job. connection was done with the standard jenkins triggering Multijob connection enables to have: 1) Single status view of each step 2) Single "stop all" button 3) Nice "cleanup if anything went wrong" steps 4) etc, etc. However multijob stuff is completely irrelevant for the particular issue. The issue is to have an ability to limit the set of triggered jobs. Why trigger all the jobs with given url?
            Hide
            javiplx Javier Palacios added a comment -

            However multijob stuff is completely irrelevant for the particular issue. The issue is to have an ability to limit the set of triggered jobs. Why trigger all the jobs with given url?

            That's a matter of design. The plugin is not actually a Trigger object, but a RootAction one. It allows to use the same url on all gitlab repositories, but reduces the number of project specific things that can be done (in particular, most of advanced features of git plugin are not honoured). My plans for trigger alike behaviour will overcome this limitation, but it is not a matter of weeks.

            Show
            javiplx Javier Palacios added a comment - However multijob stuff is completely irrelevant for the particular issue. The issue is to have an ability to limit the set of triggered jobs. Why trigger all the jobs with given url? That's a matter of design. The plugin is not actually a Trigger object, but a RootAction one. It allows to use the same url on all gitlab repositories, but reduces the number of project specific things that can be done (in particular, most of advanced features of git plugin are not honoured). My plans for trigger alike behaviour will overcome this limitation, but it is not a matter of weeks.
            Hide
            javiplx Javier Palacios added a comment -

            It is acceptable to set the webhook url to something like http://jenkins.url/gitlab/build_now/seedproject? I've made a fast check, and apparently the plugin can handle such a url, so it should be hard to just trigger seedproject in that case.

            Show
            javiplx Javier Palacios added a comment - It is acceptable to set the webhook url to something like http://jenkins.url/gitlab/build_now/seedproject? I've made a fast check, and apparently the plugin can handle such a url, so it should be hard to just trigger seedproject in that case.
            Hide
            vladimirsitnikov Vladimir Sitnikov added a comment -

            It is acceptable to set the webhook url to something like http://jenkins.url/gitlab/build_now/seedproject?

            I think so.
            However, in the long run "dumb configuration of webhook" + "smart configuration at jenkins side" might be better.

            so it should be hard to just trigger seedproject in that case.

            You mean "it should not be hard to trigger in that case", don't you?

            Show
            vladimirsitnikov Vladimir Sitnikov added a comment - It is acceptable to set the webhook url to something like http://jenkins.url/gitlab/build_now/seedproject? I think so. However, in the long run "dumb configuration of webhook" + "smart configuration at jenkins side" might be better. so it should be hard to just trigger seedproject in that case. You mean "it should not be hard to trigger in that case", don't you?
            Hide
            javiplx Javier Palacios added a comment -
            Show
            javiplx Javier Palacios added a comment - Right. Was as not hard that you have a preliminar but working version on http://maven.jenkins-ci.org/content/repositories/snapshots/org/jenkins-ci/ruby-plugins/gitlab-hook/1.4.2-SNAPSHOT/

              People

              • Assignee:
                javiplx Javier Palacios
                Reporter:
                vladimirsitnikov Vladimir Sitnikov
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: