Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31152

2.0: Pipeline as code in front & center

    XMLWordPrintable

    Details

    • Epic Name:
      2.0: Pipeline as Code
    • Similar Issues:

      Description

      Problem definition

      The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

      Proposal

      Introduce a new subsystem in Jenkins that:

      • lets you design a whole pipeline, not just a single linear set of tasks
      • stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
      • makes it automatic to set up new pipelines when Jenkinsfile is added
      • differentiates multiple branches in the same repository

      This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

      Overview

      (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

      1. In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
      2. In one of the repository in the chosen org, create a Jenkinsfile that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See workflow tutorial for details.
      3. Jenkins will automatically recognize this branch and create appropriate jobs by itself.
      4. Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
      5. When the branch is destroyed in the repository or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins as well automatically.

      In this use, there'll be a lot of Jenkinsfile in various branches & repositories. To keep them DRY, various mechanisms will be provided to promote reuse of workflow scripts, such as this.

      For more details, see Jesse Glick's slides and video recording that talks about this feature (in particular, where the demo starts.) There's also docker-based demo that you can play with on your own.

      Contents

      • Thus, this feature should be made available out of the box by default, unless users opt out (2.0 Out-of-the-box experience)
      • Provide better workflow visualization out of the box (JENKINS-31154)
      • Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
      • Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

      Internals

      • The execution engine of this is Workflow plugin (see JENKINS-26129)
      • Multi-branch workflow project type defines a new kind of folder that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroyed in the repository.
      • Organization folder that defines a new kind of folder that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
      • Branch API plugin and SCM API plugin defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
      • CloudBees GitHub Branch Source Plugin implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.

      Impact

      Open Questions

        Attachments

          Issue Links

            Activity

            Hide
            cobexer Ing. Christoph Obexer added a comment -

            It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts!
            Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide.
            See https://wiki.eclipse.org/Orion.

            Show
            cobexer Ing. Christoph Obexer added a comment - It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts! Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide. See https://wiki.eclipse.org/Orion .
            Hide
            jglick Jesse Glick added a comment -

            Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.

            Show
            jglick Jesse Glick added a comment - Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            war/src/main/js/api/plugins.js
            http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa
            Log:
            JENKINS-31152 Add Pipeline plugins, select by default

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa Log: JENKINS-31152 Add Pipeline plugins, select by default
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            war/src/main/js/api/plugins.js
            http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f
            Log:
            Merge pull request #2053 from daniel-beck/JENKINS-31152

            JENKINS-31152 Add Pipeline plugins, select by default

            Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f Log: Merge pull request #2053 from daniel-beck/ JENKINS-31152 JENKINS-31152 Add Pipeline plugins, select by default Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            2.0 has shipped, so closing this epic.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - 2.0 has shipped, so closing this epic.

              People

              • Assignee:
                marcelfrei_ublox Marcel Frei
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                25 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: