Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-1571

Automatically deploy from buildPlugin

    Details

    • Similar Issues:
    • Epic Link:

      Attachments

        Issue Links

          Activity

          Hide
          jglick Jesse Glick added a comment -

          Pending JENKINS-45970, probably best to do so only if (env.CHANGE_FORK == null).

          Show
          jglick Jesse Glick added a comment - Pending  JENKINS-45970 , probably best to do so only if (env.CHANGE_FORK == null) .
          Hide
          jglick Jesse Glick added a comment -

          Or perhaps only deploy from origin branch builds? CHANGE_ID == null

          This would have a side benefit that we would not be deploying PR merge commits, which are not useful to consume anyway. Currently https://ci.jenkins.io/job/Plugins/ appears to be configured to suppress the PR head builds for origin branches also filed as PRs.

          Show
          jglick Jesse Glick added a comment - Or perhaps only deploy from origin branch builds? CHANGE_ID == null This would have a side benefit that we would not be deploying PR merge commits, which are not useful to consume anyway. Currently https://ci.jenkins.io/job/Plugins/ appears to be configured to suppress the PR head builds for origin branches also filed as PRs.
          Hide
          jglick Jesse Glick added a comment -

          As per this guide I am prototyping whether a trusted downstream job could instead do the deployment after sanity-checking the GAV. Using this tip I was able to run

          (cd ~/.m2/repository && jar cvf0M /tmp/mrp-test.zip test/*/*-rc*.342bdb1bd972/*-rc*.342bdb1bd972*)
          status=$(curl --silent --output /dev/stderr --write-out '%{http_code}' -i -u jglick:… -T /tmp/mrp-test.zip -H 'X-Explode-Archive: true' -H 'X-Explode-Archive-Atomic: true' https://repo.jenkins-ci.org/incrementals/)
          

          which did deploy as expected.

          Show
          jglick Jesse Glick added a comment - As per this guide I am prototyping whether a trusted downstream job could instead do the deployment after sanity-checking the GAV. Using this tip I was able to run (cd ~/.m2/repository && jar cvf0M /tmp/mrp-test.zip test/*/*-rc*.342bdb1bd972/*-rc*.342bdb1bd972*) status=$(curl --silent --output /dev/stderr --write-out '%{http_code}' -i -u jglick:… -T /tmp/mrp-test.zip -H 'X-Explode-Archive: true' -H 'X-Explode-Archive-Atomic: true' https://repo.jenkins-ci.org/incrementals/) which did deploy as expected.
          Hide
          jglick Jesse Glick added a comment -
          Jenkins.instance.getAllItems(hudson.model.Job).each {p ->
            def b = p.lastSuccessfulBuild;
            if (b != null) {
              def ra = b.getAction(jenkins.scm.api.SCMRevisionAction);
              if (ra != null) {
                def rev = ra.revision;
                if (rev instanceof jenkins.plugins.git.AbstractGitSCMSource$SCMRevisionImpl) {
          	println("$b: $rev.hash");
                } else if (rev instanceof org.jenkinsci.plugins.github_branch_source.PullRequestSCMRevision) {
                  println("$b: $rev.pullHash");
                }
              }
            }
          }; null
          

          looks up the commit associated with potential upstream builds. In the case of PR merge builds, the pullHash might differ from the git HEAD in the checkout (if the PR is not up to date against its target branch), in which case the latter should not be deployed.

          Show
          jglick Jesse Glick added a comment - Jenkins.instance.getAllItems(hudson.model.Job).each {p -> def b = p.lastSuccessfulBuild; if (b != null ) { def ra = b.getAction(jenkins.scm.api.SCMRevisionAction); if (ra != null ) { def rev = ra.revision; if (rev instanceof jenkins.plugins.git.AbstractGitSCMSource$SCMRevisionImpl) { println( "$b: $rev.hash" ); } else if (rev instanceof org.jenkinsci.plugins.github_branch_source.PullRequestSCMRevision) { println( "$b: $rev.pullHash" ); } } } }; null looks up the commit associated with potential upstream builds. In the case of PR merge builds, the pullHash might differ from the git HEAD in the checkout (if the PR is not up to date against its target branch), in which case the latter should not be deployed.
          Hide
          jglick Jesse Glick added a comment -

          Better to use REST than unsandboxed code, though: JENKINS-50777

          Show
          jglick Jesse Glick added a comment - Better to use REST than unsandboxed code, though: JENKINS-50777
          Hide
          jglick Jesse Glick added a comment -

          Working on a tool to gather artifacts from an upstream build and inspect them for possible deployment.

          Show
          jglick Jesse Glick added a comment - Working on a tool to gather artifacts from an upstream build and inspect them for possible deployment.
          Hide
          jglick Jesse Glick added a comment -

          Have what seems to be a working PoC. Now need to get it properly hosted, a downstream job set up for it, and a patch to buildPlugin merged to call it.

          Show
          jglick Jesse Glick added a comment - Have what seems to be a working PoC. Now need to get it properly hosted, a downstream job set up for it, and a patch to buildPlugin merged to call it.
          Hide
          jglick Jesse Glick added a comment -

          As noted in JENKINS-50803, the tool cannot trust -Dset.changelist to block malicious changelist clashes; it also needs to verify that the 12-digit commit hash is in fact unique on GitHub amongst all forks. For example,

          curl -u …:… -i https://api.github.com/repos/jenkinsci/jenkins/commits/08dd9
          

          works (even though this commit 08dd94c20c7b2beac4149a2ae16029e2a0ce7b2b is in a fork and not origin), but

          curl -u …:… -i https://api.github.com/repos/jenkinsci/jenkins/commits/08dd
          

          gives a 404 as there are three commits with that prefix.

          Show
          jglick Jesse Glick added a comment - As noted in JENKINS-50803 , the tool cannot trust -Dset.changelist to block malicious changelist clashes; it also needs to verify that the 12-digit commit hash is in fact unique on GitHub amongst all forks. For example, curl -u …:… -i https://api.github.com/repos/jenkinsci/jenkins/commits/08dd9 works (even though this commit 08dd94c20c7b2beac4149a2ae16029e2a0ce7b2b is in a fork and not origin), but curl -u …:… -i https://api.github.com/repos/jenkinsci/jenkins/commits/08dd gives a 404 as there are three commits with that prefix.
          Hide
          jglick Jesse Glick added a comment -

          Also need to finish the RPU branch so that only authorized artifacts will be deployed, which in turn requires patching repository-permissions-updater to include a new field.

          Show
          jglick Jesse Glick added a comment - Also need to finish the RPU branch so that only authorized artifacts will be deployed, which in turn requires patching repository-permissions-updater to include a new field.
          Hide
          jglick Jesse Glick added a comment -

          Commit uniqueness check done.

          Show
          jglick Jesse Glick added a comment - Commit uniqueness check done.
          Hide
          jglick Jesse Glick added a comment - - edited

          R. Tyler Croy suggests an Azure function rather than a downstream job.

          The blocking problem at the moment is the lack of any REST API on ci.jenkins.io.

          Show
          jglick Jesse Glick added a comment - - edited R. Tyler Croy suggests an Azure function rather than a downstream job. The blocking problem at the moment is the lack of any REST API on ci.jenkins.io.
          Hide
          jglick Jesse Glick added a comment -

          The tool is now doing the Artifactory upload.

          A milestone: this PR build succeeded for the base Linux branch (I am ignoring other test failures), consuming an incremental version of Jenkins core and an incremental version of structs which was published as a submodule of the structs-plugin reactor, in turn consuming an incremental version of git; manifest:

          Jenkins-Version: 2.118-rc15700.24aa7b764a86
          Plugin-Dependencies: workflow-api:2.22,workflow-step-api:2.12,script-s
           ecurity:1.30,structs:1.15-rc128.6f68601f55b6
          
          Show
          jglick Jesse Glick added a comment - The tool is now doing the Artifactory upload. A milestone: this PR build succeeded for the base Linux branch (I am ignoring other test failures), consuming an incremental version of Jenkins core and an incremental version of structs which was published as a submodule of the structs-plugin reactor, in turn consuming an incremental version of git ; manifest: Jenkins-Version: 2.118-rc15700.24aa7b764a86 Plugin-Dependencies: workflow-api:2.22,workflow-step-api:2.12,script-s ecurity:1.30,structs:1.15-rc128.6f68601f55b6
          Hide
          jglick Jesse Glick added a comment -

          For retrieving permissions, I had initially assumed the tool would just work with a repository checkout, but R. Tyler Croy had noted that we have GitHub API access and could perhaps just fetch YAML directly from the repository. I tried this API but it is indeed limited to 1000 files (currently we have 1722 permission files), and anyway it just gives metadata so you would need separate API calls for every file, which would be impossibly slow.

          It would be possible to search to cut back on traffic, but this seems clumsy and unreliable.

          I propose instead to patch RPU to generate an index of paths as an artifact, which would be a smaller download than all the YAML (currently 432Kb).

          Show
          jglick Jesse Glick added a comment - For retrieving permissions, I had initially assumed the tool would just work with a repository checkout, but R. Tyler Croy had noted that we have GitHub API access and could perhaps just fetch YAML directly from the repository. I tried this API but it is indeed limited to 1000 files (currently we have 1722 permission files), and anyway it just gives metadata so you would need separate API calls for every file, which would be impossibly slow. It would be possible to search to cut back on traffic, but this seems clumsy and unreliable. I propose instead to patch RPU to generate an index of paths as an artifact, which would be a smaller download than all the YAML (currently 432Kb).
          Hide
          jglick Jesse Glick added a comment -

          Completed. Now R. Tyler Croy promises to reimplement it in Node to add to community-functions.

          Show
          jglick Jesse Glick added a comment - Completed. Now R. Tyler Croy promises to reimplement it in Node to add to community-functions .
          Hide
          jglick Jesse Glick added a comment -

          With https://github.com/jenkins-infra/community-functions/pull/10 I think we have a fully implemented function.

          Now on to get my RPU PR merged, and a buildPlugin patch filed, and some resolution to the lack of API access to ci.jenkins.io.

          Show
          jglick Jesse Glick added a comment - With https://github.com/jenkins-infra/community-functions/pull/10 I think we have a fully implemented function. Now on to get my RPU PR merged, and a buildPlugin patch filed, and some resolution to the lack of API access to ci.jenkins.io.
          Hide
          jglick Jesse Glick added a comment -

          I think I have now filed all the PRs I can—would be In Review if such a state existed in this project.

          Show
          jglick Jesse Glick added a comment - I think I have now filed all the PRs I can—would be In Review if such a state existed in this project.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          dist/profile/manifests/buildmaster.pp
          spec/server/jenkins_master/jenkins_master_spec.rb
          http://jenkins-ci.org/commit/jenkins-infra/f3b2324a628e1054778785bca517d35b5e6a29cc
          Log:
          INFRA-1571 Permit API requests when there is an Authorization header.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: dist/profile/manifests/buildmaster.pp spec/server/jenkins_master/jenkins_master_spec.rb http://jenkins-ci.org/commit/jenkins-infra/f3b2324a628e1054778785bca517d35b5e6a29cc Log: INFRA-1571 Permit API requests when there is an Authorization header.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: R. Tyler Croy
          Path:
          dist/profile/manifests/buildmaster.pp
          spec/server/jenkins_master/jenkins_master_spec.rb
          http://jenkins-ci.org/commit/jenkins-infra/e9cf6a830fcfa1b7651072861faacf465dbd87e5
          Log:
          Merge pull request #1006 from jglick/authenticated-api-INFRA-1571

          INFRA-1571 Permit API requests when there is an Authorization header

          Compare: https://github.com/jenkins-infra/jenkins-infra/compare/d5676df20284...e9cf6a830fcf

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: dist/profile/manifests/buildmaster.pp spec/server/jenkins_master/jenkins_master_spec.rb http://jenkins-ci.org/commit/jenkins-infra/e9cf6a830fcfa1b7651072861faacf465dbd87e5 Log: Merge pull request #1006 from jglick/authenticated-api- INFRA-1571 INFRA-1571 Permit API requests when there is an Authorization header Compare: https://github.com/jenkins-infra/jenkins-infra/compare/d5676df20284...e9cf6a830fcf
          Hide
          jglick Jesse Glick added a comment -

          https://github.com/jenkinsci/jenkins/pull/3394/commits/f80a804725d4263c77fa7546c4459bb1267faa24 was successfully deployed by CI (in ~74s), so I think this is working.

          Show
          jglick Jesse Glick added a comment - https://github.com/jenkinsci/jenkins/pull/3394/commits/f80a804725d4263c77fa7546c4459bb1267faa24 was successfully deployed by CI (in ~74s), so I think this is working.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          incrementals.md
          http://jenkins-ci.org/commit/pom/75a965f453392f6ca075856554a2fd123d2a3ea2
          Log:
          INFRA-1571 Deployed automatically.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: incrementals.md http://jenkins-ci.org/commit/pom/75a965f453392f6ca075856554a2fd123d2a3ea2 Log: INFRA-1571 Deployed automatically.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          .gitignore
          .mvn/extensions.xml
          .mvn/maven.config
          Jenkinsfile
          cli/pom.xml
          core/pom.xml
          pom.xml
          test/pom.xml
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/3fb17fb0bbb824a6c929af3809022c280cb6270a
          Log:
          JENKINS-50692 INFRA-1571 #3394: JEP-305 Incrementals in core

          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: .gitignore .mvn/extensions.xml .mvn/maven.config Jenkinsfile cli/pom.xml core/pom.xml pom.xml test/pom.xml war/pom.xml http://jenkins-ci.org/commit/jenkins/3fb17fb0bbb824a6c929af3809022c280cb6270a Log: JENKINS-50692 INFRA-1571 #3394: JEP-305 Incrementals in core * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          content/blog/2018/05/2018-05-15-incremental-deployment.adoc
          content/images/post-images/2018-05-15/incrementals-status.png
          http://jenkins-ci.org/commit/jenkins.io/6b8114150a6793c0bc13c137b3c7daf9449fcf50
          Log:
          INFRA-1571 Blog post on automated deployment to JEP-305 “Incrementals”.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: content/blog/2018/05/2018-05-15-incremental-deployment.adoc content/images/post-images/2018-05-15/incrementals-status.png http://jenkins-ci.org/commit/jenkins.io/6b8114150a6793c0bc13c137b3c7daf9449fcf50 Log: INFRA-1571 Blog post on automated deployment to JEP-305 “Incrementals”.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: R. Tyler Croy
          Path:
          content/blog/2018/05/2018-05-15-incremental-deployment.adoc
          content/images/post-images/2018-05-15/incrementals-status.png
          http://jenkins-ci.org/commit/jenkins.io/46ccf4d4ed03ca8fe4d9f7ad5e27ca2414b060bf
          Log:
          Merge pull request #1542 from jglick/blog-INFRA-1571

          INFRA-1571 Blog post on automated deployment to JEP-305 “Incrementals”

          Compare: https://github.com/jenkins-infra/jenkins.io/compare/fa4248165b59...46ccf4d4ed03
          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/blog/2018/05/2018-05-15-incremental-deployment.adoc content/images/post-images/2018-05-15/incrementals-status.png http://jenkins-ci.org/commit/jenkins.io/46ccf4d4ed03ca8fe4d9f7ad5e27ca2414b060bf Log: Merge pull request #1542 from jglick/blog- INFRA-1571 INFRA-1571 Blog post on automated deployment to JEP-305 “Incrementals” Compare: https://github.com/jenkins-infra/jenkins.io/compare/fa4248165b59...46ccf4d4ed03 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: suren
          Path:
          .gitignore
          .mvn/extensions.xml
          .mvn/jvm.config
          .mvn/maven.config
          http://jenkins-ci.org/commit/hugo-plugin/291f718729dc9b0cb03ba9d37adac4a913f35c81
          Log:
          JENKINS-50692 INFRA-1571 JEP-305 Incrementals in core

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: suren Path: .gitignore .mvn/extensions.xml .mvn/jvm.config .mvn/maven.config http://jenkins-ci.org/commit/hugo-plugin/291f718729dc9b0cb03ba9d37adac4a913f35c81 Log: JENKINS-50692 INFRA-1571 JEP-305 Incrementals in core
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: suren
          Path:
          .gitignore
          .mvn/extensions.xml
          .mvn/jvm.config
          .mvn/maven.config
          pom.xml
          http://jenkins-ci.org/commit/hugo-plugin/6f28b60a3ed6845475ba33a00b19cdc3936b5b54
          Log:
          Merge pull request #3 from LinuxSuRen/master

          JENKINS-50692 INFRA-1571 JEP-305 Incrementals

          Compare: https://github.com/jenkinsci/hugo-plugin/compare/9e12a55dd246...6f28b60a3ed6
          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: suren Path: .gitignore .mvn/extensions.xml .mvn/jvm.config .mvn/maven.config pom.xml http://jenkins-ci.org/commit/hugo-plugin/6f28b60a3ed6845475ba33a00b19cdc3936b5b54 Log: Merge pull request #3 from LinuxSuRen/master JENKINS-50692 INFRA-1571 JEP-305 Incrementals Compare: https://github.com/jenkinsci/hugo-plugin/compare/9e12a55dd246...6f28b60a3ed6 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.

            People

            • Assignee:
              jglick Jesse Glick
              Reporter:
              jglick Jesse Glick
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: