Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-1671

Let build sleep until incremental dependencies have been published

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: ci
    • Labels:
      None
    • Similar Issues:
    • Epic Link:

      Description

      It is irritating to have to wait to push a commit in a downstream plugin because the upstream build has not yet completed and published to Incrementals, even though you have built the commit locally. Would be nice if buildPlugin knew how to detect a missing incremental dependency (some kind of failure pattern from mvn validate) and go to sleep for a while before trying again. Ideally it would even be able to locate the upstream build and wait for it to complete, which would perhaps have shorter wait times, and could handle the case that the upstream build in fact failed.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Sketch of crude solution. Needs work.

            Show
            jglick Jesse Glick added a comment - Sketch of crude solution. Needs work.
            Hide
            jglick Jesse Glick added a comment -

            Most of the time, INFRA-1633 is responsible for the upstream build being so slow, though sometimes it is just an actually slow build (like core).

            Show
            jglick Jesse Glick added a comment - Most of the time, INFRA-1633 is responsible for the upstream build being so slow, though sometimes it is just an actually slow build (like core).
            Hide
            jglick Jesse Glick added a comment - - edited

            The maven container agent label would be useful for reducing overhead of the Incremental Deps stage. Still, a full mvn validate would have to download a ton of unrelated dependencies, which seems slow. Would be best to directly look for incremental versions in the POM and check for them in the repository via HTTP HEAD calls. Or could use mvn dependency:resolve -DincludeArtifactIds=… after mvn dependency:collect for more precision at the cost of greater overhead. Or there could be a new mojo in incrementals-tools which does precisely this analysis.

            Show
            jglick Jesse Glick added a comment - - edited The maven container agent label would be useful for reducing overhead of the Incremental Deps stage. Still, a full mvn validate would have to download a ton of unrelated dependencies, which seems slow. Would be best to directly look for incremental versions in the POM and check for them in the repository via HTTP HEAD calls. Or could use mvn dependency:resolve -DincludeArtifactIds=… after mvn dependency:collect for more precision at the cost of greater overhead. Or there could be a new mojo in incrementals-tools which does precisely this analysis.
            Hide
            teilo James Nord added a comment -

            if using withMaven then snapshot triggers could be extended and then a new deployment of a snapshot or incremental could retrigger all jobs that use it (won't stop the first build failure) - but otherwise is a zero executor overhead option

            Show
            teilo James Nord added a comment - if using withMaven then snapshot triggers could be extended and then a new deployment of a snapshot or incremental could retrigger all jobs that use it (won't stop the first build failure) - but otherwise is a zero executor overhead option
            Hide
            jglick Jesse Glick added a comment -

            But you only want to trigger builds of jobs which tried to use it before and failed to download it at the time, which might be tricky. Not the same case as a snapshot trigger, where the snapshot is nondeterministic and you really want to retrigger builds. And does not seem like it belongs in withMaven since that is a general-purpose step whereas this is an RFE specific to the Jenkins org.

            Show
            jglick Jesse Glick added a comment - But you only want to trigger builds of jobs which tried to use it before and failed to download it at the time, which might be tricky. Not the same case as a snapshot trigger, where the snapshot is nondeterministic and you really want to retrigger builds. And does not seem like it belongs in withMaven since that is a general-purpose step whereas this is an RFE specific to the Jenkins org.
            Hide
            teilo James Nord added a comment -

            withMaven is extensible (in the Jenkins way)  (we have a proprietary extension for it) - was just suggesting that there could exist one for just the jenkinsci infra too.

            it should be able to trigger on releases too (basically anything) a incremental is a release version (even if it pretends not to be) and so even though you should not redpoly releases you can, the plugin could just trigger if a release is redeployed.

            Show
            teilo James Nord added a comment - withMaven is extensible (in the Jenkins way)  (we have a proprietary extension for it) - was just suggesting that there could exist one for just the jenkinsci infra too. it should be able to trigger on releases too (basically anything) a incremental is a release version (even if it pretends not to be) and so even though you should not redpoly releases you can, the plugin could just trigger if a release is redeployed.
            Hide
            jglick Jesse Glick added a comment -

            withMaven is extensible

            I see what you are saying.

            a incremental is a release version (even if it pretends not to be)

            It does not pretend not to be!

            you should not [redeploy] releases

            Nor do we, and that is not the issue here. The problem is a build started before the release had been deployed.

            Show
            jglick Jesse Glick added a comment - withMaven is extensible I see what you are saying. a incremental is a release version (even if it pretends not to be) It does not pretend not to be! you should not [redeploy] releases Nor do we, and that is not the issue here. The problem is a build started before the release had been deployed.

              People

              • Assignee:
                Unassigned
                Reporter:
                jglick Jesse Glick
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: