Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46453

matrix job with dynamic axis aborting builds

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: matrix-project-plugin
    • Labels:
      None
    • Environment:
      jenkins 2.72
      matrix project plugin 1.11
      installed from debian package - running directly
      java-8-openjdk
      13 executors labelled for the failing job, label is 'builder'
    • Similar Issues:

      Description

      When building a matrix job with a large amount of dynamic axis, some of the axis builds will suddenly abort, the log output shows

      11:10:23 kernel-single-defconfig-builder » multi_v7_defconfig+CONFIG_RANDOMIZE_BASE=y,builder appears to be cancelled
      11:10:23 kernel-single-defconfig-builder » multi_v7_defconfig+CONFIG_RANDOMIZE_BASE=y,builder completed with result ABORTED

      It only seems to happen to builds that are a "new" axis or havent been built recently (possibly old one cleaned out by discarding old builds), clicking on the hyperlink in the log gives a 404, it's like the new axis doesn't get setup correctly. I had a look on the filesystem and the aborted jobs have no data in the jobs axis-label directories:

       

      root@machine:/var/lib/jenkins/jobs/kernel-single-defconfig-builder/configurations/axis-defconfig# ls -alh *RANDOMIZE*/axis-label/
       defconfig+CONFIG_RANDOMIZE_BASE=y/axis-label/:
       total 12K
       drwxr-xr-x 3 jenkins jenkins 4.0K Jul 19 13:54 .
       drwxr-xr-x 3 jenkins jenkins 4.0K Mar 27 15:21 ..
       drwxr-xr-x 3 jenkins jenkins 4.0K Aug 25 11:08 builder
      multi_v7_defconfig+CONFIG_RANDOMIZE_BASE=y/axis-label/:
       total 8.0K
       drwxr-xr-x 2 jenkins jenkins 4.0K Aug 24 18:03 .
       drwxr-xr-x 3 jenkins jenkins 4.0K Aug 24 18:03 ..
      multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y+CONFIG_RANDOMIZE_BASE=y/axis-label/:
       total 8.0K
       drwxr-xr-x 2 jenkins jenkins 4.0K Aug 24 18:03 .
       drwxr-xr-x 3 jenkins jenkins 4.0K Aug 24 18:03 ..
      omap2plus_defconfig+CONFIG_RANDOMIZE_BASE=y/axis-label/:
       total 8.0K
       drwxr-xr-x 2 jenkins jenkins 4.0K Aug 24 18:03 .
       drwxr-xr-x 3 jenkins jenkins 4.0K Aug 24 18:03 ..
      

       

      The label used is 'builder' and the first axis that contains this directory built fine but the others were aborted.

      Seems to be a repeat of Jenkins-13972 that was closed as fixed without really any code changes or resolution.
      I also found this google presentation which would seem to discuss the same issue:
      https://docs.google.com/presentation/d/1ybtB-Bhkb4c3dhb5ZMArr4prtEZ-pjLqH9Vk7yhdZTg/edit#slide=id.g2c21d8fdc_00

       

      Note that I have a 'staging' setup of the same plugins and jenkins version doing the same build which does not show this problem, but only has 1 builder so they are not executed concurrently.

       

        Attachments

          Issue Links

            Activity

            Hide
            pjdarton pjdarton added a comment -

            We're also seeing this same issue on Jenkins version 2.150.2 ... and I suspect we've been experiencing it (but not been aware that this was the cause) for some time.

            Show
            pjdarton pjdarton added a comment - We're also seeing this same issue on Jenkins version 2.150.2 ... and I suspect we've been experiencing it (but not been aware that this was the cause) for some time.

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                mattface Matt Hart
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: