Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24445

Retriggering builds via the "Manual Trigger" feature of the Gerrit Trigger Plugin causes wrong verification

    Details

    • Similar Issues:

      Description

      There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.

      Problem

      This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination.

      This behaviour leads to inconsistent verification results in Gerrit, because it fails to wait for all tests to complete.

      Replication instructions

      1. Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
      2. Install the gerrit-trigger plugin (arbitrary version, as all are affected)
      3. Increase the executor count of the "master" node to at least 6.
      4. Restart the server and create a connection to a Gerrit service.
      5. Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
        • Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-${JOB_NAME}-${BUILD_NUMBER}" in "${JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
      6. (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".

      After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

      This will cause a build of "Test_1" and "Test_2" to be started.
      Now, while they are running (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
      http://[jenkins]/gerrit_manual_trigger/?

      Search for the changeset and issue a retrigger. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".

      Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

      Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.

      To test this, first delete the two original lock files:

      ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
      ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

      This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.

      Now, to expose the bug, ONLY delete the lockfile for the re-triggered TEST_2 job:
      ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

      This will let Test_2 finish successfully (as Test_2 always succeeds).
      The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.

      If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".

      This behaviour is absolutely wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to inconsistent results in Gerrit.

      Example output in Gerrit, when error occurs

      Here's an excerpt of how this behaviour looks like in Gerrit:

      ##############################################################################
      Jenkins
      Patch Set 1: -Verified
      Build Started Test_2/1/ (1/2)
      -------------------------------------------------------
      Jenkins
      Patch Set 1:
      Build Started Test_1/1/ (2/2)
      -------------------------------------------------------
      Jenkins
      Patch Set 1:
      Build Started Test_2/2/ (2/2)
      -------------------------------------------------------
      Jenkins
      Patch Set 1:
      Build Started Test_1/2/ (2/2)
      -------------------------------------------------------
      Jenkins
      Patch Set 1: Verified+1
      Build Successful 
      Test_2/2/ : SUCCESS
      -------------------------------------------------------
      Jenkins
      Patch Set 1: Verified-1
      Build Failed 
      Test_1/2/ : FAILURE
      ##############################################################################
      

      You can see two bugs outline above at work here:

      1.) The plugin gets confused when counting the number of jobs started
      2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.

      [EDIT: Improved text formatting. Now the report is more than just a wall of text]

        Attachments

          Activity

          mhschroe Martin Schröder created issue -
          mhschroe Martin Schröder made changes -
          Field Original Value New Value
          Environment Operating System independent (verified on Linux as well as Unix)
          Tested in 3 latest Jenkins LTS releases (1.509, 1.532 and 1.554)
          Operating System independent (verified on Linux as well as Windows)
          Tested in 3 latest Jenkins LTS releases (1.509, 1.532 and 1.554)
          mhschroe Martin Schröder made changes -
          Description There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination, and decide to post the Verification or Code Review update even if not all test have passed yet.

          This behaviour leads to inconsistent verification results in Gerrit and can be easily replicated as follows:


          1.) Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).

          2.) Install the gerrit-trigger plugin (arbitrary version, as all are affected)

          3.) Increase the executor count of the "master" node to at least 6.

          4.) Restart the server and create a connection to a Gerrit service.

          5.) Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.

          5.a ) Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-${JOB_NAME}-${BUILD_NUMBER}" in "${JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.

          6.) (Optional) Alter the Test_1 and Test_2 parameters to listen to a specific repository. By default repo and branch are set to "**".



          After this set-up is done, upload a patchset to the Gerrit service in question.

          This will cause a build of "Test_1" and "Test_2" to be started. Now, while they are running (they are wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a retrigger. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the others to finish.


          Now, to expose the bug, ONLY delete the lockfile for the retriggered TEST_2 job:
          ${JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds). The bug that happens now is, that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is absolutely wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to inconsistent results in Gerrit.


          Here's an excerpt of how this behaviour looks like in Gerrit:

          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################


          You can see two bugs at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.
          There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination
          This causes it to decide to post the Verification or Code Review update *even if not all test have passed yet*.

          This behaviour leads to inconsistent verification results in Gerrit and can be easily replicated as follows:


          # Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
          # Install the gerrit-trigger plugin (arbitrary version, as all are affected)
          # Increase the executor count of the "master" node to at least 6.
          # Restart the server and create a connection to a Gerrit service.
          # Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
          #* Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-$\{JOB_NAME}-$\{BUILD_NUMBER}" in "$\{JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
          # (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".
          \\


          After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

          This will cause a build of "Test_1" and "Test_2" to be started.
          Now, *while they are running* (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a *retrigger*. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.


          Now, to expose the bug, *ONLY* delete the lockfile for the re-triggered TEST_2 job:
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds).
          *The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.*

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is _absolutely_ wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to *inconsistent results* in Gerrit.


          Here's an excerpt of how this behaviour looks like in Gerrit:

          {noformat}
          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################
          {noformat}

          You can see two bugs at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.
          mhschroe Martin Schröder made changes -
          Description There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination
          This causes it to decide to post the Verification or Code Review update *even if not all test have passed yet*.

          This behaviour leads to inconsistent verification results in Gerrit and can be easily replicated as follows:


          # Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
          # Install the gerrit-trigger plugin (arbitrary version, as all are affected)
          # Increase the executor count of the "master" node to at least 6.
          # Restart the server and create a connection to a Gerrit service.
          # Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
          #* Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-$\{JOB_NAME}-$\{BUILD_NUMBER}" in "$\{JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
          # (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".
          \\


          After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

          This will cause a build of "Test_1" and "Test_2" to be started.
          Now, *while they are running* (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a *retrigger*. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.


          Now, to expose the bug, *ONLY* delete the lockfile for the re-triggered TEST_2 job:
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds).
          *The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.*

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is _absolutely_ wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to *inconsistent results* in Gerrit.


          Here's an excerpt of how this behaviour looks like in Gerrit:

          {noformat}
          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################
          {noformat}

          You can see two bugs at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.
          There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.


          h4. Problem

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination.

          This behaviour leads to inconsistent verification results in Gerrit, because *it fails wait for all tests to complete.*



          h4. Replication instructions

          # Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
          # Install the gerrit-trigger plugin (arbitrary version, as all are affected)
          # Increase the executor count of the "master" node to at least 6.
          # Restart the server and create a connection to a Gerrit service.
          # Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
          #* Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-$\{JOB_NAME}-$\{BUILD_NUMBER}" in "$\{JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
          # (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".
          \\


          After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

          This will cause a build of "Test_1" and "Test_2" to be started.
          Now, *while they are running* (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a *retrigger*. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.


          Now, to expose the bug, *ONLY* delete the lockfile for the re-triggered TEST_2 job:
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds).
          *The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.*

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is _absolutely_ wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to *inconsistent results* in Gerrit.


          h4. Example output in Gerrit, when error occurs

          Here's an excerpt of how this behaviour looks like in Gerrit:

          {noformat}
          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################
          {noformat}

          You can see two bugs outline above at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.


          [EDIT: Improved text formatting. Now the report is more than just a wall of text]
          mhschroe Martin Schröder made changes -
          Description There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.


          h4. Problem

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination.

          This behaviour leads to inconsistent verification results in Gerrit, because *it fails wait for all tests to complete.*



          h4. Replication instructions

          # Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
          # Install the gerrit-trigger plugin (arbitrary version, as all are affected)
          # Increase the executor count of the "master" node to at least 6.
          # Restart the server and create a connection to a Gerrit service.
          # Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
          #* Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-$\{JOB_NAME}-$\{BUILD_NUMBER}" in "$\{JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
          # (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".
          \\


          After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

          This will cause a build of "Test_1" and "Test_2" to be started.
          Now, *while they are running* (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a *retrigger*. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.


          Now, to expose the bug, *ONLY* delete the lockfile for the re-triggered TEST_2 job:
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds).
          *The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.*

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is _absolutely_ wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to *inconsistent results* in Gerrit.


          h4. Example output in Gerrit, when error occurs

          Here's an excerpt of how this behaviour looks like in Gerrit:

          {noformat}
          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################
          {noformat}

          You can see two bugs outline above at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.


          [EDIT: Improved text formatting. Now the report is more than just a wall of text]
          There is a serious issue with the Gerrit trigger plugin and its "retrigger" functionality.


          h4. Problem

          This issue causes the plugin to get confused about how many and which builds it started for a particular changeset/patchset combination.

          This behaviour leads to inconsistent verification results in Gerrit, because *it fails to wait for all tests to complete.*



          h4. Replication instructions

          # Start a blank, vanilla Jenkins server of an arbitrary version on your local host (the jobs below assume that a BASH shell is available).
          # Install the gerrit-trigger plugin (arbitrary version, as all are affected)
          # Increase the executor count of the "master" node to at least 6.
          # Restart the server and create a connection to a Gerrit service.
          # Shut down the server and extract the two jobs (Test_1 and Test_2) into the "jobs" directory of the Jenkins server.
          #* Test_1 will always fail, Test_2 will always succeed. Both will create a lock-file named "LOCKFILE-$\{JOB_NAME}-$\{BUILD_NUMBER}" in "$\{JENKINS_HOME}/workspace/Central" and wait for it to be deleted, before succeeding or failing.
          # (Optional) Alter the Test_1 and Test_2 Gerrit Trigger settings to listen to a specific repository. By default, both repo and branch are set to "**".
          \\


          After this set-up is done, upload a patchset to the Gerrit service that you configured Jenkins to listen to.

          This will cause a build of "Test_1" and "Test_2" to be started.
          Now, *while they are running* (remember that they will wait until you delete their lock-file), go into the "manual trigger" interface:
          http://[jenkins]/gerrit_manual_trigger/?

          Search for the changeset and issue a *retrigger*. This will cause two additional builds to spawn, one for "Test_1" and one for "Test_2".


          Now that 4 tests are running, the correct behaviour of the Gerrit trigger plugin is to ignore the first two runs that have been started by the "Patchset Created" event.

          Instead, it should wait until BOTH of the retriggered builds have finished and then upload a "Verified -1" message to Gerrit.


          To test this, first delete the two original lock files:

          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_1-1
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-1

          This tests the first half of the problem. This works fine. After both have finished, the Gerrit Trigger plugin will still wait for the other two jobs to finish.


          Now, to expose the bug, *ONLY* delete the lockfile for the re-triggered TEST_2 job:
          $\{JENKINS_HOME}/workspace/Central/LOCKFILE-Test_2-2

          This will let Test_2 finish successfully (as Test_2 always succeeds).
          *The bug that happens now, is that the trigger plugin uploads a "Verified +1" message, DESPITE "Test_1 #2" not yet being finished.*

          If you then also let "Test_1" finish, it uploads yet another verification, this time "Verified -1".


          This behaviour is _absolutely_ wrong, as it sends a verification state without waiting for all jobs to complete, thus leading to *inconsistent results* in Gerrit.


          h4. Example output in Gerrit, when error occurs

          Here's an excerpt of how this behaviour looks like in Gerrit:

          {noformat}
          ##############################################################################
          Jenkins
          Patch Set 1: -Verified
          Build Started Test_2/1/ (1/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/1/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_2/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1:
          Build Started Test_1/2/ (2/2)
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified+1
          Build Successful
          Test_2/2/ : SUCCESS
          -------------------------------------------------------
          Jenkins
          Patch Set 1: Verified-1
          Build Failed
          Test_1/2/ : FAILURE
          ##############################################################################
          {noformat}

          You can see two bugs outline above at work here:

          1.) The plugin gets confused when counting the number of jobs started
          2.) The plugin does not wait for (Test_1 #2) after (Test_2 #2) has finished.


          [EDIT: Improved text formatting. Now the report is more than just a wall of text]
          mhschroe Martin Schröder made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          rin_ne rin_ne made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 157426 ] JNJira + In-Review [ 195705 ]

            People

            • Assignee:
              rsandell rsandell
              Reporter:
              mhschroe Martin Schröder
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: