Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50703

Jenkins slave jar retained context when Jenkins master was offline

    Details

    • Similar Issues:

      Description

      The issue was first encountered as a set of jobs that were behaving as if they were Allowing Concurrent Jobs even though they were not set to allow this.

       

        Even with a single run occurring, starting from an empty workspace (The workspace on the slave was completely wiped) it was creating jobname@2 to run the job.  We cleared this situation up by forcing a restart of the slave.jar process running on the jenkins slave by using Disconnect, confirming that it had shut down, and then reconnected and then backtraced what had actually happened.

      Earlier in the week, the Jenkins master had been suddenly restarted while active jobs on some slaves were processing.  These jobs were left in an undetermined state on the slaves (the master decided they had failed) and were the 1 job that the slaves were seeing.  The jenkins master, having no knowledge of these ongoing jobs, would fire off a new job which the slave would treat as a concurrent job.  This state persisted long after the job would have normally finished (Days later).

       

      Repro step:
      1) Have one jenkins master, 1 slave.
      2) Start long enough job on slave.
      3) Restart master, interrupting connectivity.
      4) Verify job has failed on master.
      5) Run job again.  @2 should occur.

      Jenkins version 2.107.1 (had just been upgraded a little before this).

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Seems to be just another issue with workspace reference cache

          Show
          oleg_nenashev Oleg Nenashev added a comment - Seems to be just another issue with workspace reference cache
          Hide
          jthompson Jeff Thompson added a comment -

          Can you provide additional information on reproducing this? Anything relevant about configuration, plugins, environment, or how this is observed. Logs would certainly be useful.

          My initial, simple attempts at reproduction were unsuccessful.

          Show
          jthompson Jeff Thompson added a comment - Can you provide additional information on reproducing this? Anything relevant about configuration, plugins, environment, or how this is observed. Logs would certainly be useful. My initial, simple attempts at reproduction were unsuccessful.
          Hide
          jthompson Jeff Thompson added a comment -

          Closing for lack of response providing sufficient reproduction or diagnostic information.

          Show
          jthompson Jeff Thompson added a comment - Closing for lack of response providing sufficient reproduction or diagnostic information.
          Hide
          pbloom Philip Bloom added a comment -

          Sorry about that.  I actually only just saw a request for such.  I'm not sure what would be relevant.  This is 6 months back at this point, so we aren't on the same version of Jenkins anymore.  What logs would we be sending?  It mostly is a:

          Running on build-1-008 in /home/jenkins/jenkins-slave/workspace/<Jobname>@2 . 

          In the logs when it happened.  At this point, we just keep an assert in our base framework for "@" being part of a workspace name, since we never want that to be the case.

          Show
          pbloom Philip Bloom added a comment - Sorry about that.  I actually only just saw a request for such.  I'm not sure what would be relevant.  This is 6 months back at this point, so we aren't on the same version of Jenkins anymore.  What logs would we be sending?  It mostly is a: Running on  build-1-008  in /home/jenkins/jenkins-slave/workspace/<Jobname>@2 .  In the logs when it happened.  At this point, we just keep an assert in our base framework for "@" being part of a workspace name, since we never want that to be the case.
          Hide
          jthompson Jeff Thompson added a comment -

          Any information that would be useful for me or someone else to reproduce it would be helpful. With the provided information I wasn't able to reproduce. Unfortunately, I really couldn't say what would be needed to reproduce.

          If it no longer occurs on current versions, then it's probably not worth bothering with.

          Without actionable information I don't think it makes much sense to leave this report open. If you have info you can provide, please feel free to re-open.

          Show
          jthompson Jeff Thompson added a comment - Any information that would be useful for me or someone else to reproduce it would be helpful. With the provided information I wasn't able to reproduce. Unfortunately, I really couldn't say what would be needed to reproduce. If it no longer occurs on current versions, then it's probably not worth bothering with. Without actionable information I don't think it makes much sense to leave this report open. If you have info you can provide, please feel free to re-open.

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              pbloom Philip Bloom
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: