Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32917

Locked resource in use by multiple jobs

    Details

    • Similar Issues:

      Description

      I frequently see two jobs using the same resource, whereas only one job has actually locked the resource. These jobs all have a SCM configured.

      I suspect that the locking occurs too late (only after the check out). So, while the first job is updating the workspace the resource is not locked yet and the other job thinks he can use that same resource.

        Attachments

          Activity

          Hide
          jwstric2 Jonathan Strickland added a comment - - edited

          Running Core Jenkins 1.651
          Externeral Dispatcher: 1.1.0

          Just started hitting this issue just recently as we are expanding our pipelines and resources. From reviewing the code (very briefly), I don't see anywhere that would prevent multiple threads obtaining the resource lock at the same time. Following the code from https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/ExternalResourceQueueTaskDispatcher.java#L113 to https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/utils/resourcemanagers/ResourceMonitorExternalResourceManager.java#L203 there appear to be multiple places that could have mutual exclusion problems. Perhaps the job dispatcher in Jenkins runs in a single thread; but I would have to review the architecture to be certain and from what is being observed this may not be the case.

          The really unfortunate part of this and any mutual exclusion problem is I turned on full logging to observe the behavior and it went away. I'm going to fork this off and see if we can propose a possible fix.

          Unless, the Queue handler in Jenkins is itself running in a single thread.

          Show
          jwstric2 Jonathan Strickland added a comment - - edited Running Core Jenkins 1.651 Externeral Dispatcher: 1.1.0 Just started hitting this issue just recently as we are expanding our pipelines and resources. From reviewing the code (very briefly), I don't see anywhere that would prevent multiple threads obtaining the resource lock at the same time. Following the code from https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/ExternalResourceQueueTaskDispatcher.java#L113 to https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/utils/resourcemanagers/ResourceMonitorExternalResourceManager.java#L203 there appear to be multiple places that could have mutual exclusion problems. Perhaps the job dispatcher in Jenkins runs in a single thread; but I would have to review the architecture to be certain and from what is being observed this may not be the case. The really unfortunate part of this and any mutual exclusion problem is I turned on full logging to observe the behavior and it went away. I'm going to fork this off and see if we can propose a possible fix. Unless, the Queue handler in Jenkins is itself running in a single thread.
          Hide
          terrapin Jan Hollevoet added a comment -

          I solved this by increasing the reservation time in the External Resource Dispatcher Configuration and limit the checkout time (shorter than the reservation time of the jobs that ). But this is only workaround of course.

          I guess that the actually locking of the resources happens too late: now it happens in the prebuild method of JobProperty whereas it should happen earlier (in the onStarted method of RunListener).

           

           

          Show
          terrapin Jan Hollevoet added a comment - I solved this by increasing the reservation time in the External Resource Dispatcher Configuration and limit the checkout time (shorter than the reservation time of the jobs that ). But this is only workaround of course. I guess that the actually locking of the resources happens too late: now it happens in the prebuild method of JobProperty whereas it should happen earlier (in the onStarted method of RunListener).    

            People

            • Assignee:
              terrapin Jan Hollevoet
              Reporter:
              terrapin Jan Hollevoet
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: