Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33725

ThinBackup plugin causes Jenkins to be stuck in quiet mode

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: thinbackup-plugin
    • Labels:
      None
    • Environment:
      ThinBackup 1.7.4
      Jenkins 1.580
      RHEL 7
    • Similar Issues:

      Description

      We are using matrix jobs and also jobs which are running other jobs (Build Flow plugin). We enabled "Wait until Jenkins/Hudson is idle to perform a backup" in ThinBackup configuration, unfortunately, this has caused situation that our Jenkins instance is in quiet mode indefinitely (or until we abort the running jobs).

      The problem occurs when a matrix parent job (or another parent job) is running, some of its child jobs are not running yet (but they have been put in the job queue), and at this moment ThinBackup forces Jenkins into quiet mode. Since the parent jobs will not finish until their children finish, and the child jobs will never be taken from the queue and executed (because of Jenkins quiet mode), Jenkins is stuck in quiet mode forever. The quiet mode cannot be cancelled, because ThinBackup immediately resumes the quiet mode.

      There is obvious short-term "solution" - abort the stuck parent jobs, this will make ThinBackup to proceed with backup and finally cancel Jenkins quiet mode. Unfortunately, this requires restarting the affected jobs which is not acceptable (there are too many of them) and time consuming.

      Since we are making backups every night and we need to run a lot of jobs, our situation in the past few days resulted in encountering Jenkins in quiet mode each morning and having to abort several jobs in order to get it back from quiet mode.

      We have disabled "Wait until Jenkins/Hudson is idle to perform a backup" configuration option for now, but we do not feel comfortable with this "solution" because of potentially inconsistent backups.

        Attachments

          Activity

          Hide
          crussell52 Chris Russell added a comment - - edited

          I am also having this issue. Seems like a very common scenario, which is not at all related to matrix plugins.

          Setup:

          "Job A" builds "Job B" as a blocking operation.

          Problem Sequence:

          1. "Job A" starts
          2. Thinbackup invokes quiet mode. "Job A" continues because it is already running, as expected.
          3. "Job A" invokes "Job B" as a blocking operation
          4. "Job B" is queued, but not permitted to start due to quite mode. "Job A" runs forever. Quiet mode runs forever since "Job A" is still running.

          Workaround:

          Sometimes, if your lucky, you can "cancel" quite mode long enough for "Job B" to start being worked on. More often, you have to manually cancel jobs and then restart them after backup.

          The really bad part is that either way involves a human identifying the condition before it can be resolved. This is far from ideal for an automation tool.

          Show
          crussell52 Chris Russell added a comment - - edited I am also having this issue. Seems like a very common scenario, which is not at all related to matrix plugins. Setup: "Job A" builds "Job B" as a blocking operation. Problem Sequence: 1. "Job A" starts 2. Thinbackup invokes quiet mode. "Job A" continues because it is already running, as expected. 3. "Job A" invokes "Job B" as a blocking operation 4. "Job B" is queued, but not permitted to start due to quite mode. "Job A" runs forever. Quiet mode runs forever since "Job A" is still running. Workaround: Sometimes, if your lucky, you can "cancel" quite mode long enough for "Job B" to start being worked on. More often, you have to manually cancel jobs and then restart them after backup. The really bad part is that either way involves a human identifying the condition before it can be resolved. This is far from ideal for an automation tool.

            People

            • Assignee:
              tofuatjava Thomas F├╝rer
              Reporter:
              mwinkler Marek Winkler
            • Votes:
              7 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: