Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18289

Detect & repair out-of-order build records

    Details

    • Similar Issues:

      Description

      The fix of JENKINS-15652 merely downgrades the error but does not clean up build records. It would make sense to have the code detect anomalous conditions and initiate a builds folder cleanup. Take only the last build of a given number, and discard any earlier with higher number. I.e. just set <number> to 0 or something and ignore such builds, or rename build.xml to build-broken.xml.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            A script like the following can be useful for diagnosis:

            CANONICAL_ID_FORMATTER = new java.text.SimpleDateFormat("yyyy-MM-dd_HH-mm-ss");
            NUMBER_XPATH = "/node()/number";
            xpath = javax.xml.xpath.XPathFactory.newInstance().newXPath();
            for (j in Jenkins.instance.getAllItems(AbstractProject)) {
              d = new File(j.rootDir, 'builds');
              m = new TreeMap();
              for (b in d.listFiles()) {
                xml = new File(b, 'build.xml');
                if (!xml.file) {
                  if (!b.getName().startsWith('last')) {
                    println("not a build dir: ${b}");
                  }
                  continue;
                }
                try {
                  CANONICAL_ID_FORMATTER.parse(b.name);
                  xpath.reset();
                  number = xpath.evaluate(NUMBER_XPATH, new org.xml.sax.InputSource(new java.io.FileReader(xml)), javax.xml.xpath.XPathConstants.NODE);
                  if (number) {
                    m.put(b.name, number.textContent);
                  } else {
                    println("cannot parse ${xml}");
                  }
                } catch (java.text.ParseException e) {
                  if (!Util.isSymlink(b)) {
                    println("${b} should be a symlink but is not");
                  }
                }
              }
              original = m.values().collect({Integer.parseInt(it)});
              if (!original.equals(original.sort(false))) {
                println("out of order in ${d}: ${m}");
              }
            }
            
            Show
            jglick Jesse Glick added a comment - A script like the following can be useful for diagnosis: CANONICAL_ID_FORMATTER = new java.text.SimpleDateFormat( "yyyy-MM-dd_HH-mm-ss" ); NUMBER_XPATH = "/node()/number" ; xpath = javax.xml.xpath.XPathFactory.newInstance().newXPath(); for (j in Jenkins.instance.getAllItems(AbstractProject)) { d = new File(j.rootDir, 'builds' ); m = new TreeMap(); for (b in d.listFiles()) { xml = new File(b, 'build.xml' ); if (!xml.file) { if (!b.getName().startsWith( 'last' )) { println( "not a build dir: ${b}" ); } continue ; } try { CANONICAL_ID_FORMATTER.parse(b.name); xpath.reset(); number = xpath.evaluate(NUMBER_XPATH, new org.xml.sax.InputSource( new java.io.FileReader(xml)), javax.xml.xpath.XPathConstants.NODE); if (number) { m.put(b.name, number.textContent); } else { println( "cannot parse ${xml}" ); } } catch (java.text.ParseException e) { if (!Util.isSymlink(b)) { println( "${b} should be a symlink but is not" ); } } } original = m.values().collect({ Integer .parseInt(it)}); if (!original.equals(original.sort( false ))) { println( "out of order in ${d}: ${m}" ); } }
            Hide
            danielbeck Daniel Beck added a comment -

            Jenkins should not destroy build data without administrator confirmation. Something like Manage Old Data for build records would be best.

            Show
            danielbeck Daniel Beck added a comment - Jenkins should not destroy build data without administrator confirmation. Something like Manage Old Data for build records would be best.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java
            core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java
            core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java
            core/src/main/java/jenkins/diagnostics/ooom/Problem.java
            core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java
            core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
            core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
            core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties
            core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy
            core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties
            core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy
            core/src/main/resources/jenkins/security/Messages.properties
            core/src/main/resources/jenkins/security/Messages_zh_TW.properties
            core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy
            core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties
            core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy
            test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy
            http://jenkins-ci.org/commit/jenkins/d90d47d1b76ebb5d753e919afefb5d2457664da2
            Log:
            [FIXED JENKINS-18289]

            The code is smart enough to compute a minimum set of builds to remove to
            restore order consistency.

            Compare: https://github.com/jenkinsci/jenkins/compare/0c6ab8491bff...d90d47d1b76e

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java core/src/main/java/jenkins/diagnostics/ooom/Problem.java core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy core/src/main/resources/jenkins/security/Messages.properties core/src/main/resources/jenkins/security/Messages_zh_TW.properties core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy http://jenkins-ci.org/commit/jenkins/d90d47d1b76ebb5d753e919afefb5d2457664da2 Log: [FIXED JENKINS-18289] The code is smart enough to compute a minimum set of builds to remove to restore order consistency. Compare: https://github.com/jenkinsci/jenkins/compare/0c6ab8491bff...d90d47d1b76e
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            The code moves the offending build records into another directory.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - The code moves the offending build records into another directory.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2766
            [FIXED JENKINS-18289] (Revision d90d47d1b76ebb5d753e919afefb5d2457664da2)

            Result = UNSTABLE
            kohsuke : d90d47d1b76ebb5d753e919afefb5d2457664da2
            Files :

            • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties
            • core/src/main/resources/jenkins/security/Messages_zh_TW.properties
            • core/src/main/java/jenkins/diagnostics/ooom/Problem.java
            • core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java
            • core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy
            • test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy
            • core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties
            • core/src/main/resources/jenkins/security/Messages.properties
            • core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java
            • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy
            • changelog.html
            • core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties
            • core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy
            • core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
            • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy
            • core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java
            • core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
            • core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2766 [FIXED JENKINS-18289] (Revision d90d47d1b76ebb5d753e919afefb5d2457664da2) Result = UNSTABLE kohsuke : d90d47d1b76ebb5d753e919afefb5d2457664da2 Files : core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties core/src/main/resources/jenkins/security/Messages_zh_TW.properties core/src/main/java/jenkins/diagnostics/ooom/Problem.java core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties core/src/main/resources/jenkins/security/Messages.properties core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy changelog.html core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                jglick Jesse Glick
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: