Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52249

Parallel build led to AIOOBE in Plot.savePlotData

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: plot-plugin
    • Labels:
      None
    • Environment:
      The latest and greatest:
      * Jenkins (core) 2.129
      * Plot Plugin 2.1.0
    • Similar Issues:

      Description

      2 parallel builds (scripted pipeline) executed the "same" plot step at very, very much the same time (as far as I can tell it was seemingly definitely at least the same second, but of course it must have been really the very, very same time) leading to one of the two builds failing with:

      java.lang.ArrayIndexOutOfBoundsException: 2
          at hudson.plugins.plot.Plot.savePlotData(Plot.java:967)
          at hudson.plugins.plot.Plot.addBuild(Plot.java:717)
          at hudson.plugins.plot.PlotBuilder.perform(PlotBuilder.java:223)
          at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
          at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
          at hudson.security.ACL.impersonate(ACL.java:290)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      
      • ... whereas the other one was successful
      • Presumably the plot CSV file in the pipeline job folder was corrupted by that, because the next build (not in parallel anymore) failed with the same symptom.
      • But must have kind of fixed the corrupt plot CSV file in the pipeline job folder. But in a rather forceful way...
      • Because subsequent builds (also not in parallel anymore) then succeeded.

      Sad side effects:

      • Please note:
        • there are in fact 4 consecutive plot step calls
        • seemingly the 3rd one collided
      • The "interesting"/sad result is:
        • in the plot CSV files of the plot step calls #1 and #2 "only" the values/entries/lines of the successful build are missing
        • in the plot CSV file of the colliding plot step call #3 there is a huge gap of values/entries/lines: in fact the bulk of the history is lost and the remaining file has less than 10% of the size it had before
        • in the plot CSV file of the plot step call #4 (which was skipped due to the failures) the values/entries/lines of the two failing builds are missing; which makes of course sense

        Attachments

          Activity

          Hide
          reinholdfuereder Reinhold Füreder added a comment -

          (IMHO not related to JENKINS-6913 or JENKINS-43708; but maybe similar to old Jenkins core bugs JENKINS-22516 and JENKINS-25930?)

          Jesse Glick Can standard JVM synchronization mechanism (e.g. synchronized keyword for synchronized methods) be used in (a) Jenkins plugin code (that is called via pipelines that are executed via groovy CPS whatever magic) and (b) shared Jenkins pipeline library code (that again is called via pipelines that are executed via groovy CPS whatever magic)? Or is something like https://gist.github.com/pditommaso/4616192e50cd333487211d1e37e81d58 necessary? Thanks for any hints!

          Show
          reinholdfuereder Reinhold Füreder added a comment - (IMHO not related to JENKINS-6913 or JENKINS-43708 ; but maybe similar to old Jenkins core bugs JENKINS-22516 and JENKINS-25930 ?) Jesse Glick Can standard JVM synchronization mechanism (e.g. synchronized keyword for synchronized methods) be used in (a) Jenkins plugin code (that is called via pipelines that are executed via groovy CPS whatever magic) and (b) shared Jenkins pipeline library code (that again is called via pipelines that are executed via groovy CPS whatever magic)? Or is something like https://gist.github.com/pditommaso/4616192e50cd333487211d1e37e81d58 necessary? Thanks for any hints!
          Hide
          jglick Jesse Glick added a comment -

          Reinhold Füreder (a) yes, sure, which is why this is a bug in the plot plugin; (b) no, since this is a virtual machine using green threads, but there is a lock step.

          Show
          jglick Jesse Glick added a comment - Reinhold Füreder (a) yes, sure, which is why this is a bug in the plot plugin; (b) no, since this is a virtual machine using green threads, but there is a lock step.
          Hide
          reinholdfuereder Reinhold Füreder added a comment -

          Jesse Glick Sorry for this potentially irritating or annoying follow-up question (that is only partially related with this issue):

          Let's claim the synchronization is in both cases – that is for (a) Jenkins plugin code providing a pipeline step and (b) shared Jenkins pipeline library code – needed for protecting a shared file in the pipeline job folder ("/var/lib/jenkins/jobs/...") that is read and written by each pipeline execution (= pipeline build); and this shared file access is being done in (a) and (b) respectively.

          My naive expectation would have been that when calling the pipeline step – that internally reads and writes a shared file in the pipeline from (a) from either a Jenkinsfile or from (b) – then any standard JVM synchronization mechanism does not help either, because it is also called in context of the "virtual machine using green threads" (as you wrote)?

          I try to depict this:

          • "Jenkinsfile" -> custom step from custom shared pipeline library -> "plot" step from "Plot" plugin -> synchronized method in "Plot" plugin
          • "Jenkinsfile" -> custom step from custom shared pipeline library -> synchronized method in custom shared pipeline library

          I see one (most of the times) disadvantage of using the lock step for such a let's call it rather low-level synchronization: the resource will not just be kind of living forever, but especially bloating up the "Lockable Resources" view (and the "Lockable Resources" section in "Configure System"), see JENKINS-38906; but maybe I am just stubborn and such a distinction in low-level vs. high-level synchronization is unfair?

          Show
          reinholdfuereder Reinhold Füreder added a comment - Jesse Glick Sorry for this potentially irritating or annoying follow-up question (that is only partially related with this issue): Let's claim the synchronization is in both cases – that is for (a) Jenkins plugin code providing a pipeline step and (b) shared Jenkins pipeline library code – needed for protecting a shared file in the pipeline job folder ("/var/lib/jenkins/jobs/...") that is read and written by each pipeline execution (= pipeline build); and this shared file access is being done in (a) and (b) respectively. My naive expectation would have been that when calling the pipeline step – that internally reads and writes a shared file in the pipeline from (a) from either a Jenkinsfile or from (b) – then any standard JVM synchronization mechanism does not help either, because it is also called in context of the "virtual machine using green threads" (as you wrote)? I try to depict this: "Jenkinsfile" -> custom step from custom shared pipeline library -> "plot" step from "Plot" plugin -> synchronized method in "Plot" plugin "Jenkinsfile" -> custom step from custom shared pipeline library -> synchronized method in custom shared pipeline library I see one (most of the times) disadvantage of using the lock step for such a let's call it rather low-level synchronization: the resource will not just be kind of living forever, but especially bloating up the "Lockable Resources" view (and the "Lockable Resources" section in "Configure System"), see JENKINS-38906 ; but maybe I am just stubborn and such a distinction in low-level vs. high-level synchronization is unfair?
          Hide
          reinholdfuereder Reinhold Füreder added a comment -

          Since it happened again (the 2nd time overall and as expected rather seldom: the 2nd occurrence >6 months after the 1st one; but again with loss of historical plot data) I really implemented the suggested workaround and changed my shared Jenkins pipeline library code to use a lock step as a so-called critical section.

          Show
          reinholdfuereder Reinhold Füreder added a comment - Since it happened again (the 2nd time overall and as expected rather seldom: the 2nd occurrence >6 months after the 1st one; but again with loss of historical plot data) I really implemented the suggested workaround and changed my shared Jenkins pipeline library code to use a lock step as a so-called critical section.

            People

            • Assignee:
              vgaidarji Veaceslav Gaidarji
              Reporter:
              reinholdfuereder Reinhold Füreder
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: