Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26510

NPE from UnsafeFieldAccessorImpl.ensureObj in SimpleXStreamFlowNodeStorage on J9

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: pipeline
    • Labels:
    • Environment:
      Jenkins 1.588, 1.1 of workflow plugins (Aggregator, API, Basic Steps, Durable TaskStep, Execution Support, Global Shared Library for CPS workflow, Groovy CPS Execution, Job, SCM Step, Step API)
    • Similar Issues:

      Description

      The workflow throws an NPE and hangs when it gets to the node block at the end of the workflow. The stack trace for the npe suggests is is related to copyLogs and SimpleXStreamFlowNodeStorage. The contents of my workflow are listed at the end.

      The workflow has 3 main parts. The first part runs on a node which I'll call node A. Only 1 slave with 1 executor matches this node. The second part runs 55 sub-jobs in parallel across 18 executors on about a dozen slaves. The third part runs on node 'A' again and is intended to pull down the results from the parallel sub-jobs. The first and second parts of the workflow succeed. The workflow fails to start the 3rd part, as it throws a null pointer exception from SimpleXStreamFlowNodeStorage. I have a work around which involves using a separate job for part 3. However, I wanted to post this issue in case it provides details that help with the hardening of the workflow plugins.

      One other bit of environment information. My master cannot directly see the slaves due to a firewall. The slaves are started via a script on the slave and they are able to see and connect to the master. Usage of cloudant does not appear to be related to the issue. Cloudant is used to store files for sharing because in the workflow archiving to the master was slow and had other issues as well.

      log excerpt:

      Done parsing cookbook-recipes-0054.txt
      Running: Allocate node : Body : End
      Running: Allocate node : End
      Running: Execute sub-workflows in parallel : Body : End
      ERROR: failed to start build
      java.lang.NullPointerException
      	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:48)
      	at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.set(UnsafeQualifiedObjectFieldAccessorImpl.java:71)
      	at java.lang.reflect.Field.set(Field.java:670)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadInner(SimpleXStreamFlowNodeStorage.java:218)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadOuter(SimpleXStreamFlowNodeStorage.java:205)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.access$100(SimpleXStreamFlowNodeStorage.java:177)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:624)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.copyLogs(WorkflowRun.java:233)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.waitForCompletion(WorkflowRun.java:217)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:178)
      	at hudson.model.ResourceController.execute(ResourceController.java:89)
      	at hudson.model.Executor.run(Executor.java:240)
      	at hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)
      
      def recipeFileList
      def CATEGORY_ID = ''
      def branches = [:]
      def cloudantSnippets = new com.ibm.devops.CloudantSnippets()
      
      node('A') {
        checkout scm: [ $class : 'GitSCM', branches: [[$class: 'hudson.plugins.git.BranchSpec', name: '*/develop']], userRemoteConfigs : [[ url: 'https://***', credentialsId: '***'  ] ] ] 
      
        dir('resources') {
          echo ':::::::::::::::::::: divide list of recipes into files each containing 200 ::::::::::::::::::::'
          sh 'rm -f cookbook-recipes*.txt'
          sh 'cat all-cookbook-recipes-*.txt >cookbook-recipes.txt'
          sh 'awk \'NR%200==1{x=sprintf("cookbook-recipes-%04d.txt", ++i);}{print > x}\' cookbook-recipes.txt'
          sh 'ls cookbook-recipes-*.txt | sort >sorted-recipe-list.txt'
          recipeFileList = readFile('sorted-recipe-list.txt').tokenize()
        }
        sh 'rm -f workspace.b*'
        sh 'tar cjf workspace.bz2 config data deepqa.home logging.properties pom.xml resources settings.arti3.xml.template src test descriptors'
        CATEGORY_ID = cloudantSnippets.shareFile('workspace.bz2', VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
      
        echo ':::::::::::::::::::: Use maven to run RecipeListParseTest in parallel ::::::::::::::::::::'
        int 
        for (int i = 0; i < recipeFileList.size(); i++) {
          String recipeFileName = "${recipeFileList.get(i)}"
          branches["${recipeFileName}"] = {
            node('parsing') {
      
              sh 'rm -rf *'
              cloudantSnippets.getSharedFile('workspace.bz2', CATEGORY_ID, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
              sh 'tar xjf workspace.bz2'
              echo ":::::::::::::::::::: ${recipeFileName} : generate settings.xml file with encrypted passwords to be used for accessing artifactory ::::::::::::::::::::"
              sh 'wget -N --quiet http://artifactory:8081/artifactory/plugins-release-local/com/ibm/devops/GenerateSecureSettings/1.0/GenerateSecureSettings-1.0-all.jar'
              def mvnHome = tool 'Local maven-3.2.3'
              env.PATH = "${mvnHome}/bin:${env.PATH}"
              env.SOME_PASSWORD = "$SOME_PASSWORD"
              sh "${tool 'local IBM Java 7'}/bin/java -jar GenerateSecureSettings-1.0-all.jar"
              env.SOME_PASSWORD = "not set"
              echo " :::::::::::::::::::: Running RecipeListParseTest with $recipeFileName ::::::::::::::::::::"
              sh "${tool 'Local maven-3.2.3'}/bin/mvn -s settings.xml -DrunSuite='**/*****.class'  -DrecipeFile='${recipeFileName}' -DDB_SERVER=9.12.246.170 -Duima_datapath='resources' -DDB_LAZY=YES -DoutputDir='output/subset' -DJAVA_ARG_LINE='-Xmx5G -Xms1G -XX:MaxPermSize=256m -XX:HeapDumpPath=/dev/null' clean test -U | grep -v 'process.*start' | grep -v 'process.*end' | grep -v 'invoked.*returning' | tee ${recipeFileName}.mvn.log | grep '======' "   
              echo "Upload JSON Output for ${recipeFileName}"
              sh "tar cjf target/classes/output/${recipeFileName}.json.bz2 target/classes/output/subset"
              sh "tar cjf ${recipeFileName}.mvn.log.bz2 ${recipeFileName}.mvn.log"
              env.CATEGORY_ID="${CATEGORY_ID}"
              cloudantSnippets.shareFile("target/classes/output/${recipeFileName}.json.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
              echo "Upload ${recipeFileName}.mvn.log.bz2"
              cloudantSnippets.shareFile("${recipeFileName}.mvn.log.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
              echo "Done parsing ${recipeFileName}"
            }
          }
        }
      }
      
      parallel branches
      
      node('A') {
        echo " :::::::::::::::::::: Retrieve all generated json into workspace ::::::::::::::::::::"
        for (int i = 0; i < recipeFileList.size(); i++) {
          String recipeFileName = "${recipeFileList.get(i)}"
          catchError {
              echo " :::::::::::::::::::: Retrieve json for $recipeFileName ::::::::::::::::::::"        
              cloudantSnippets.getSharedFile("target/classes/output/${recipeFileName}.json.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD)
              cloudantSnippets.getSharedFile("${recipeFileName}.mvn.log.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD)
              sh "tar xjf target/classes/output/${recipeFileName}.json.bz2 "  
              sh "tar xjf ${recipeFileName}.mvn.log.bz2 "  
            }
        }
      }
      

        Attachments

          Activity

          Hide
          jglick Jesse Glick added a comment -

          The NPE is coming from Java code, yet I cannot find a version of UnsafeFieldAccessorImpl in Oracle JDK 6, 7, or 8 that has line 48 in ensureObj, so I cannot tell what is actually failing. What version of Java are you running the Jenkins master on?

          Show
          jglick Jesse Glick added a comment - The NPE is coming from Java code, yet I cannot find a version of UnsafeFieldAccessorImpl in Oracle JDK 6, 7, or 8 that has line 48 in ensureObj , so I cannot tell what is actually failing. What version of Java are you running the Jenkins master on?
          Hide
          canova Tom Canova added a comment -

          java version "1.6.0"
          Java(TM) SE Runtime Environment (build pxa6460sr15fp1-20140110_01(SR15 FP1))
          IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr15-20131231_180656 (JIT enabled, AOT enabled)
          J9VM - 20131231_180656
          JIT - r9_20130920_46510ifx3
          GC - GA24_Java6_SR15_20131231_1152_B180656)
          JCL - 20140107_01

          Show
          canova Tom Canova added a comment - java version "1.6.0" Java(TM) SE Runtime Environment (build pxa6460sr15fp1-20140110_01(SR15 FP1)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr15-20131231_180656 (JIT enabled, AOT enabled) J9VM - 20131231_180656 JIT - r9_20130920_46510ifx3 GC - GA24_Java6_SR15_20131231_1152_B180656) JCL - 20140107_01
          Hide
          jglick Jesse Glick added a comment -

          J9 is rarely tested by Jenkins developers, and you are probably the first to run Workflow on it. Looks like there is some bug in the J9 class libraries, which may or may not be triggered by some mistake in Workflow code. I would first check to see if it is reproducible on the newest version of the IBM JDK, and whether IBM is aware of this stack trace.

          Show
          jglick Jesse Glick added a comment - J9 is rarely tested by Jenkins developers, and you are probably the first to run Workflow on it. Looks like there is some bug in the J9 class libraries, which may or may not be triggered by some mistake in Workflow code. I would first check to see if it is reproducible on the newest version of the IBM JDK, and whether IBM is aware of this stack trace.
          Hide
          danielbeck Daniel Beck added a comment -

          A minimal test case may be useful. Or is all of that actually needed to reproduce the problem?

          Show
          danielbeck Daniel Beck added a comment - A minimal test case may be useful. Or is all of that actually needed to reproduce the problem?
          Hide
          canova Tom Canova added a comment -

          Thanks Jesse. Will see if the latest JDK makes a difference. There have been issues in the past with the JIT compiler that might cause things like this. http://www-01.ibm.com/support/docview.wss?uid=swg1IV44794

          Show
          canova Tom Canova added a comment - Thanks Jesse. Will see if the latest JDK makes a difference. There have been issues in the past with the JIT compiler that might cause things like this. http://www-01.ibm.com/support/docview.wss?uid=swg1IV44794
          Hide
          canova Tom Canova added a comment -

          Upgrading to the current IBM JDK appears to have resolved the problem. I also refactored the code to reduce the chattiness between the master and slaves, though the refactored code also did not work until the JDK update. Here's the refactored code:

          
          def recipeFileList
          def CATEGORY_ID = ''
          def branches = [:]
          def cloudantSnippets = new com.ibm.devops.CloudantSnippets()
          
          node('A') {
            checkout scm: [ $class : 'GitSCM', branches: [[$class: 'hudson.plugins.git.BranchSpec', name: '*/develop']], userRemoteConfigs : [[ url: 'https://hub.jazz.net/git/user/project', credentialsId: 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa'  ] ] ] 
          
            dir('resources') {
              sh """
                 date
                 echo ':::::::::::::::::::: divide list of recipes into files each containing 200 ::::::::::::::::::::'
                 rm -f cookbook-recipes*.txt
                 cat all-cookbook-recipes-*.txt >cookbook-recipes.txt
                 awk 'NR%200==1{x=sprintf("cookbook-recipes-%04d.txt", ++i);}{print > x}' cookbook-recipes.txt
                 ls cookbook-recipes-*.txt | sort >sorted-recipe-list.txt
              """
              recipeFileList = readFile('sorted-recipe-list.txt').tokenize()
            }
            sh """
              rm -f workspace.b*
              tar cjf workspace.bz2 config data logging.properties pom.xml resources src test descriptors
            """
            CATEGORY_ID = cloudantSnippets.shareFile('workspace.bz2', VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
            archive 'CLOUDANT_ID'
          
            for (int i = 0; i < recipeFileList.size(); i++) {
              String recipeFileName = "${recipeFileList.get(i)}"
              branches["${recipeFileName}"] = {
                node('parsing') {
                  def mvnHome = tool 'Local maven-3.2.3'
                  sh 'rm -rf *'
                  cloudantSnippets.getSharedFile('workspace.bz2', CATEGORY_ID, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD)
                  sh """
                     tar xjf workspace.bz2
                     echo ':::::::::::::::::::: ${recipeFileName} : generate settings.xml file with encrypted passwords to be used for accessing artifactory ::::::::::::::::::::'
                     wget -N --quiet http://artifactory:8081/artifactory/plugins-release-local/com/ibm/devops/GenerateSecureSettings/1.0/GenerateSecureSettings-1.0-all.jar
                     export PATH=${mvnHome}/bin:${env.PATH}
                     set +vx
                     export SOME_PASSWORD='$SOME_PASSWORD'
                     ${tool 'local IBM Java 7'}/bin/java -jar GenerateSecureSettings-1.0-all.jar
                     unset SOME_PASSWORD
                     echo " :::::::::::::::::::: Running RecipeListParseTest with $recipeFileName ::::::::::::::::::::"
                     mkdir -p target
                     set -vx
                     ${tool 'Local maven-3.2.3'}/bin/mvn -s settings.xml -DrunSuite='**/RecipeListParseTest.class'  -DrecipeFile='${recipeFileName}' -DDB_SERVER=1.1.1.1 -Duima_datapath='resources' -DDB_LAZY=YES -DoutputDir='output/subset' -DJAVA_ARG_LINE='-Xmx5G -Xms1G -XX:MaxPermSize=256m -XX:HeapDumpPath=/dev/null' test -U | grep -v 'process.*start' | grep -v 'process.*end' | grep -v 'invoked.*returning' | tee target/${recipeFileName}.mvn.log | grep '======' 
                  """
                  sh """
                     echo "Upload JSON Output for ${recipeFileName}"
                     tar cjf target/classes/output/${recipeFileName}.json.bz2 target/classes/output/subset
                     tar cjf target/${recipeFileName}.mvn.log.bz2 target/${recipeFileName}.mvn.log
                  """
                  
                  cloudantSnippets.shareFile("target/classes/output/${recipeFileName}.json.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD, CATEGORY_ID)
                  cloudantSnippets.shareFile("target/${recipeFileName}.mvn.log.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD, CATEGORY_ID)
                }
              }
            }
          }
          
          parallel branches
          
          node('A') {
            for (int i = 0; i < recipeFileList.size(); i++) {
              String recipeFileName = "${recipeFileList.get(i)}"
              try {
                  cloudantSnippets.getSharedFile("target/classes/output/${recipeFileName}.json.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD)
                  cloudantSnippets.getSharedFile("target/${recipeFileName}.mvn.log.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD)
                  sh """
                     echo " :::::::::::::::::::: Retrieve json for $recipeFileName ::::::::::::::::::::"        
                     tar xjf target/classes/output/${recipeFileName}.json.bz2
                     tar xjf target/${recipeFileName}.mvn.log.bz2
                  """
                } catch (Exception ex) {
                  echo "Error retrieving ${recipeFileName}"
                  ex.printStackTrace()
                }
            }
            sh """
               date
               echo "Running on \$HOSTNAME \$(ifconfig|grep inet|head -1|awk '{print \$2}') in the workspace \${PWD}. Are you finished with this workspace?">input_prompt
            """
            def input_prompt=readFile('input_prompt')
            input input_prompt
          }
          
          Show
          canova Tom Canova added a comment - Upgrading to the current IBM JDK appears to have resolved the problem. I also refactored the code to reduce the chattiness between the master and slaves, though the refactored code also did not work until the JDK update. Here's the refactored code: def recipeFileList def CATEGORY_ID = '' def branches = [:] def cloudantSnippets = new com.ibm.devops.CloudantSnippets() node( 'A' ) { checkout scm: [ $class : 'GitSCM' , branches: [[$class: 'hudson.plugins.git.BranchSpec' , name: '*/develop' ]], userRemoteConfigs : [[ url: 'https: //hub.jazz.net/git/user/project' , credentialsId: 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa' ] ] ] dir( 'resources' ) { sh """ date echo ':::::::::::::::::::: divide list of recipes into files each containing 200 ::::::::::::::::::::' rm -f cookbook-recipes*.txt cat all-cookbook-recipes-*.txt >cookbook-recipes.txt awk 'NR%200==1{x=sprintf( "cookbook-recipes-%04d.txt" , ++i);}{print > x}' cookbook-recipes.txt ls cookbook-recipes-*.txt | sort >sorted-recipe-list.txt """ recipeFileList = readFile( 'sorted-recipe-list.txt' ).tokenize() } sh """ rm -f workspace.b* tar cjf workspace.bz2 config data logging.properties pom.xml resources src test descriptors """ CATEGORY_ID = cloudantSnippets.shareFile( 'workspace.bz2' , VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) archive 'CLOUDANT_ID' for ( int i = 0; i < recipeFileList.size(); i++) { String recipeFileName = "${recipeFileList.get(i)}" branches[ "${recipeFileName}" ] = { node( 'parsing' ) { def mvnHome = tool 'Local maven-3.2.3' sh 'rm -rf *' cloudantSnippets.getSharedFile( 'workspace.bz2' , CATEGORY_ID, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) sh """ tar xjf workspace.bz2 echo ':::::::::::::::::::: ${recipeFileName} : generate settings.xml file with encrypted passwords to be used for accessing artifactory ::::::::::::::::::::' wget -N --quiet http: //artifactory:8081/artifactory/plugins-release-local/com/ibm/devops/GenerateSecureSettings/1.0/GenerateSecureSettings-1.0-all.jar export PATH=${mvnHome}/bin:${env.PATH} set +vx export SOME_PASSWORD= '$SOME_PASSWORD' ${tool 'local IBM Java 7' }/bin/java -jar GenerateSecureSettings-1.0-all.jar unset SOME_PASSWORD echo " :::::::::::::::::::: Running RecipeListParseTest with $recipeFileName ::::::::::::::::::::" mkdir -p target set -vx ${tool 'Local maven-3.2.3' }/bin/mvn -s settings.xml -DrunSuite= '**/RecipeListParseTest.class' -DrecipeFile= '${recipeFileName}' -DDB_SERVER=1.1.1.1 -Duima_datapath= 'resources' -DDB_LAZY=YES -DoutputDir= 'output/subset' -DJAVA_ARG_LINE= '-Xmx5G -Xms1G -XX:MaxPermSize=256m -XX:HeapDumpPath=/dev/ null ' test -U | grep -v 'process.*start' | grep -v 'process.*end' | grep -v 'invoked.*returning' | tee target/${recipeFileName}.mvn.log | grep '======' """ sh """ echo "Upload JSON Output for ${recipeFileName}" tar cjf target/classes/output/${recipeFileName}.json.bz2 target/classes/output/subset tar cjf target/${recipeFileName}.mvn.log.bz2 target/${recipeFileName}.mvn.log """ cloudantSnippets.shareFile( "target/classes/output/${recipeFileName}.json.bz2" , VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD, CATEGORY_ID) cloudantSnippets.shareFile( "target/${recipeFileName}.mvn.log.bz2" , VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD, CATEGORY_ID) } } } } parallel branches node( 'A' ) { for ( int i = 0; i < recipeFileList.size(); i++) { String recipeFileName = "${recipeFileList.get(i)}" try { cloudantSnippets.getSharedFile( "target/classes/output/${recipeFileName}.json.bz2" , CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD) cloudantSnippets.getSharedFile( "target/${recipeFileName}.mvn.log.bz2" , CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD) sh """ echo " :::::::::::::::::::: Retrieve json for $recipeFileName ::::::::::::::::::::" tar xjf target/classes/output/${recipeFileName}.json.bz2 tar xjf target/${recipeFileName}.mvn.log.bz2 """ } catch (Exception ex) { echo "Error retrieving ${recipeFileName}" ex.printStackTrace() } } sh """ date echo "Running on \$HOSTNAME \$(ifconfig|grep inet|head -1|awk '{print \$2}' ) in the workspace \${PWD}. Are you finished with this workspace?" >input_prompt """ def input_prompt=readFile( 'input_prompt' ) input input_prompt }
          Hide
          canova Tom Canova added a comment -

          This was a defect with the IBM JDK - likely with the JIT Compiler. Resolved by updating the JDK.

          Show
          canova Tom Canova added a comment - This was a defect with the IBM JDK - likely with the JIT Compiler. Resolved by updating the JDK.
          Hide
          jglick Jesse Glick added a comment -

          OK, glad you managed to resolve this.

          Show
          jglick Jesse Glick added a comment - OK, glad you managed to resolve this.

            People

            • Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              canova Tom Canova
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: