Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27421

Unserializable iterator & entry classes from Java Collections

    Details

    • Similar Issues:

      Description

      Not currently possible to use a Java 5-style for loop on an ArrayList (for example) from within CPS-transformed code, since its Iterator implementation is not marked Serializable.

        Attachments

          Issue Links

            Activity

            Hide
            sumdumgai A C added a comment -

            bump The inability to use collection iterators at all is a very significant limitation.

            Show
            sumdumgai A C added a comment - bump The inability to use collection iterators at all is a very significant limitation.
            Hide
            jglick Jesse Glick added a comment -

            Workarounds:

            · Use a plain C-style for-loop.
            · Wrap the code in a method marked @NonCps, assuming it does not invoke anything interruptible (incl. calling Workflow steps).

            Show
            jglick Jesse Glick added a comment - Workarounds: · Use a plain C-style for -loop. · Wrap the code in a method marked @NonCps , assuming it does not invoke anything interruptible (incl. calling Workflow steps).
            Hide
            owenmehegan Owen Mehegan added a comment -

            Being able to write:

            node('linux_slave', 'windows_slave')

            { // some steps to run in parallel on both slaves }

            would be fantastic.

            Show
            owenmehegan Owen Mehegan added a comment - Being able to write: node('linux_slave', 'windows_slave') { // some steps to run in parallel on both slaves } would be fantastic.
            Hide
            jglick Jesse Glick added a comment -

            Owen Mehegan you can already do that using

            def labels = ['linux', 'windows']
            for (int i = 0; i < labels.size(); i++) {
              node(labels.get(i)) {
                // something, though you presumably need to select sh vs. bat here
              }
            }
            
            Show
            jglick Jesse Glick added a comment - Owen Mehegan you can already do that using def labels = [ 'linux' , 'windows' ] for ( int i = 0; i < labels.size(); i++) { node(labels.get(i)) { // something, though you presumably need to select sh vs. bat here } }
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7
            Log:
            JENKINS-27421 Reproduced in test.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7 Log: JENKINS-27421 Reproduced in test.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
            http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e
            Log:
            Merge pull request #206 from jglick/Itr-JENKINS-27421

            JENKINS-27421 Investigating ArrayList.Itr problem

            Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e Log: Merge pull request #206 from jglick/Itr- JENKINS-27421 JENKINS-27421 Investigating ArrayList.Itr problem Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238
            Hide
            ssides steve sides added a comment - - edited

            Here's another context where moving away from foreach was not effective.
            I was reading some mercurial repositories with source and subdir (to check out to) from a json file:

            for(repomap in jsonrepos.configs.repos) {
                for ( key in repomap.keySet() ){
                    source = repomap.get(key)['source']
                    subdir = repomap.get(key)['subdir']
                    checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
               }
            

            This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here):

            def slurper = new JsonSlurper()
            def jsonText = readFile 'flow-configs/prebuild.json'
            jsonrepos = slurper.parseText( jsonText )
            repoCount = jsonrepos.configs.repos.size
            println "repositories: ${repoCount}"
            for ( i = 0 ; i < repoCount ; i++ ) {
                rmap = jsonrepos.configs.repos[i]
                key = rmap.keySet().toList().getAt(0)
                source = rmap.get(key)['source']
                subdir = rmap.get(key)['subdir']
                println("  source: ${source}  subdir: ${subdir}")
                checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
            }
            

            It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious).
            This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug.

            java.io.NotSerializableException: groovy.json.JsonSlurper
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
            at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
            at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
            at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
            at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
            at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
            at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
            at java.util.HashMap.writeObject(HashMap.java:1354)
            at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
            at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
            at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
            at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
            at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
            at java.util.TreeMap.writeObject(TreeMap.java:2434)
            at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
            at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
            at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
            at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
            at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
            at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: an exception which occurred:
            in field locals
            in field parent
            in field parent
            in field caller
            in field e
            in field program
            in field threads
            in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4
            Finished: FAILURE

            Show
            ssides steve sides added a comment - - edited Here's another context where moving away from foreach was not effective. I was reading some mercurial repositories with source and subdir (to check out to) from a json file: for (repomap in jsonrepos.configs.repos) { for ( key in repomap.keySet() ){ source = repomap.get(key)[ 'source' ] subdir = repomap.get(key)[ 'subdir' ] checkout([$class: 'MercurialSCM' , credentialsId: '', installation: ' (Default)', source: "${source}" , subdir: "${subdir}" ]) } This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here): def slurper = new JsonSlurper() def jsonText = readFile 'flow-configs/prebuild.json' jsonrepos = slurper.parseText( jsonText ) repoCount = jsonrepos.configs.repos.size println "repositories: ${repoCount}" for ( i = 0 ; i < repoCount ; i++ ) { rmap = jsonrepos.configs.repos[i] key = rmap.keySet().toList().getAt(0) source = rmap.get(key)[ 'source' ] subdir = rmap.get(key)[ 'subdir' ] println( " source: ${source} subdir: ${subdir}" ) checkout([$class: 'MercurialSCM' , credentialsId: '', installation: ' (Default)', source: "${source}" , subdir: "${subdir}" ]) } It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious). This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug. java.io.NotSerializableException: groovy.json.JsonSlurper at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1777) at java.util.HashMap.writeObject(HashMap.java:1354) at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.TreeMap.writeObject(TreeMap.java:2434) at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: an exception which occurred: in field locals in field parent in field parent in field caller in field e in field program in field threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4 Finished: FAILURE
            Hide
            jglick Jesse Glick added a comment -

            steve sides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS. See the tutorial.

            Show
            jglick Jesse Glick added a comment - steve sides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS . See the tutorial.
            Hide
            svanoort Sam Van Oort added a comment - - edited

            Jesse Glick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works:

            import com.cloudbees.groovy.cps.NonCPS
            
            @NonCPS
            List<Map.Entry> get_map_entries(map) {
                // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know
                // Also set iterators are forbidden, so you need an indexed collection
                def myarray = []
                myarray.addAll(map.entrySet())
                return myarray
            }
            
            node {
              def test_envs = [:]
               test_envs["debian:wheezy"] = ["echo goober"]
               def entries = get_map_entries(test_envs)
               
               for (int i=0; i<entries.size(); i++){
                   String key = entries.get(i).key
                   String value =  entries.get(i).value
                   echo "Key $key and value $value"
               }
            }
            
            Show
            svanoort Sam Van Oort added a comment - - edited Jesse Glick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works: import com.cloudbees.groovy.cps.NonCPS @NonCPS List<Map.Entry> get_map_entries(map) { // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know // Also set iterators are forbidden, so you need an indexed collection def myarray = [] myarray.addAll(map.entrySet()) return myarray } node { def test_envs = [:] test_envs[ "debian:wheezy" ] = [ "echo goober" ] def entries = get_map_entries(test_envs) for ( int i=0; i<entries.size(); i++){ String key = entries.get(i).key String value = entries.get(i).value echo "Key $key and value $value" } }
            Hide
            svanoort Sam Van Oort added a comment -

            Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so:

            def myarray = map.entrySet().toArray()
            

            Will give this error:
            hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0]
            Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange)
            at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
            at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
            at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
            at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
            at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15)
            at WorkflowScript.run(WorkflowScript:31)
            at Unknown.Unknown(Unknown)
            at __cps.transform__(Native Method)
            at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
            at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
            at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
            at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
            at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
            at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
            at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
            at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
            at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
            at com.cloudbees.groovy.cps.Next.step(Next.java:58)
            at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
            at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181)
            at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            Finished: FAILURE

            Show
            svanoort Sam Van Oort added a comment - Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so: def myarray = map.entrySet().toArray() Will give this error: hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0] Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55) at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15) at WorkflowScript.run(WorkflowScript:31) at Unknown.Unknown(Unknown) at __ cps.transform __(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79) at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22) at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55) at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE
            Hide
            jglick Jesse Glick added a comment -

            The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i).

            Show
            jglick Jesse Glick added a comment - The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i) .
            Hide
            jglick Jesse Glick added a comment -

            NonCPS need not be imported—Workflow scripts get it by default.

            Show
            jglick Jesse Glick added a comment - NonCPS need not be imported—Workflow scripts get it by default.
            Hide
            jglick Jesse Glick added a comment -

            If you use an asynchronous step

            sh "echo Key $key and value $value"
            

            you will see that your workaround does not really work:

            Started by user anonymous
            [Workflow] Allocate node : Start
            Running on master in …
            [Workflow] node {
            [Workflow] sh
            [flow] Running shell script
            [Workflow] } //node
            [Workflow] Allocate node : End
            [Workflow] End of Workflow
            java.io.NotSerializableException: java.util.LinkedHashMap$Entry
            	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
            	at …
            	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
            	at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
            	at java.util.HashMap.writeObject(HashMap.java:1354)
            	at …
            	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
            	at java.util.TreeMap.writeObject(TreeMap.java:2434)
            	at …
            	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
            	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347)
            	at …
            Caused by: an exception which occurred:
            	in field locals
            	in field parent
            	in field parent
            	in field caller
            	in field e
            	in field program
            	in field threads
            	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287
            Finished: FAILURE
            

            Here is a working script:

            @NonCPS
            List<List<Object>> get_map_entries(map) {
                map.collect {k, v -> [k, v]}
            }
            node {
               def mymap = [a: 1, b: 2]
               def entries = get_map_entries(mymap)
               for (int i=0; i<entries.size(); i++){
                   String key = entries[i][0]
                   String value =  entries[i][1]
                   sh "echo Key $key and value $value"
               }
            }
            
            Show
            jglick Jesse Glick added a comment - If you use an asynchronous step sh "echo Key $key and value $value" you will see that your workaround does not really work: Started by user anonymous [Workflow] Allocate node : Start Running on master in … [Workflow] node { [Workflow] sh [flow] Running shell script [Workflow] } //node [Workflow] Allocate node : End [Workflow] End of Workflow java.io.NotSerializableException: java.util.LinkedHashMap$Entry at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1777) at java.util.HashMap.writeObject(HashMap.java:1354) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.TreeMap.writeObject(TreeMap.java:2434) at … at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347) at … Caused by: an exception which occurred: in field locals in field parent in field parent in field caller in field e in field program in field threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287 Finished: FAILURE Here is a working script: @NonCPS List<List< Object >> get_map_entries(map) { map.collect {k, v -> [k, v]} } node { def mymap = [a: 1, b: 2] def entries = get_map_entries(mymap) for ( int i=0; i<entries.size(); i++){ String key = entries[i][0] String value = entries[i][1] sh "echo Key $key and value $value" } }
            Hide
            svanoort Sam Van Oort added a comment - - edited

            Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:

            org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
            at

            Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.

            Show
            svanoort Sam Van Oort added a comment - - edited Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure at Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.
            Hide
            jglick Jesse Glick added a comment -

            That method is fine to whitelist.

            Show
            jglick Jesse Glick added a comment - That method is fine to whitelist.
            Hide
            svanoort Sam Van Oort added a comment -

            Yes, with note that it is safe because the passed closure is subject to the security rules.

            Show
            svanoort Sam Van Oort added a comment - Yes, with note that it is safe because the passed closure is subject to the security rules.
            Hide
            abayer Andrew Bayer added a comment -

            Is this related to JENKINS-26481 or is it a separate cause?

            Show
            abayer Andrew Bayer added a comment - Is this related to JENKINS-26481 or is it a separate cause?
            Hide
            jglick Jesse Glick added a comment -

            Andrew Bayer an unrelated issue.

            Show
            jglick Jesse Glick added a comment - Andrew Bayer an unrelated issue.
            Hide
            jglick Jesse Glick added a comment -

            Blocking at least the initially attempted use case from JENKINS-26481.

            Show
            jglick Jesse Glick added a comment - Blocking at least the initially attempted use case from JENKINS-26481 .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
            cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629
            Log:
            [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629 Log: [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
            Hide
            jglick Jesse Glick added a comment -

            Not in master yet, be patient JIRA link daemon…

            Show
            jglick Jesse Glick added a comment - Not in master yet, be patient JIRA link daemon…
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cps/pom.xml
            cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
            cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90
            Log:
            Merge pull request #372 from jenkinsci/eachClosure-JENKINS-26481

            JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure)

            Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/pom.xml cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90 Log: Merge pull request #372 from jenkinsci/eachClosure- JENKINS-26481 JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure) Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5
            Log:
            JENKINS-27421 Reproduced in test.
            Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5 Log: JENKINS-27421 Reproduced in test. Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
            cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505
            Log:
            [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
            Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505 Log: [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr, Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629
            Hide
            jglick Jesse Glick added a comment -

            Best to reopen since the original fix covered only ArrayList, not other collections.

            Show
            jglick Jesse Glick added a comment - Best to reopen since the original fix covered only ArrayList , not other collections.
            Hide
            mscharp Michael Scharp added a comment -

            Hey Jesse Glick

            I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry

            After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap. If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code?

            import groovy.json.JsonSlurper;
            
            /**
             * Determine if there is a build for a specific branch
             * currently running.  There is no jenkins api to determine
             * this.
             *
             * @return isBranchBuilding {Boolean}
             */
            def getIsBranchBuilding()
            {
                sh "curl ${env.JOB_URL}api/json?pretty=true > api.json"
            
                def json = readFile("api.json")
                def result = getApi( json )
                def previousBuild = result.lastBuild.number - 1;
            
                for( def i = previousBuild; i > 1; i-- )
                {
                    sh "curl ${env.JOB_URL}${i}/api/json?pretty=true > api2.json"
            
                    def buildApi = readFile( "api2.json" )
                    def buildInfo = getApi( buildApi )
            
                    if( buildInfo.building ) // we've found a build already in progress
                        return true;
                    else if( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete
                        break;
                }
            
                return false;
            }
            
            @NonCPS
            def getApi( jsonString )
            {
                def slurper = new JsonSlurper()
                return slurper.parseText( jsonString );
            }
            
            Show
            mscharp Michael Scharp added a comment - Hey Jesse Glick I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap . If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code? import groovy.json.JsonSlurper; /** * Determine if there is a build for a specific branch * currently running. There is no jenkins api to determine * this . * * @ return isBranchBuilding { Boolean } */ def getIsBranchBuilding() { sh "curl ${env.JOB_URL}api/json?pretty= true > api.json" def json = readFile( "api.json" ) def result = getApi( json ) def previousBuild = result.lastBuild.number - 1; for ( def i = previousBuild; i > 1; i-- ) { sh "curl ${env.JOB_URL}${i}/api/json?pretty= true > api2.json" def buildApi = readFile( "api2.json" ) def buildInfo = getApi( buildApi ) if ( buildInfo.building ) // we've found a build already in progress return true ; else if ( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete break ; } return false ; } @NonCPS def getApi( jsonString ) { def slurper = new JsonSlurper() return slurper.parseText( jsonString ); }
            Hide
            jglick Jesse Glick added a comment -

            Michael Scharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.

            Show
            jglick Jesse Glick added a comment - Michael Scharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            docs/BEST_PRACTICES.md
            http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247
            Log:
            Merge pull request #48 from jenkinsci/map-entries

            JENKINS-27421 Documenting standard workaround for iterating Map.entrySet

            Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: docs/BEST_PRACTICES.md http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247 Log: Merge pull request #48 from jenkinsci/map-entries JENKINS-27421 Documenting standard workaround for iterating Map.entrySet Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9
            Hide
            maxzilla Max Wahler added a comment -

            I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a

            for (String a : items) {}

            worked perfectly, now I get a

            java.io.NotSerializableException: java.util.AbstractList$Itr

            Show
            maxzilla Max Wahler added a comment - I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a for (String a : items) {} worked perfectly, now I get a java.io.NotSerializableException: java.util.AbstractList$Itr
            Hide
            genunix Filip Pytloun added a comment - - edited

            This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.

            Show
            genunix Filip Pytloun added a comment - - edited This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.
            Hide
            johan_piet Johan Piet added a comment -

            I have the same issue as Filip Pytloun is mentioning.

            Show
            johan_piet Johan Piet added a comment - I have the same issue as Filip Pytloun is mentioning.
            Hide
            aburdukovskiy Anton B added a comment -

            Curious if there has been any progress on this. Really simple map iteration seems to be broken right now.

            def testMap = ["test1" : "test1v", "test2" : "test2v"]
            testMap.each { k, v -> echo k }
            

            prints only the first item in cps scripts

            Show
            aburdukovskiy Anton B added a comment - Curious if there has been any progress on this. Really simple map iteration seems to be broken right now. def testMap = [ "test1" : "test1v" , "test2" : "test2v" ] testMap.each { k, v -> echo k } prints only the first item in cps scripts
            Hide
            nickolayr Nickolay Rumyantsev added a comment -

            Anton B It hasn't been working for a long time already, see JENKINS-26481

            Show
            nickolayr Nickolay Rumyantsev added a comment - Anton B It hasn't been working for a long time already, see JENKINS-26481
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540
            Log:
            Simpler workaround for JENKINS-27421.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540 Log: Simpler workaround for JENKINS-27421 .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3
            Log:
            Merge pull request #67 from jglick/test-update

            Simpler workaround for map iteration JENKINS-27421

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3 Log: Merge pull request #67 from jglick/test-update Simpler workaround for map iteration JENKINS-27421 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541
            Hide
            jglick Jesse Glick added a comment -

            The supported way to iterate a Map is to define a helper function:

            @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
            

            and then to call it like:

            for (def e in entries(map)) {
              echo "got ${e[0]} → ${e[1]}"
            }
            
            Show
            jglick Jesse Glick added a comment - The supported way to iterate a Map is to define a helper function: @NonCPS def entries(m) {m.collect {k, v -> [k, v]}} and then to call it like: for (def e in entries(map)) { echo "got ${e[0]} → ${e[1]}" }
            Hide
            b_dean Ben Dean added a comment -

            Jesse Glick, is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like

            @NonCPS def safeArray(c) {
                c.collect { v -> v }
            }
            
            // ...
            def someMap = [a: 1, b:2]
            for (def thing in safeArray(someMap.values())) {
              echo "do something with $thing"
            }
            

            in fact, why bother with .collect, it just needs to be an ArrayList

            def someMap = [a: 1, b:2]
            for (def thing in new ArrayList(someMap.values())) {
              echo "do something with $thing"
            }
            

            which needs script security approval, But that's basically what .collect is doing. I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList.

            Show
            b_dean Ben Dean added a comment - Jesse Glick , is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like @NonCPS def safeArray(c) { c.collect { v -> v } } // ... def someMap = [a: 1, b:2] for (def thing in safeArray(someMap.values())) { echo " do something with $thing" } in fact, why bother with .collect , it just needs to be an ArrayList def someMap = [a: 1, b:2] for (def thing in new ArrayList(someMap.values())) { echo " do something with $thing" } which needs script security approval, But that's basically what .collect is doing . I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList .
            Hide
            jglick Jesse Glick added a comment -

            is there going to be a better way to do that?

            If and when I can figure out how to resolve this issue.

            I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList

            Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map, and returns a serializable List<List>. You could probably also have something like (untested):

            @NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}}
            for (def e in entrySet(map)) {
              echo "got ${e.key} → ${e.value}"
            }
            

            which at least looks more like the standard looping using

            for (def e in map.entrySet()) {
              echo "got ${e.key} → ${e.value}"
            }
            
            Show
            jglick Jesse Glick added a comment - is there going to be a better way to do that? If and when I can figure out how to resolve this issue. I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map , and returns a serializable List<List> . You could probably also have something like (untested): @NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}} for (def e in entrySet(map)) { echo "got ${e.key} → ${e.value}" } which at least looks more like the standard looping using for (def e in map.entrySet()) { echo "got ${e.key} → ${e.value}" }
            Hide
            nickolayr Nickolay Rumyantsev added a comment -

            A little addition to Jesse Glick's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable.

            If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not.
            Jesse Glick, am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

            Show
            nickolayr Nickolay Rumyantsev added a comment - A little addition to Jesse Glick 's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable. If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not. Jesse Glick , am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?
            Hide
            maxzilla Max Wahler added a comment -

            Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this?

            Jesse Glick: I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?

            Show
            maxzilla Max Wahler added a comment - Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this? Jesse Glick : I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?
            Hide
            macdrega Joerg Schwaerzler added a comment - - edited

            +1
            Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts.
            For us the resume feature currently is of no use.
            However is case you once are able to model the full build pipeline in one job it might be useful.
            I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.

            Show
            macdrega Joerg Schwaerzler added a comment - - edited +1 Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts. For us the resume feature currently is of no use. However is case you once are able to model the full build pipeline in one job it might be useful. I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.
            Hide
            jglick Jesse Glick added a comment -

            am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

            Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS.

            introducing a switch where the user can decide if a pipeline is resumable or not?

            Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761. This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments.

            we still need to be able to trigger parts of the pipeline from repository changes.

            This is actually possible, though awkward; please do not discuss it here as it is off topic.

            Show
            jglick Jesse Glick added a comment - am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor? Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS . introducing a switch where the user can decide if a pipeline is resumable or not? Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761 . This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments. we still need to be able to trigger parts of the pipeline from repository changes. This is actually possible, though awkward; please do not discuss it here as it is off topic.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585
            Log:
            JENKINS-27421 Improve behavior of map iteration workaround a bit.
            · Pick up new default whitelist entries.
            · Use a revised workaround that more closely matches the original syntax.
            · Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585 Log: JENKINS-27421 Improve behavior of map iteration workaround a bit. · Pick up new default whitelist entries. · Use a revised workaround that more closely matches the original syntax. · Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36
            Log:
            Merge pull request #77 from jglick/mapIterator-JENKINS-27421

            JENKINS-27421 Improve behavior of map iteration workaround a bit

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36 Log: Merge pull request #77 from jglick/mapIterator- JENKINS-27421 JENKINS-27421 Improve behavior of map iteration workaround a bit Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39
            Hide
            b_dean Ben Dean added a comment -

            Jesse Glick, the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList.

            Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.

            Show
            b_dean Ben Dean added a comment - Jesse Glick , the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList . Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.
            Hide
            svanoort Sam Van Oort added a comment -

            Jesse Glick I see this one is still in progress since October - has it been completed or is it still open and impacting users?

            Show
            svanoort Sam Van Oort added a comment - Jesse Glick I see this one is still in progress since October - has it been completed or is it still open and impacting users?
            Hide
            jglick Jesse Glick added a comment -

            It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

            Show
            jglick Jesse Glick added a comment - It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.
            Hide
            harrygg Harry G. added a comment -

            I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

            Show
            harrygg Harry G. added a comment - I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.
            Hide
            jglick Jesse Glick added a comment -

            Yes this issue is well known. To recap:

            • some kinds of collections, such as ArrayList, have a fix
            • for other collections, you can either
              • use a pre-Java-5-style for loop with an index
              • or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body
              • convert to an ArrayList, either via its constructor for another List or Set, or for a Map via the aforementioned helper method:
            @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
            
            Show
            jglick Jesse Glick added a comment - Yes this issue is well known. To recap: some kinds of collections, such as ArrayList , have a fix for other collections, you can either use a pre-Java-5-style for loop with an index or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body convert to an ArrayList , either via its constructor for another List or Set , or for a Map via the aforementioned helper method: @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3
            Log:
            JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3 Log: JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
            src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5
            Log:
            Merge pull request #124 from jglick/GroovyCategorySupport-JENKINS-26481

            JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5 Log: Merge pull request #124 from jglick/GroovyCategorySupport- JENKINS-26481 JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Great to see that a solution is implemented.

            I've got one question, though:

            Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

            Show
            macdrega Joerg Schwaerzler added a comment - Great to see that a solution is implemented. I've got one question, though: Will the solution work for any collections and allow all kind of different possibilities to iterate through them?
            Hide
            jglick Jesse Glick added a comment -

            The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

            Show
            jglick Jesse Glick added a comment - The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.
            Hide
            autarchprinceps autarch princeps added a comment -

            This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

            Show
            autarchprinceps autarch princeps added a comment - This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.
            Hide
            rpocase Robby Pocase added a comment -

            For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

            Show
            rpocase Robby Pocase added a comment - For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.
            Hide
            jglick Jesse Glick added a comment -

            autarch princeps / Robby Pocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

            Show
            jglick Jesse Glick added a comment - autarch princeps / Robby Pocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                38 Vote for this issue
                Watchers:
                58 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: