Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56567

Build fails when reading more than 125 environment credentials at the same time

    Details

    • Similar Issues:

      Description

      Builds fail with the following error when reading more than 125 credentials into the environment in the Jenkinsfile.

      This prevents us from deploying our services, hence is very blocking.
       

      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: -1: Map expressions can only contain up to 125 entries @ line -1, column -1.
      1 error
      
      	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      It seems weird to use a data structure that only supports up to 125 values in this context, since performance is not ever going to be blocking there. 

      Note that this error does not happen if the credentials are registered in Jenkins and the environment variables are created from string litterals, which is why I beleive the problem comes from this plugin (credentials-binding).

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            There are no steps to reproduce but from the sound of it I assume this is something specific to Declarative Pipeline, as the credentials-binding plugin itself does nothing special with Groovy.

            Show
            jglick Jesse Glick added a comment - There are no steps to reproduce but from the sound of it I assume this is something specific to Declarative Pipeline, as the credentials-binding plugin itself does nothing special with Groovy.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            The underlying issue is the same as JENKINS-47363. Here is the commit to groovy-cps that added the error message for map expressions with more than 125 entries. Maybe something could be done in Declarative to change the underlying construct used for environment blocks so that it isn't using Groovy map expressions directly.

            Show
            dnusbaum Devin Nusbaum added a comment - The underlying issue is the same as  JENKINS-47363 . Here  is the commit to groovy-cps that added the error message for map expressions with more than 125 entries. Maybe something could be done in Declarative to change the underlying construct used for environment blocks so that it isn't using Groovy map expressions directly.
            Hide
            ten0 Thomas Bessou added a comment - - edited

            Maybe Groovy should use the fix that was proposed here since it works around the problem instead of keeping that crazy limitation?

            Or dig a little more into why there are only 125 entries allowed in that data structure and maybe use another one ?

            Show
            ten0 Thomas Bessou added a comment - - edited Maybe Groovy should use the fix that was proposed here since it works around the problem instead of keeping that crazy limitation? Or dig a little more into why there are only 125 entries allowed in that data structure and maybe use another one ?
            Hide
            pete312 Pete Moore added a comment - - edited

            In case there is any problems recreating this it's present in scripted pipelines. This code will produce test.txt to reproduce the error

            bash-4.2$ python << EOF | tee test.txt
            print('node {\n parallel(\n')
            for i in range(200):
              print('   item%s : { sh "echo this is item %s" },' % (i,i))
            print(' )\n}\n')
            EOF

            copy paste the output from that to jenkins window or run load "test.txt". and you get this error

            WorkflowScript: 4: Map expressions can only contain up to 125 entries @ line 4, column 12.
            item0 : { sh "echo this is item 0" },

            Show
            pete312 Pete Moore added a comment - - edited In case there is any problems recreating this it's present in scripted pipelines. This code will produce test.txt to reproduce the error bash-4.2$ python << EOF | tee test.txt print('node {\n parallel(\n') for i in range(200):   print('   item%s : { sh "echo this is item %s" },' % (i,i)) print(' )\n}\n') EOF copy paste the output from that to jenkins window or run load "test.txt". and you get this error WorkflowScript: 4: Map expressions can only contain up to 125 entries @ line 4, column 12. item0 : { sh "echo this is item 0" },
            Hide
            pete312 Pete Moore added a comment -

            I should add that this is limiting on large projects where you want to make a lot of jobs go parallel. 

            Show
            pete312 Pete Moore added a comment - I should add that this is limiting on large projects where you want to make a lot of jobs go parallel. 
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Pete Moore This limitation only affects map and list literals. In scripted Pipeline, you should be able to adapt your script as a workaround. Instead of creating a parallel step with branches inline like this:

            parallel(branch1: { ... }, branch2: { ... })
            

            Create the branches separately like this:

            Map branches = [:]
            branches['branch1'] = { ... }
            branches['branch2'] = { ... }
            parallel(branches)
            
            Show
            dnusbaum Devin Nusbaum added a comment - - edited Pete Moore This limitation only affects map and list literals. In scripted Pipeline, you should be able to adapt your script as a workaround. Instead of creating a parallel step with branches inline like this: parallel(branch1: { ... }, branch2: { ... }) Create the branches separately like this: Map branches = [:] branches[ 'branch1' ] = { ... } branches[ 'branch2' ] = { ... } parallel(branches)
            Hide
            pete312 Pete Moore added a comment -

            Thank you Devin your workaround is a lot cleaner than my workaround which was this.

            parallel(
              group_0 : { 
                parallel( 
                  item0 : { sh "echo this is item 0" },
                  item1 : { sh "echo this is item 1" },
                  ...
                )
              },
              group_1 : { 
                parallel( 
                  item200 : { sh "echo this is item 200" },
                  item201 : { sh "echo this is item 201" },
                  ...
                )
              },
            )
            
            

            Is the limit in Jenkins or groovy? If the limit is in Jenkins can we look forward to a fix?  New users (Ive talked to) like me are pretty stunned by such a small limit.

            Show
            pete312 Pete Moore added a comment - Thank you Devin your workaround is a lot cleaner than my workaround which was this. parallel( group_0 : { parallel( item0 : { sh "echo this is item 0" }, item1 : { sh "echo this is item 1" }, ... ) }, group_1 : { parallel( item200 : { sh "echo this is item 200" }, item201 : { sh "echo this is item 201" }, ... ) }, ) Is the limit in Jenkins or groovy? If the limit is in Jenkins can we look forward to a fix?  New users (Ive talked to) like me are pretty stunned by such a small limit.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                ten0 Thomas Bessou
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: