Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-607

Extract pipeline step implementations from all the plugins

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Similar Issues:

      Description

      We want to generate the reference manual for the pipeline plugin to be put on the website (see WEBSITE-129.) This requires an ability to automatically scan all the plugins for step implementations and generate them into some document format, such as asciidoc, HTML, etc.

      IIUC, the current thinking is to extend backend-extension-indexer for this job, given that it's already used to scan all the plugins.

      Another approach discussed in the past includes loading them in classpath and reusing existing reference generator from the pipeline plugin.

        Attachments

          Issue Links

            Activity

            Hide
            rtyler R. Tyler Croy added a comment -

            Recalling where I last left off on this. ci.jenkins.io has agents coming online with limits at 4096 file descriptors, which is clearly not enough for this :-P

            Caused by: java.io.FileNotFoundException: /home/azureuser/workspace/Infra/pipeline-steps-doc-generator/ci-jenkinsfile/docFolder/./plugins/clang-scanbuild-plugin-1.7/WEB-INF/lib/clang-scanbuild-plugin.jar (Too many open files)
            	at java.util.zip.ZipFile.open(Native Method)
            	at java.util.zip.ZipFile.<init>(ZipFile.java:219)
            	at java.util.zip.ZipFile.<init>(ZipFile.java:149)
            	at java.util.jar.JarFile.<init>(JarFile.java:166)
            	at java.util.jar.JarFile.<init>(JarFile.java:103)
            	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
            	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
            	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
            	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
            	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
            	at java.net.URL.openStream(URL.java:1045)
            	at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:151)
            	... 9 more
            
            Show
            rtyler R. Tyler Croy added a comment - Recalling where I last left off on this. ci.jenkins.io has agents coming online with limits at 4096 file descriptors, which is clearly not enough for this :-P Caused by: java.io.FileNotFoundException: /home/azureuser/workspace/Infra/pipeline-steps-doc-generator/ci-jenkinsfile/docFolder/./plugins/clang-scanbuild-plugin-1.7/WEB-INF/lib/clang-scanbuild-plugin.jar (Too many open files) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) at java.net.URL.openStream(URL.java:1045) at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:151) ... 9 more
            Hide
            rtyler R. Tyler Croy added a comment -

            Huin, looks like even when running on some Puppet-managed infrastructure our ulimit is too low

            Unable to obtain resource from ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar: 
            java.io.FileNotFoundException: ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar (Too many open files)
            	at java.util.zip.ZipFile.open(Native Method)
            	at java.util.zip.ZipFile.<init>(ZipFile.java:219)
            	at java.util.zip.ZipFile.<init>(ZipFile.java:149)
            	at java.util.jar.JarFile.<init>(JarFile.java:166)
            	at java.util.jar.JarFile.<init>(JarFile.java:130)
            	at jenkins.util.AntClassLoader.getResourceURL(AntClassLoader.java:1017)
            	at jenkins.util.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:157)
            	at jenkins.util.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:118)
            	at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:964)
            	at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:946)
            	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:498)
            	at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44)
            	at jenkins.ClassLoaderReflectionToolkit._findResources(ClassLoaderReflectionToolkit.java:103)
            	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$UberPlusClassLoader.findResources(HyperLocalPluginManger.java:325)
            	at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
            	at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:142)
            	at net.java.sezpoz.Index$LazyIndexIterator.hasNext(Index.java:185)
            	at com.google.common.collect.Lists.newArrayList(Lists.java:138)
            	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256)
            	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217)
            	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.getIndices(HyperLocalPluginManger.java:397)
            	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.scout(HyperLocalPluginManger.java:460)
            	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$ModClassicPluginStrategy.findComponents(HyperLocalPluginManger.java:358)
            	at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.findSteps(PipelineStepExtractor.java:69)
            	at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.main(PipelineStepExtractor.java:49)
            
            
            Show
            rtyler R. Tyler Croy added a comment - Huin, looks like even when running on some Puppet-managed infrastructure our ulimit is too low Unable to obtain resource from ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar: java.io.FileNotFoundException: ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar (Too many open files) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:130) at jenkins.util.AntClassLoader.getResourceURL(AntClassLoader.java:1017) at jenkins.util.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:157) at jenkins.util.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:118) at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:964) at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:946) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44) at jenkins.ClassLoaderReflectionToolkit._findResources(ClassLoaderReflectionToolkit.java:103) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$UberPlusClassLoader.findResources(HyperLocalPluginManger.java:325) at java.lang. ClassLoader .getResources( ClassLoader .java:1142) at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:142) at net.java.sezpoz.Index$LazyIndexIterator.hasNext(Index.java:185) at com.google.common.collect.Lists.newArrayList(Lists.java:138) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.getIndices(HyperLocalPluginManger.java:397) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.scout(HyperLocalPluginManger.java:460) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$ModClassicPluginStrategy.findComponents(HyperLocalPluginManger.java:358) at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.findSteps(PipelineStepExtractor.java:69) at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.main(PipelineStepExtractor.java:49)
            Hide
            kwhetstone Kristin Whetstone added a comment -

            I had to bump this to a really high number in order to get this to work. My one test system is at 32768 if that helps any.

            Show
            kwhetstone Kristin Whetstone added a comment - I had to bump this to a really high number in order to get this to work. My one test system is at 32768 if that helps any.
            Hide
            danielbeck Daniel Beck added a comment -

            JarUrlConnection leaks. We recently had the problem in Jenkins. Oleg's PR: https://github.com/jenkinsci/jenkins/pull/2516

            Show
            danielbeck Daniel Beck added a comment - JarUrlConnection leaks. We recently had the problem in Jenkins. Oleg's PR: https://github.com/jenkinsci/jenkins/pull/2516
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: R. Tyler Croy
            Path:
            .fixtures.yml
            Puppetfile
            Vagrantfile
            dist/profile/manifests/buildslave.pp
            hieradata/common.yaml
            hieradata/roles/buildslave.yaml
            hieradata/roles/jenkins::agent.yaml
            spec/classes/profile/buildslave_spec.rb
            spec/classes/role/cucumber_spec.rb
            http://jenkins-ci.org/commit/jenkins-infra/eacd0706d79cf4171eee484ca8fbf635baaee713
            Log:
            Add support for tweaking limits for various users via the erwbgy/limits module

            References INFRA-607

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: .fixtures.yml Puppetfile Vagrantfile dist/profile/manifests/buildslave.pp hieradata/common.yaml hieradata/roles/buildslave.yaml hieradata/roles/jenkins::agent.yaml spec/classes/profile/buildslave_spec.rb spec/classes/role/cucumber_spec.rb http://jenkins-ci.org/commit/jenkins-infra/eacd0706d79cf4171eee484ca8fbf635baaee713 Log: Add support for tweaking limits for various users via the erwbgy/limits module References INFRA-607

              People

              • Assignee:
                rtyler R. Tyler Croy
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: