Details

    • Similar Issues:

      Description

      Pipeline plugin is supposed to deliver auto-generated documentation from step implementations across plugins.

      Currently pending the pipeline plugin developers to deliver its side, but then the expectation is that this should be a part of the website content.

        Attachments

          Issue Links

            Activity

            Hide
            kwhetstone Kristin Whetstone added a comment -

            I have a rough version of this that piggybacks the backend-extender. I'm currently refining the discovery process to gather more complete results.

            Show
            kwhetstone Kristin Whetstone added a comment - I have a rough version of this that piggybacks the backend-extender. I'm currently refining the discovery process to gather more complete results.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Kristin Whetstone has discussed this with R. Tyler Croy. She is going to set up another job on ci.jenkins-ci.org that generates a zip file full of asciidocs.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Kristin Whetstone has discussed this with R. Tyler Croy . She is going to set up another job on ci.jenkins-ci.org that generates a zip file full of asciidocs.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Project now resides at pipeline-steps-doc-generator Next steps on this work item would be to review it for any comments and feedback before attacking INFRA-607.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Project now resides at pipeline-steps-doc-generator Next steps on this work item would be to review it for any comments and feedback before attacking INFRA-607 .
            Hide
            kwhetstone Kristin Whetstone added a comment -

            Have created a repository for the job: pipeline-steps-doc-generator. Reviews on this repository is welcome!

            A bit of background on the implementation:
            I started with an implementation that extended the backend-extender which worked well for not only scanning the maven repository for plugins, but also identifying the Pipeline steps. The problem with this approach manifested when it was time to dive into the parameters and their documentation. Since essentially the backend-extender treats each plugin separately, there wasn't a good connection between what the parameters of the plugin could be. For example, the 'checkout' step provides different options based on what plugins are installed in Jenkins; the global documentation should include all options. Trying to provide this level of detail was pretty much impossible. I decided to simulate a PluginManager to take advantage of all the heavy lifting of linking, and eventually listing the plugin steps exactly as it's done in the Snippetizer. (There's the secondary benefit that the asciidoc piece could be easily converted to a new endpoint similar to DSLD in the workflow-plugin)

            Going with a hybridized approach appeared to solve my problems: use the MavenRepository benefits from backend-extender to create my "plugins" and my new PluginManager to create the new documentation. However, I hit another issue with severely incompatible jar files between update-center2 and jenkins: update-center2 uses a deprecated google-collections jar while jenkins uses the replacement option guice. Unfortunately, there is no overlap for the particular methods used in these jars in either releases of collections or in guice. I'm of the opinion that we should update update-center2 to use guice since Google Code where google-collections is shutting down. This was out of scope for this work and I needed something to work for the jenkinsio release, I created a jar with all the maven repository connections and setup within the "scripts" folder. All this does is access maven, and download the hpi files to a "plugins" folder. This isn't the ideal solution to this problem, but until update-center2 is able to updated, it works.

            Along with these other restrictions, I found the positive/negative that you can't have a Jenkins with every single possible plugin installed. While this is likely good news for anyone who has to admin Jenkins, it means there needs to be a list of inclusions or exclusions so we know what plugins to scan for documentation generation. Currently I'm restricting documentation to the list of plugins defined in the workflow-plugin COMPATABILITY.md file. Since there could be plugins made compatible in the future/there might be more out there already, I'm doing future work to make that list a configuration file.

            The new Jenkins,io website has nifty hooks to go out and pull down external sources. I have a working changeset to add the results as another source. That'll be pending this job's results.

            Show
            kwhetstone Kristin Whetstone added a comment - Have created a repository for the job: pipeline-steps-doc-generator . Reviews on this repository is welcome! A bit of background on the implementation: I started with an implementation that extended the backend-extender which worked well for not only scanning the maven repository for plugins, but also identifying the Pipeline steps. The problem with this approach manifested when it was time to dive into the parameters and their documentation. Since essentially the backend-extender treats each plugin separately, there wasn't a good connection between what the parameters of the plugin could be. For example, the 'checkout' step provides different options based on what plugins are installed in Jenkins; the global documentation should include all options. Trying to provide this level of detail was pretty much impossible. I decided to simulate a PluginManager to take advantage of all the heavy lifting of linking, and eventually listing the plugin steps exactly as it's done in the Snippetizer. (There's the secondary benefit that the asciidoc piece could be easily converted to a new endpoint similar to DSLD in the workflow-plugin) Going with a hybridized approach appeared to solve my problems: use the MavenRepository benefits from backend-extender to create my "plugins" and my new PluginManager to create the new documentation. However, I hit another issue with severely incompatible jar files between update-center2 and jenkins: update-center2 uses a deprecated google-collections jar while jenkins uses the replacement option guice. Unfortunately, there is no overlap for the particular methods used in these jars in either releases of collections or in guice. I'm of the opinion that we should update update-center2 to use guice since Google Code where google-collections is shutting down . This was out of scope for this work and I needed something to work for the jenkinsio release, I created a jar with all the maven repository connections and setup within the "scripts" folder. All this does is access maven, and download the hpi files to a "plugins" folder. This isn't the ideal solution to this problem, but until update-center2 is able to updated, it works. Along with these other restrictions, I found the positive/negative that you can't have a Jenkins with every single possible plugin installed. While this is likely good news for anyone who has to admin Jenkins, it means there needs to be a list of inclusions or exclusions so we know what plugins to scan for documentation generation. Currently I'm restricting documentation to the list of plugins defined in the workflow-plugin COMPATABILITY.md file. Since there could be plugins made compatible in the future/there might be more out there already, I'm doing future work to make that list a configuration file. The new Jenkins,io website has nifty hooks to go out and pull down external sources. I have a working changeset to add the results as another source. That'll be pending this job's results.
            Hide
            jglick Jesse Glick added a comment -

            you can't have a Jenkins with every single possible plugin installed

            Why not, specifically? Or at least why can you not create a ClassLoader able to access classes & resources of all known plugins? (There is no need to actually start the Jenkins webapp, since the scanning trick in DescribableHelper works with any ClassLoader, even if Jenkins.getInstance() == null.)

            I'm restricting documentation to the list of plugins defined in the workflow-plugin COMPATABILITY.md file

            This will not suffice, even if that list were complete. Consider for example a plugin offering an implementation of hudson.plugins.git.extensions.GitSCMExtension.isRevExcluded as its only functionality. There is no reason why it would be listed as “Pipeline-compatible” since nothing about it refers to Pipeline or needed to be touched to work with Pipeline. It is just one option among many for entries in GitSCM.extensions.

            go out and pull down external sources. I have a working changeset to add the results as another source.

            What would be the use case? We are only interested in documenting functionality of plugins hosted in the Jenkins update center.

            Show
            jglick Jesse Glick added a comment - you can't have a Jenkins with every single possible plugin installed Why not, specifically? Or at least why can you not create a ClassLoader able to access classes & resources of all known plugins? (There is no need to actually start the Jenkins webapp, since the scanning trick in DescribableHelper works with any ClassLoader , even if Jenkins.getInstance() == null .) I'm restricting documentation to the list of plugins defined in the workflow-plugin COMPATABILITY.md file This will not suffice, even if that list were complete. Consider for example a plugin offering an implementation of hudson.plugins.git.extensions.GitSCMExtension.isRevExcluded as its only functionality. There is no reason why it would be listed as “Pipeline-compatible” since nothing about it refers to Pipeline or needed to be touched to work with Pipeline. It is just one option among many for entries in GitSCM.extensions . go out and pull down external sources. I have a working changeset to add the results as another source. What would be the use case? We are only interested in documenting functionality of plugins hosted in the Jenkins update center.
            Hide
            jglick Jesse Glick added a comment -

            This is a 16Mb binary file. Use BFG to fix it.

            Show
            jglick Jesse Glick added a comment - This is a 16Mb binary file. Use BFG to fix it.
            Hide
            jglick Jesse Glick added a comment -

            This should use checkout scm.

            Show
            jglick Jesse Glick added a comment - This should use checkout scm .
            Hide
            jglick Jesse Glick added a comment -

            This seems redundant.

            Show
            jglick Jesse Glick added a comment - This seems redundant.
            Hide
            jglick Jesse Glick added a comment -

            This is probably way too much work. Who needs ExtensionFinder, ExtensionComponent, etc. just for this? You can call Index.load directly and delete a lot of junk.

            Show
            jglick Jesse Glick added a comment - This is probably way too much work. Who needs ExtensionFinder , ExtensionComponent , etc. just for this? You can call Index.load directly and delete a lot of junk.
            Hide
            jglick Jesse Glick added a comment -

            All this code should be brought up to date with the new structs plugin API as shown here.

            Show
            jglick Jesse Glick added a comment - All this code should be brought up to date with the new structs plugin API as shown here .
            Hide
            jglick Jesse Glick added a comment -

            can you not create a ClassLoader able to access classes & resources of all known plugins?

            For that matter, did you try my original suggestion, which is simply to dump all the plugin JARs and other WEB-INF/lib/*.jar into a flat classpath, ignoring dependencies and version clashes, and scan them from a plain old URLClassLoader?

            Yes there will be linkage errors if you try to load certain classes, or call certain methods of loaded classes, this way. But my bet is that very few, if any, of the calls relevant to reference documentation generation will be in that set. That is because these classes are Describable, which generally refer only to a small set of Jenkins core or plugin APIs, since basically they are just there for form binding, and the method calls we make are things like getStepFunctionName (always returning a constant) or getDisplayName (returning either a constant or a call to a simple Messages method which returns a constant). Any actual calls to third-party libraries would typically be in methods, or separate classes, which are untouched by the reference generation. You would only get into trouble if there are

            • references to core/plugin APIs relating to form binding which have changed incompatibly (unlikely, and if we find any of these, we need to fix them ASAP anyway)
            • references to third-party libraries from the signatures of the form binding classes themselves, which is possible but unlikely, e.g.
            public class Something extends SomeExtensionPoint implements SomeThirdPartyListener {
              @DataBoundConstructor public Something() {}
              @Override void theExtensionPointMethod() {SomeThirdPartyLib.use(this);}
              @Override void thirdPartyNotification(ThirdPartyType arg) {}
              @Extension public static class DescriptorImpl extends Descriptor<SomeExtensionPoint> {
                @Override public String getDisplayName() {return "Something";}
              }
            }
            

            and I cannot think of any such examples offhand. If there are one or two, probably it is OK for documentation for those extensions to be skipped with a warning for now, and we just clean up that plugin code.

            Show
            jglick Jesse Glick added a comment - can you not create a ClassLoader able to access classes & resources of all known plugins? For that matter, did you try my original suggestion, which is simply to dump all the plugin JARs and other WEB-INF/lib/*.jar into a flat classpath, ignoring dependencies and version clashes, and scan them from a plain old URLClassLoader ? Yes there will be linkage errors if you try to load certain classes, or call certain methods of loaded classes, this way. But my bet is that very few, if any, of the calls relevant to reference documentation generation will be in that set. That is because these classes are Describable , which generally refer only to a small set of Jenkins core or plugin APIs, since basically they are just there for form binding, and the method calls we make are things like getStepFunctionName (always returning a constant) or getDisplayName (returning either a constant or a call to a simple Messages method which returns a constant). Any actual calls to third-party libraries would typically be in methods, or separate classes, which are untouched by the reference generation. You would only get into trouble if there are references to core/plugin APIs relating to form binding which have changed incompatibly (unlikely, and if we find any of these, we need to fix them ASAP anyway) references to third-party libraries from the signatures of the form binding classes themselves, which is possible but unlikely, e.g. public class Something extends SomeExtensionPoint implements SomeThirdPartyListener { @DataBoundConstructor public Something() {} @Override void theExtensionPointMethod() {SomeThirdPartyLib.use( this );} @Override void thirdPartyNotification(ThirdPartyType arg) {} @Extension public static class DescriptorImpl extends Descriptor<SomeExtensionPoint> { @Override public String getDisplayName() { return "Something" ;} } } and I cannot think of any such examples offhand. If there are one or two, probably it is OK for documentation for those extensions to be skipped with a warning for now, and we just clean up that plugin code.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            In order to get some documentation on the site for the jenkins.io launch, I created PR-150 which adds the documentation as static files. As soon as the above comments are addressed, I'll add the piece in to download them from an external job.

            Show
            kwhetstone Kristin Whetstone added a comment - In order to get some documentation on the site for the jenkins.io launch, I created PR-150 which adds the documentation as static files. As soon as the above comments are addressed, I'll add the piece in to download them from an external job.
            Hide
            jglick Jesse Glick added a comment -

            You need to help review https://github.com/jenkinsci/structs-plugin/pull/1 and then the current code needs to be adjusted to use DescribableModel after that is released.

            Show
            jglick Jesse Glick added a comment - You need to help review https://github.com/jenkinsci/structs-plugin/pull/1 and then the current code needs to be adjusted to use DescribableModel after that is released.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            Why not, specifically?

            I ran into "too many files open" when attempting to load the plugins. I know restricting plugins isn't the answer either since they depend on each other in more intricate ways; that's the exact reason I wanted to use the power of PluginManager and DescribableHelper.

            I probably didn't describe that in the best way possible as I also "allowed" some plugins that I know were options for pipeline steps. However, I have minimal knowledge about all the interactions, and I really don't think the best solution is to have yet another place to have to update whenever something goes pipeline-compatible.

            The "local change" I have for jenkins.io is the code pulling the generated documentation from an assumed future job. It's only looking at the Jenkins update center. Don't want it to get too complicated! Unfortunately I'm running into (more than the occasional) "ArtifactNotFoundException: Unable to download the artifact from any repository" exception. It's not just me, the backend-extender is hitting the same issue. But at least we're hitting the same thing.

            Show
            kwhetstone Kristin Whetstone added a comment - Why not, specifically? I ran into "too many files open" when attempting to load the plugins. I know restricting plugins isn't the answer either since they depend on each other in more intricate ways; that's the exact reason I wanted to use the power of PluginManager and DescribableHelper. I probably didn't describe that in the best way possible as I also "allowed" some plugins that I know were options for pipeline steps. However, I have minimal knowledge about all the interactions, and I really don't think the best solution is to have yet another place to have to update whenever something goes pipeline-compatible. The "local change" I have for jenkins.io is the code pulling the generated documentation from an assumed future job. It's only looking at the Jenkins update center. Don't want it to get too complicated! Unfortunately I'm running into (more than the occasional) "ArtifactNotFoundException: Unable to download the artifact from any repository" exception. It's not just me, the backend-extender is hitting the same issue. But at least we're hitting the same thing.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            You need to help review https://github.com/jenkinsci/structs-plugin/pull/1

            Sent in a review, thank you for making that change. I'm converting to use that plugin. I'm also testing one other thing with the ClassLoader that might help with the maximum number of files issue. Will update then.

            Show
            kwhetstone Kristin Whetstone added a comment - You need to help review https://github.com/jenkinsci/structs-plugin/pull/1 Sent in a review, thank you for making that change. I'm converting to use that plugin. I'm also testing one other thing with the ClassLoader that might help with the maximum number of files issue. Will update then.
            Hide
            jglick Jesse Glick added a comment -

            I ran into "too many files open" when attempting to load the plugins.

            Try increasing the ulimit.

            Show
            jglick Jesse Glick added a comment - I ran into "too many files open" when attempting to load the plugins. Try increasing the ulimit .
            Hide
            rtyler R. Tyler Croy added a comment -

            What's the status of this? The content on jenkins.io is still hard-coded and slowly becoming out of date.

            Show
            rtyler R. Tyler Croy added a comment - What's the status of this? The content on jenkins.io is still hard-coded and slowly becoming out of date.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            Just sent in a bunch of changes for improvements to this whole structure:
            Submitted PR10 to backend-extender to create the appropriate folder
            Added new changes to pipeline-steps-doc-generator with the changes needed to run a job successfully generating the documentation. Check out that repo for all the changes..

            There were a few suggestions for pipeline-steps-doc-generator, and it is now checking every plugin that's found in the Jenkins `public` maven repository. I'm recording the documentation as it comes from .jelly files so any changes are on a plugin basis. The header can be changed as needed; I followed the "simple-template" as a starting point.

            Waiting in the wings:
            PR to jenkinsio (have written and would definitely need a review) after the pipeline-steps-doc-generator is added as a job, and run successfully once.

            Show
            kwhetstone Kristin Whetstone added a comment - Just sent in a bunch of changes for improvements to this whole structure: Submitted PR10 to backend-extender to create the appropriate folder Added new changes to pipeline-steps-doc-generator with the changes needed to run a job successfully generating the documentation. Check out that repo for all the changes.. There were a few suggestions for pipeline-steps-doc-generator, and it is now checking every plugin that's found in the Jenkins `public` maven repository. I'm recording the documentation as it comes from .jelly files so any changes are on a plugin basis. The header can be changed as needed; I followed the "simple-template" as a starting point. Waiting in the wings: PR to jenkinsio (have written and would definitely need a review) after the pipeline-steps-doc-generator is added as a job, and run successfully once.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            Created a PR for handling the automatic documentation generation. All reviews are welcome.

            Show
            kwhetstone Kristin Whetstone added a comment - Created a PR for handling the automatic documentation generation. All reviews are welcome.
            Hide
            kwhetstone Kristin Whetstone added a comment -

            PR merged. Closing and using INFRA-607 as the tracking ticket since there are a few pieces of infrastructure that need to be addressed.

            Show
            kwhetstone Kristin Whetstone added a comment - PR merged. Closing and using INFRA-607 as the tracking ticket since there are a few pieces of infrastructure that need to be addressed.

              People

              • Assignee:
                kwhetstone Kristin Whetstone
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: