Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31582

Log / document the plugin usage in the flow nodes

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins adds the plugin names and versions to the XML configuration, e.g. like plugin="groovy@1.27". Analyzing config.xml files it is possible to see what pluins are used by a job.

      With workflow this is not possible, since the information what plugin is used, is in the script file. It is probably difficult to analyze the script statically and extract plugin information.

      However, when a script is executed, workflow has the needed information. The steps are already logged to builds/<job-id>/workflow/<step-id>.xml.

      For a git checkout, via the generic SCM API it logs:

      <?xml version='1.0' encoding='UTF-8'?>
      <org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage_-Tag plugin="workflow-support@1.11">
        <node class="org.jenkinsci.plugins.workflow.cps.nodes.StepAtomNode" plugin="workflow-cps@1.11">
          <parents class="com.google.common.collect.ImmutableList">
            <org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode>5</org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode>
          </parents>
          <id>6</id>
          <descriptorId>org.jenkinsci.plugins.workflow.steps.scm.GenericSCMStep</descriptorId>
        </node>
      ...
      

      Improvement:

      Each step node contains also the used plugin(s) and version. In the above case it needs to log workflow-scm-step and git.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Suggested generalization of API: a new Action to be attached to a StepNode which records the step function name and its configuration in the format understood by DescribableModel. Helper methods could try to reconstruct the actual Step object, or produce a list of plugins involved in its definition.

            Would lead to more storage being used by the flow graph, but I think it is not too bad, so long as people refrain from putting ten-page shell scripts inline in their Jenkinsfile (anything complex should anyway be moved to a separate file). There are a lot of potential features related to visualization and analysis which could benefit from access to the detailed step definition.

            Show
            jglick Jesse Glick added a comment - Suggested generalization of API: a new Action to be attached to a StepNode which records the step function name and its configuration in the format understood by DescribableModel . Helper methods could try to reconstruct the actual Step object, or produce a list of plugins involved in its definition. Would lead to more storage being used by the flow graph, but I think it is not too bad, so long as people refrain from putting ten-page shell scripts inline in their Jenkinsfile (anything complex should anyway be moved to a separate file). There are a lot of potential features related to visualization and analysis which could benefit from access to the detailed step definition.
            Hide
            jglick Jesse Glick added a comment -

            (Currently StepAtomNode and StepStartNode record a descriptorId, which could be deprecated in favor of the new action, without breaking compatibility for callers of StepNode.getDescriptor().)

            Show
            jglick Jesse Glick added a comment - (Currently StepAtomNode and StepStartNode record a descriptorId , which could be deprecated in favor of the new action, without breaking compatibility for callers of StepNode.getDescriptor() .)
            Hide
            michaelneale Michael Neale added a comment -

            sounds interesting. I think peoeple may put several line scripts, but not pages, so this may not be too bad.

            Regarding display, would be nice to show a summary of what it did (with ability to display the whole script invoked if warranted), could be quite neat.

            Show
            michaelneale Michael Neale added a comment - sounds interesting. I think peoeple may put several line scripts, but not pages, so this may not be too bad. Regarding display, would be nice to show a summary of what it did (with ability to display the whole script invoked if warranted), could be quite neat.
            Hide
            jglick Jesse Glick added a comment -

            Would suffice to deprecate StepNode and instead introduce a [FlowNode]Action into workflow-api which serializes (via XStream) the actual Step being run. (Details of how to do such a migration compatibly are TBD.) This metadata could be used for many purposes. For JENKINS-34002 we could simply scan for plugin="…" attributes. For JENKINS-37324 (vague as it is) various visualizations which show step-level details could show particular configuration; would also allow us to solve the problem of metasteps. For JENKINS-33995 we would be able to reconstruct the originally requested label expression, not just the node name this ultimately resolved to.

            Show
            jglick Jesse Glick added a comment - Would suffice to deprecate StepNode and instead introduce a [FlowNode]Action into workflow-api which serializes (via XStream) the actual Step being run. (Details of how to do such a migration compatibly are TBD.) This metadata could be used for many purposes. For JENKINS-34002 we could simply scan for plugin="…" attributes. For JENKINS-37324 (vague as it is) various visualizations which show step-level details could show particular configuration; would also allow us to solve the problem of metasteps . For JENKINS-33995 we would be able to reconstruct the originally requested label expression, not just the node name this ultimately resolved to.
            Hide
            svanoort Sam Van Oort added a comment -

            I'll pick this one up initially with a proposal to implement Step serialization in an Action (though reserving the right to simplify where needed to make the API easier to use or sidestep compatibility issue). Explicitly not committing to an ETA on this, just interest.

            May classify that work under JENKINS-37324 or here, depending on scope (I'd like to deal with step parameters first since it's far more necessary than plugin use reporting and I think much simpler too, surprisingly)

            Show
            svanoort Sam Van Oort added a comment - I'll pick this one up initially with a proposal to implement Step serialization in an Action (though reserving the right to simplify where needed to make the API easier to use or sidestep compatibility issue). Explicitly not committing to an ETA on this, just interest. May classify that work under JENKINS-37324 or here, depending on scope (I'd like to deal with step parameters first since it's far more necessary than plugin use reporting and I think much simpler too, surprisingly)
            Hide
            jglick Jesse Glick added a comment -

            I'd like to deal with step parameters first since it's […] much simpler

            Seems doubtful. A Step is a Describable. So are non-trivial step parameters.

            Show
            jglick Jesse Glick added a comment - I'd like to deal with step parameters first since it's […] much simpler Seems doubtful. A Step is a Describable . So are non-trivial step parameters.
            Hide
            jglick Jesse Glick added a comment -

            This is sort of fixed, except the original requirement to record plugin version was apparently dropped.

            Show
            jglick Jesse Glick added a comment - This is sort of fixed, except the original requirement to record plugin version was apparently dropped.
            Hide
            jglick Jesse Glick added a comment -

            would also allow us to solve the problem of metasteps

            Tracking as JENKINS-45109.

            Show
            jglick Jesse Glick added a comment - would also allow us to solve the problem of metasteps Tracking as  JENKINS-45109 .
            Hide
            jglick Jesse Glick added a comment -

            Unfortunately there are problems with the API beyond the lack of plugin version information; it is tricky to reconstruct what the actual Step object looked like. For example, after running junit '*.xml' the singleton value of the map from ArgumentsAction.getArguments is an UninstantiatedDescribable (this seems to not be documented anywhere); yet the model is null, so you have to redo some of the work of determining that the junit symbol referred to JUnitResultArchiver. Presumably if the $class syntax had been used, you would get something else, even though these are purely aspects of surface syntax that is of no relevance after the build completes.

            Show
            jglick Jesse Glick added a comment - Unfortunately there are problems with the API beyond the lack of plugin version information; it is tricky to reconstruct what the actual Step object looked like. For example, after running junit '*.xml' the singleton value of the map from ArgumentsAction.getArguments is an UninstantiatedDescribable (this seems to not be documented anywhere); yet the model is null, so you have to redo some of the work of determining that the junit symbol referred to JUnitResultArchiver . Presumably if the $class syntax had been used, you would get something else, even though these are purely aspects of surface syntax that is of no relevance after the build completes.
            Hide
            jglick Jesse Glick added a comment -

            The tricky part is reconstructing what the arguments actually referred to, since what is stored is not the fully-qualified names of the classes that the running build actually used, but symbols and abbreviated $class names as written in the script—both of which are potentially ambiguous without the context of the parameter using the nested object. Which we have, but the API does not make it straightforward to process this. Perhaps ArgumentsAction should offer a new API like getNormalizedArguments which would, for each nested object, produce an UninstantiatedDescribable whose model has been set, if the required classes are actually still available; from DescribableModel.type you could call PluginManager.whichPlugin or whatever you needed. (Doing this automatically in the existing getArguments is also an option, though the resolution is potentially expensive and not all clients will necessarily need it—though for JENKINS-45109 and JENKINS-34002 it is known to be needed.)

            Show
            jglick Jesse Glick added a comment - The tricky part is reconstructing what the arguments actually referred to, since what is stored is not the fully-qualified names of the classes that the running build actually used, but symbols and abbreviated $class names as written in the script—both of which are potentially ambiguous without the context of the parameter using the nested object. Which we have, but the API does not make it straightforward to process this. Perhaps ArgumentsAction  should offer a new API like getNormalizedArguments which would, for each nested object, produce an UninstantiatedDescribable whose model has been set, if the required classes are actually still available; from DescribableModel.type you could call PluginManager.whichPlugin or whatever you needed. (Doing this automatically in the existing getArguments is also an option, though the resolution is potentially expensive and not all clients will necessarily need it—though for JENKINS-45109 and JENKINS-34002 it is known to be needed.)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/8b2ef48c68b6adf1c2af0e0e53cdfaf5e8de1268
            Log:
            JENKINS-31582 Introduced ArgumentsAction.getResolvedArguments.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/8b2ef48c68b6adf1c2af0e0e53cdfaf5e8de1268 Log: JENKINS-31582 Introduced ArgumentsAction.getResolvedArguments.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/f105670fc62c365da3fc4ea0fda21a80253d511f
            Log:
            Merge pull request #44 from jglick/ArgumentsAction.getResolvedArguments-JENKINS-31582

            JENKINS-31582 ArgumentsAction.getResolvedArguments

            Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/78744ae7045c...f105670fc62c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/f105670fc62c365da3fc4ea0fda21a80253d511f Log: Merge pull request #44 from jglick/ArgumentsAction.getResolvedArguments- JENKINS-31582 JENKINS-31582 ArgumentsAction.getResolvedArguments Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/78744ae7045c...f105670fc62c
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/0a0e05093fecc8adcf8ba1eaf541d655ba7a7464
            Log:
            JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/0a0e05093fecc8adcf8ba1eaf541d655ba7a7464 Log: JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109 .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java
            src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/b78991523e2ff0868610b770bd3719452452fb2d
            Log:
            Merge pull request #151 from jglick/ArgumentsAction.getResolvedArguments-JENKINS-31582

            JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4f703cf1b2c6...b78991523e2f

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/b78991523e2ff0868610b770bd3719452452fb2d Log: Merge pull request #151 from jglick/ArgumentsAction.getResolvedArguments- JENKINS-31582 JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4f703cf1b2c6...b78991523e2f
            Hide
            jglick Jesse Glick added a comment -

            API committed which gives you access to plugin names used in step configurations. No version numbers available. That would require a complete redesign of the API; too late now.

            Show
            jglick Jesse Glick added a comment - API committed which gives you access to plugin names used in step configurations. No version numbers available. That would require a complete redesign of the API; too late now.
            Hide
            batmat Baptiste Mathus added a comment -

            It would have a bit better possibly, but I don't think it's such a big issue. It's easier to get the plugin version, than the plugin's artifactId from the pluginManager's UI anyway.

            Show
            batmat Baptiste Mathus added a comment - It would have a bit better possibly, but I don't think it's such a big issue. It's easier to get the plugin version, than the plugin's artifactId from the pluginManager's UI anyway.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                cruftex Jens Wilke
              • Votes:
                4 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: