Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34038

New flow analyzer algorithm in pipeline stage view based on a visitor pattern

    Details

    • Similar Issues:

      Description

      Implementation of the fixes for https://issues.jenkins-ci.org/browse/JENKINS-33624 greatly increases scalability of stage view with large flow graphs, but it revealed some major underlying issues with the existing algorithms, in correctness, scalability, and extensibility. They're O(n log n) at least, and in practice something like O(m*n + n log n) with m being stage count.

      • This needs to track FlowSegments, which are linear sections of the flow graph with a defined start and end. This is needed because you need to handle and work with the structure of the flow (stages, labels, parallel blocks) without having to be aware of every node within, since there may be thousands.
      • This needs to support parallel flowsegments correctly, enabling JENKINS-33185
      • It needs to be aware of block scoping to correctly assign executor node labels to FlowNodes - Root cause for JENKINS-33290 and eventually for block labelling.
      • It needs to offer an (eventually pluggable) filter option to allow searching for specific nodes.
      • It needs to be largely O( n ), and where possible only do one pass through flownodes.
      • It should offer an API that is SAX-like, which tracks segment start/end, block start/end, and node visiting (in stage) with stages being mapped as a segment type.
      • It should be extensible for plugin authors to define their own analysis of flowgraphs (probably for visualization)

      If it's considered high enough quality, this can be added to the main pipeline plugin, as an available API.

      I have a PR (linked) that is addressing this need, which is currently a work in progress but close to covering this.

        Attachments

          Activity

          Hide
          svanoort Sam Van Oort added a comment -

          This is still in progress, and it will be consuming the ForkScanner workflow API work from https://github.com/jenkinsci/workflow-api-plugin/pull/2 because this is the only sane way to handle analysis of blocks and stages in the order encountered (otherwise you need to keep a complex record of all blocks).

          Show
          svanoort Sam Van Oort added a comment - This is still in progress, and it will be consuming the ForkScanner workflow API work from https://github.com/jenkinsci/workflow-api-plugin/pull/2 because this is the only sane way to handle analysis of blocks and stages in the order encountered (otherwise you need to keep a complex record of all blocks).
          Hide
          svanoort Sam Van Oort added a comment -

          Yes, it's still in progress. It's mutated into a more general API set since we have a cluster of similar problems cropping up.

          Show
          svanoort Sam Van Oort added a comment - Yes, it's still in progress. It's mutated into a more general API set since we have a cluster of similar problems cropping up.
          Hide
          svanoort Sam Van Oort added a comment -

          Out of review and merged to master, pending release as v2.0 after a bit of additional testing.

          Show
          svanoort Sam Van Oort added a comment - Out of review and merged to master, pending release as v2.0 after a bit of additional testing.
          Hide
          svanoort Sam Van Oort added a comment -

          Released as version 2.0

          Show
          svanoort Sam Van Oort added a comment - Released as version 2.0
          Hide
          think think think added a comment -

          since the latest release I cannot color failing stages anymore. Before I could color failing stages dark red. Am I doing something wrong or is it because of this change?

          Show
          think think think added a comment - since the latest release I cannot color failing stages anymore. Before I could color failing stages dark red. Am I doing something wrong or is it because of this change?
          Hide
          andrzejchm Andrzej Chmielewski added a comment - - edited

          yeah, since the 2.0 release, the stages no longer indicate "failed" status, has anything changed on that matter Sam Van Oort ?

          Show
          andrzejchm Andrzej Chmielewski added a comment - - edited yeah, since the 2.0 release, the stages no longer indicate "failed" status, has anything changed on that matter Sam Van Oort ?
          Hide
          svanoort Sam Van Oort added a comment -

          think think Could you clarify what you mean in a separate ticket with instructions to reproduce please?

          I suspect you're referring to a fix done much earlier (several versions ago) that ensured caught errors would not result in a failed stage – unfortunately some people were taking advantage of that bug for tagging stages as failed... but can't tell without more info.

          Show
          svanoort Sam Van Oort added a comment - think think Could you clarify what you mean in a separate ticket with instructions to reproduce please? I suspect you're referring to a fix done much earlier (several versions ago) that ensured caught errors would not result in a failed stage – unfortunately some people were taking advantage of that bug for tagging stages as failed... but can't tell without more info.
          Hide
          svanoort Sam Van Oort added a comment -

          Andrzej Zadorozny Same question as above: please open a separate ticket with instructions to reproduce. I suspect what you're wanting will be covered under JENKINS-26522 but cannot confirm off the top of my head.

          Show
          svanoort Sam Van Oort added a comment - Andrzej Zadorozny Same question as above: please open a separate ticket with instructions to reproduce. I suspect what you're wanting will be covered under JENKINS-26522 but cannot confirm off the top of my head.

            People

            • Assignee:
              svanoort Sam Van Oort
              Reporter:
              svanoort Sam Van Oort
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: