Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41897

Karaoke: steps and nodes are limited to 100 - need to increase limit

    Details

    • Sprint:
      iapetus, 1.0
    • Similar Issues:

      Description

      Thorsten Scherler the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc Vivek Pandey)

       In Scope: 

      Out of scope: 

      • steps api (separate ticket)
      • making graph cope with loading large graphs of nodes in a paginated fashion (this needs more thinking and design). 

       

      This could be addressed with a refactoring of Karaoke I guess?
      This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

      It seems that we only get a limited number of steps

      Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

      @Library('test-writer') import writeTest
      import longLog
      node {
          stage 'fin'
          def xml = writeTest()
          sh "echo '$xml' > TEST-some.xml"
          step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
          sh 'echo `date` fin;sleep 3; echo `date` fin;'
          sh 'echo yeah > foo.txt'
          archiveArtifacts 'foo.txt'
          longLog(10000, false)
          stage 'NoSteps'
      }
      

        Attachments

          Issue Links

            Activity

            tscherler Thorsten Scherler created issue -
            jamesdumay James Dumay made changes -
            Field Original Value New Value
            Sprint iapetus [ 216 ]
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Epic Link JENKINS-35750 [ 171713 ]
            Hide
            michaelneale Michael Neale added a comment -

            Thorsten Scherler is this a limit in the data or what the UI shows?

            Show
            michaelneale Michael Neale added a comment - Thorsten Scherler is this a limit in the data or what the UI shows?
            michaelneale Michael Neale made changes -
            Description It seems that we only get a limited number of steps

            Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

            {code}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

            It seems that we only get a limited number of steps

            Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

            {code}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            michaelneale Michael Neale made changes -
            Description [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

            It seems that we only get a limited number of steps

            Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

            {code}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

            This could be addressed with a refactoring of Karaoke I guess?
            This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

            {code}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            michaelneale Michael Neale made changes -
            Summary Karaoke: Steps are limited to 100 Karaoke: Steps are limited to 100 - need to use pagination to fetch more
            michaelneale Michael Neale made changes -
            Assignee Thorsten Scherler [ tscherler ]
            michaelneale Michael Neale made changes -
            Labels karoke
            Hide
            michaelneale Michael Neale added a comment -

            Karaoke needs some rework

            Show
            michaelneale Michael Neale added a comment - Karaoke needs some rework
            michaelneale Michael Neale made changes -
            Link This issue relates to JENKINS-38523 [ JENKINS-38523 ]
            michaelneale Michael Neale made changes -
            Rank Ranked lower
            jamesdumay James Dumay made changes -
            Sprint iapetus [ 216 ] iapetus, 1.0 [ 216, 221 ]
            jamesdumay James Dumay made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            jamesdumay James Dumay made changes -
            Priority Major [ 3 ] Minor [ 4 ]
            jamesdumay James Dumay made changes -
            Sprint iapetus, 1.0 [ 216, 221 ] iapetus, 1.0, 1.0-rc1 [ 216, 221, 261 ]
            Hide
            tscherler Thorsten Scherler added a comment - - edited

            Michael Neale the problem is lies in the server response, that is limiting to 100 for `nodes`. There is no easy solution for this in rendering the pipelineGraph since that needs ALL data to be returned by the rest backend and not  only a subset. That is due to the fact that the code is using the "edges" to determine the next step and if that is not present it will fail.

             

            Show
            tscherler Thorsten Scherler added a comment - - edited Michael Neale the problem is lies in the server response, that is limiting to 100 for `nodes`. There is no easy solution for this in rendering the pipelineGraph since that needs ALL data to be returned by the rest backend and not  only a subset. That is due to the fact that the code is using the "edges" to determine the next step and if that is not present it will fail.  
            tscherler Thorsten Scherler made changes -
            Summary Karaoke: Steps are limited to 100 - need to use pagination to fetch more Karaoke: Steps and nodes are limited to 100 - need to use pagination to fetch more
            Hide
            michaelneale Michael Neale added a comment -

            Thorsten Scherler understood. I guess we can bump up the limit, but know there is a limit. A more digorous solution is perhaps to change how data is returned in large cases, as I think it practically makes no sense to show > 100 nodes, probably right? 

            Ie in large cases, we have to flatten out the steps in the stage similar to how stage view shows it (or else we have UI work to fold some away...)

             

            James Dumay I think we need to think on this some more on the practicalities. There are a few solutions, but just fetching all the data, pagination or not, obviously wont work for all cases. 

            Show
            michaelneale Michael Neale added a comment - Thorsten Scherler understood. I guess we can bump up the limit, but know there is a limit. A more digorous solution is perhaps to change how data is returned in large cases, as I think it practically makes no sense to show > 100 nodes, probably right?  Ie in large cases, we have to flatten out the steps in the stage similar to how stage view shows it (or else we have UI work to fold some away...)   James Dumay I think we need to think on this some more on the practicalities. There are a few solutions, but just fetching all the data, pagination or not, obviously wont work for all cases. 
            Hide
            michaelneale Michael Neale added a comment - - edited

            Well as a medium term measure: https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000 I think is ok IMO. 

            People with large numbers of nodes will have a not great time no matter what we do I think, so that is probably the best bet. 

            If you have 1000s of nodes, parallel or otherwise (firstly, parallel?) then a karaoke like visualisation is not appropriate anyway. So we raise the limit (in future we could recalculate things server side to collapse nodes, which makes the visualisation not representable, but manageable). 

             

            EDIT: Thorsten Scherler Vivek Pandey it would be nice to support a paginated list of steps under a node - but I am not sure if they are included in the total node count/graph? 

            Show
            michaelneale Michael Neale added a comment - - edited Well as a medium term measure: https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000  I think is ok IMO.  People with large numbers of nodes will have a not great time no matter what we do I think, so that is probably the best bet.  If you have 1000s of nodes, parallel or otherwise (firstly, parallel?) then a karaoke like visualisation is not appropriate anyway. So we raise the limit (in future we could recalculate things server side to collapse nodes, which makes the visualisation not representable, but manageable).    EDIT: Thorsten Scherler Vivek Pandey it would be nice to support a paginated list of steps under a node - but I am not sure if they are included in the total node count/graph? 
            michaelneale Michael Neale made changes -
            Summary Karaoke: Steps and nodes are limited to 100 - need to use pagination to fetch more Karaoke: nodes are limited to 100 - need to increase limit
            michaelneale Michael Neale made changes -
            Description [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

            This could be addressed with a refactoring of Karaoke I guess?
            This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

            {code}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

             

             

            This could be addressed with a refactoring of Karaoke I guess?
             This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and [https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919]
            {code:java}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            michaelneale Michael Neale made changes -
            Summary Karaoke: nodes are limited to 100 - need to increase limit Karaoke: steps and nodes are limited to 100 - need to increase limit
            Hide
            michaelneale Michael Neale added a comment -

            OK - lets keep the scope of this to do with increasing the limit for nodes - I will open another ticket for loading more steps as that can use pagination.  

            Show
            michaelneale Michael Neale added a comment - OK - lets keep the scope of this to do with increasing the limit for nodes - I will open another ticket for loading more steps as that can use pagination.  
            michaelneale Michael Neale made changes -
            Description [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

             

             

            This could be addressed with a refactoring of Karaoke I guess?
             This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and [https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919]
            {code:java}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

             In Scope: 
             * Adjust nodes api to fetch more [https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000]
             * Test

            Out of scope: 
             * steps api (separate ticket)
             * making graph cope with loading large graphs of nodes in a paginated fashion (this needs more thinking and design). 

             

            This could be addressed with a refactoring of Karaoke I guess?
             This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and [https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919]
            {code:java}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            michaelneale Michael Neale made changes -
            Link This issue relates to JENKINS-42781 [ JENKINS-42781 ]
            michaelneale Michael Neale made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            Hide
            jamesdumay James Dumay added a comment -

            Michael Neale certainly we need to figure out what a practical limit is for how many nodes to display. There is a design TODO for that in the backlog.

            Show
            jamesdumay James Dumay added a comment - Michael Neale certainly we need to figure out what a practical limit is for how many nodes to display. There is a design TODO for that in the backlog.
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay right - so we can bump some limit, but need to know what it would look like in future for these cases. I really don't think 500 dots across by 500 down is a great UX - but we can at least render something. 

             

            I am sure there is a longer term solution out there. 

            Show
            michaelneale Michael Neale added a comment - James Dumay right - so we can bump some limit, but need to know what it would look like in future for these cases. I really don't think 500 dots across by 500 down is a great UX - but we can at least render something.    I am sure there is a longer term solution out there. 
            tscherler Thorsten Scherler made changes -
            Description [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

             In Scope: 
             * Adjust nodes api to fetch more [https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000]
             * Test

            Out of scope: 
             * steps api (separate ticket)
             * making graph cope with loading large graphs of nodes in a paginated fashion (this needs more thinking and design). 

             

            This could be addressed with a refactoring of Karaoke I guess?
             This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and [https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919]
            {code:java}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000)
                stage 'NoSteps'
            }
            {code}
            [~tscherler] the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc [~vivek])

             In Scope: 
             * Adjust nodes api to fetch more [https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000]
             * Test

            Out of scope: 
             * steps api (separate ticket)
             * making graph cope with loading large graphs of nodes in a paginated fashion (this needs more thinking and design). 

             

            This could be addressed with a refactoring of Karaoke I guess?
             This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

            It seems that we only get a limited number of steps

            Using the following pipelineScript and [https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919]
            {code:java}
            @Library('test-writer') import writeTest
            import longLog
            node {
                stage 'fin'
                def xml = writeTest()
                sh "echo '$xml' > TEST-some.xml"
                step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
                sh 'echo `date` fin;sleep 3; echo `date` fin;'
                sh 'echo yeah > foo.txt'
                archiveArtifacts 'foo.txt'
                longLog(10000, false)
                stage 'NoSteps'
            }
            {code}
            Hide
            vivek Vivek Pandey added a comment -

            Maybe short term of 500 is fine. '/nodes/:id/steps/?start=0&limit=500' would fetch 0 to 500 on first and then on subsequent requests client can look at Link HTTP header, fetch next and simply use it.

            For example:

            https://ci.blueocean.io/blue/rest/organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=0&limit=2
            

            HTTP link Header:

            link:</blue/rest/organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=2&limit=2>; rel="next"
            
            Show
            vivek Vivek Pandey added a comment - Maybe short term of 500 is fine. '/nodes/:id/steps/?start=0&limit=500' would fetch 0 to 500 on first and then on subsequent requests client can look at Link HTTP header, fetch next and simply use it. For example: https: //ci.blueocean.io/blue/ rest /organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=0&limit=2 HTTP link Header: link:</blue/ rest /organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=2&limit=2>; rel= "next"
            Hide
            jamesdumay James Dumay added a comment -

            Closing this as a duplicate of JENKINS-39770.

            Show
            jamesdumay James Dumay added a comment - Closing this as a duplicate of JENKINS-39770 .
            jamesdumay James Dumay made changes -
            Link This issue duplicates JENKINS-39770 [ JENKINS-39770 ]
            jamesdumay James Dumay made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Duplicate [ 3 ]
            jamesdumay James Dumay made changes -
            Sprint iapetus, 1.0, 1.0-rc1 [ 216, 221, 261 ] iapetus, 1.0 [ 216, 221 ]
            jbriden Jenn Briden made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              • Assignee:
                tscherler Thorsten Scherler
                Reporter:
                tscherler Thorsten Scherler
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: