Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52394

api/json?depth=2 - Duration not available

    Details

    • Similar Issues:

      Description

      We use a pipeline of 40+ VMs to run our tests. Our goal is to find the total time it takes for all the VMs to run all of our tests - this is all available through the flowGraphTable. But the flowGraphTable is not easily available to obtain through the API, so we use api/json?depth=2 to find all of the nodes that have ran in each build. We use their link and add the endpoint "wfapi/describe" (code below) to retrieve the time it took for that node to run. However, there are two major issues that arise by using that workaround

       

      {
      "_links":{
           "self":{
                "href":"http://<jenkins_url>/job/<jenkins_server_info>/wfapi/describe"
           }
      },
      "id":"",
      "name":"Build",
      "status":"FAILED",
      "startTimeMillis":1000000,
      "endTimeMillis":2000000,
      "durationMillis":51,
      "queueDurationMillis":5,
      "pauseDurationMillis":0,
      "stages":[{
           "_links":{
                "self":{
                     "href":""
                }
           },
           "id":"5",
           "name":"Resource Check",
           "execNode":"",
           "status":"SUCCESS",
           "startTimeMillis":10000,
           "durationMillis":508,
           "pauseDurationMillis":0
      }
      

       

       

      1. It takes a few minutes to parse through all the nodes and make each request
      2. Many nodes lead to a JSON file that is null - no data is part of the response

      The 2nd point is bold as that is our major roadblock. The nodes that should be providing us the most useful information are giving us a response of null.

      We have worked with adding the endpoint "wfapi/describe" after the build number. We iterate through all of the stages and go through their individual responses. The issue here, however, is that only first 100 tests, including their time, are shown - everything else is not there. For us, this is the most valuable information as it allows us to see how long each VM took to run tests. 

      I am proposing three new added keys and values to the endpoint "api/json?depth=2" as shown below:

       

      {
      "_class":"org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode",
      "actions":[{},{},{}],
      "displayName":"",
      "iconColor":"",
      "id":"",
      "parents":[""],
      "running":false,
      "durationMillis":0,
      "startTimeMillis":100000,
      "queueDurationMillis":0,
      "pauseDurationMillis":0,
      "url":""
      }
      

       

      1. "durationMillis" is the time it took for the rest to run - in milliseconds
      2. "startTimeMillis" is the time that process started - in UNIX milliseconds
      3. "queueDurationMillis" is the time the process was waiting in queue before being ran - in milliseconds
      4. "pauseDurationMillis" is the time that the process paused for before running again - in milliseconds

       

        Attachments

          Issue Links

            Activity

            nateshkukreja Natesh Kukreja created issue -
            nateshkukreja Natesh Kukreja made changes -
            Field Original Value New Value
            Description We use a pipeline of 40+ VMs to run our tests. Our goal is to find the total time it takes for all the VMs to run all of our tests - this is all available through the flowGraphTable. But the flowGraphTable is not easily available to obtain through the API, so we use api/json?depth=2 to find all of the nodes that have ran in each build. We use their link and add the endpoint "wfapi/describe" (code below) to retrieve the time it took for that node to run. However, there are two major issues that arise by using that workaround

             
            {code:java}
            {
            "_links":{
                 "self":{
                      "href":"http://&lt;jenkins_url&gt;/job/&lt;jenkins_server_info&gt;/wfapi/describe"
                 }
            },
            "id":"",
            "name":"Build",
            "status":"FAILED",
            "startTimeMillis":1000000,
            "endTimeMillis":2000000,
            "durationMillis":51,
            "queueDurationMillis":5,
            "pauseDurationMillis":0,
            "stages":[{
                 "_links":{
                      "self":{
                           "href":""
                      }
                 },
                 "id":"5",
                 "name":"Resource Check",
                 "execNode":"",
                 "status":"SUCCESS",
                 "startTimeMillis":10000,
                 "durationMillis":508,
                 "pauseDurationMillis":0
            }
            {code}
             

             
             # It takes a few minutes to parse through all the nodes and make each request
             # *Many nodes lead to a JSON file that is null - no data is part of the response*

            The 2nd point is bold as that is our major roadblock. The nodes should be providing us the most useful information are giving us a response of null.

            We have worked with adding the endpoint "wfapi/describe" after the build number. We iterate through all of the stages and go through their individual responses. The issue here, however, is that only first 100 tests, including their time, are shown - everything else is not there. For us, this is the most valuable information as it allows us to see how long each VM took to run tests. 

            I am proposing three new added keys and values to the endpoint "api/json?depth=2" as shown below:

             
            {code:java}
            {
            "_class":"org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode",
            "actions":[{},{},{}],
            "displayName":"",
            "iconColor":"",
            "id":"",
            "parents":[""],
            "running":false,
            "durationMillis":0,
            "startTimeMillis":100000,
            "queueDurationMillis":0,
            "pauseDurationMillis":0,
            "url":""
            }
            {code}
             
             # "durationMillis" is the time it took for the rest to run - in milliseconds
             # "startTimeMillis" is the time that process started - in UNIX milliseconds
             # "queueDurationMillis" is the time the process was waiting in queue before being ran - in milliseconds
             # "pauseDurationMillis" is the time that the process paused for before running again - in milliseconds

             
            We use a pipeline of 40+ VMs to run our tests. Our goal is to find the total time it takes for all the VMs to run all of our tests - this is all available through the flowGraphTable. But the flowGraphTable is not easily available to obtain through the API, so we use api/json?depth=2 to find all of the nodes that have ran in each build. We use their link and add the endpoint "wfapi/describe" (code below) to retrieve the time it took for that node to run. However, there are two major issues that arise by using that workaround

             
            {code:java}
            {
            "_links":{
                 "self":{
                      "href":"http://&lt;jenkins_url&gt;/job/&lt;jenkins_server_info&gt;/wfapi/describe"
                 }
            },
            "id":"",
            "name":"Build",
            "status":"FAILED",
            "startTimeMillis":1000000,
            "endTimeMillis":2000000,
            "durationMillis":51,
            "queueDurationMillis":5,
            "pauseDurationMillis":0,
            "stages":[{
                 "_links":{
                      "self":{
                           "href":""
                      }
                 },
                 "id":"5",
                 "name":"Resource Check",
                 "execNode":"",
                 "status":"SUCCESS",
                 "startTimeMillis":10000,
                 "durationMillis":508,
                 "pauseDurationMillis":0
            }
            {code}
             

             
             # It takes a few minutes to parse through all the nodes and make each request
             # *Many nodes lead to a JSON file that is null - no data is part of the response*

            The 2nd point is bold as that is our major roadblock. The nodes that should be providing us the most useful information are giving us a response of null.

            We have worked with adding the endpoint "wfapi/describe" after the build number. We iterate through all of the stages and go through their individual responses. The issue here, however, is that only first 100 tests, including their time, are shown - everything else is not there. For us, this is the most valuable information as it allows us to see how long each VM took to run tests. 

            I am proposing three new added keys and values to the endpoint "api/json?depth=2" as shown below:

             
            {code:java}
            {
            "_class":"org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode",
            "actions":[{},{},{}],
            "displayName":"",
            "iconColor":"",
            "id":"",
            "parents":[""],
            "running":false,
            "durationMillis":0,
            "startTimeMillis":100000,
            "queueDurationMillis":0,
            "pauseDurationMillis":0,
            "url":""
            }
            {code}
             
             # "durationMillis" is the time it took for the rest to run - in milliseconds
             # "startTimeMillis" is the time that process started - in UNIX milliseconds
             # "queueDurationMillis" is the time the process was waiting in queue before being ran - in milliseconds
             # "pauseDurationMillis" is the time that the process paused for before running again - in milliseconds

             
            oleg_nenashev Oleg Nenashev made changes -
            Component/s _unsorted [ 19622 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > api/json?depth=2 to find all of the nodes that have ran in each build

            Well, this core's logic will not work for Pipeline now. Generally there should be some node tracking API, e.g. as requested in JENKINS-32225 and JENKINS-33995

            So, routing to the Pipeline team

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - > api/json?depth=2 to find all of the nodes that have ran in each build Well, this core's logic will not work for Pipeline now. Generally there should be some node tracking API, e.g. as requested in  JENKINS-32225 and  JENKINS-33995 So, routing to the Pipeline team  
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue duplicates JENKINS-32225 [ JENKINS-32225 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue duplicates JENKINS-33995 [ JENKINS-33995 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue duplicates JENKINS-32225 [ JENKINS-32225 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue duplicates JENKINS-33995 [ JENKINS-33995 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-32225 [ JENKINS-32225 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-33995 [ JENKINS-33995 ]
            oleg_nenashev Oleg Nenashev made changes -
            Component/s pipeline [ 21692 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ] Sam Van Oort [ svanoort ]
            Hide
            svanoort Sam Van Oort added a comment -

            > Our goal is to find the total time it takes for all the VMs to run all of our tests - this is all available through the flowGraphTable

            Yep, I added that last year - glad it's handy.

            Anyway, the wfapi API described here is provided by the Pipeline Stage View plugin which isn't getting a lot development priority at the moment.  The depth parameter isn't – and won't – be supported, because it's based on the standard Jenkins API mechanism. 

            However, what I do hear here is "we need a way to get detailed information about node uses" and I think that's a reasonable request. 

            Show
            svanoort Sam Van Oort added a comment - > Our goal is to find the total time it takes for all the VMs to run all of our tests - this is all available through the flowGraphTable Yep, I added that last year - glad it's handy. Anyway, the wfapi API described here is provided by the Pipeline Stage View plugin which isn't getting a lot development priority at the moment.  The depth parameter isn't – and won't – be supported, because it's based on the standard Jenkins API mechanism.  However, what I do hear here is "we need a way to get detailed information about node uses" and I think that's a reasonable request. 
            Hide
            svanoort Sam Van Oort added a comment -

            What's being asked here won't be done – but I think there's a really good argument for JENKINS-32225 in a more general form (provide an API to get node usage information out of Pipeline more directly).

            Natesh Kukreja What I'd appreciate if you could do is add a short comment to that JIRA that describes the information you'd want out of a Pipeline about nodes and how you'd use it.

            Show
            svanoort Sam Van Oort added a comment - What's being asked here won't be done – but I think there's a really good argument for JENKINS-32225 in a more general form (provide an API to get node usage information out of Pipeline more directly). Natesh Kukreja What I'd appreciate if you could do is add a short comment to that JIRA that describes the information you'd want out of a Pipeline about nodes and how you'd use it.
            svanoort Sam Van Oort made changes -
            Link This issue duplicates JENKINS-32225 [ JENKINS-32225 ]
            svanoort Sam Van Oort made changes -
            Status Open [ 1 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Hide
            nateshkukreja Natesh Kukreja added a comment -

            For sure, thanks!

            Show
            nateshkukreja Natesh Kukreja added a comment - For sure, thanks!

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                nateshkukreja Natesh Kukreja
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: