Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17887

Jenkins TAP plugin should scale to millions of tests in thousands of test sets

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

      My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

      prove --archive $WORKSPACE/tap-output t

      And passing that on to Jenkins TAP plugin, we see:

      Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

      I'd suggest setting these goals:

      • minimise on-disk storage of TAP output & parsed TAP results
      • minimise load-time of TAP Results & TAP Test Report pages
      • enhance usability of these pages

      Here are a few ideas:

      • Don't pre-render TAP results
        • At a glance, I don't think the Jenkins TAP plugin is, but included just in case
        • Avoids: large files on disk of pre-rendered content
      • Render results client-side
        • Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
        • Render test-set summaries by default
        • Only render test-set details on request
          • Avoids: a high up-front load time for large result sets
        • Gracefully degrade to individual pages
      • Introduce a filter for failed tests
      • Stop storing parsed TAP results in build.xml
        • Move to an sqlite db?
        • At least: remove extra whitespace
      • Compress everything in tap-master-files
        • deliver it to client compressed to minimise over-the-wire transfer size?

      See also:

      • JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.

        Attachments

          Issue Links

            Activity

            Hide
            kinow Bruno P. Kinoshita added a comment -

            +1, sounds doable and very reasonable.

            I think the last release is bugged, so I'll try to fix all the bugs, cut a bugfix release and include this issue in a major release. Hopefully I'll manage release both versions within three or four weeks.

            Thanks Steve.

            Show
            kinow Bruno P. Kinoshita added a comment - +1, sounds doable and very reasonable. I think the last release is bugged, so I'll try to fix all the bugs, cut a bugfix release and include this issue in a major release. Hopefully I'll manage release both versions within three or four weeks. Thanks Steve.
            Hide
            toddr Todd Rinaldo added a comment - - edited

            Our test count is close to 100,000

            FWIW, I don't really care so much about the passing tests. Most of the time, I'd like to filter out either all the files that pass or sometimes, I just want ONLY the tests that didn't pass, not the whole files. Note: I don't say failed, rather didn't pass. Reason being that interesting things to look at are:
            (In order of importance)
            1. Failed
            2. TODO Passed
            3. TODO Failed
            4. Skipped

            Also turning captured debug comments on/off might be nice.

            Show
            toddr Todd Rinaldo added a comment - - edited Our test count is close to 100,000 FWIW, I don't really care so much about the passing tests. Most of the time, I'd like to filter out either all the files that pass or sometimes, I just want ONLY the tests that didn't pass, not the whole files. Note: I don't say failed, rather didn't pass. Reason being that interesting things to look at are: (In order of importance) 1. Failed 2. TODO Passed 3. TODO Failed 4. Skipped Also turning captured debug comments on/off might be nice.
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hi @Steve

            I'm preparing to cut a new release of tap-plugin and unfortunately I won't have enough time to fix your issue.

            But I've re-read it and now think I understand it better. Probably I will create sub issues for some of your suggestions, and will try to include them in a next release.

            @Todd
            Created https://issues.jenkins-ci.org/browse/JENKINS-19472 to work on filters. I'm linking that issue to this one too, but probably you'll want to watch that one too.

            > Also turning captured debug comments on/off might be nice.

            It's disabled by default now, but you can turn it on under the advanced panel of the TAP post build step.

            Thanks!
            Bruno

            Show
            kinow Bruno P. Kinoshita added a comment - Hi @Steve I'm preparing to cut a new release of tap-plugin and unfortunately I won't have enough time to fix your issue. But I've re-read it and now think I understand it better. Probably I will create sub issues for some of your suggestions, and will try to include them in a next release. @Todd Created https://issues.jenkins-ci.org/browse/JENKINS-19472 to work on filters. I'm linking that issue to this one too, but probably you'll want to watch that one too. > Also turning captured debug comments on/off might be nice. It's disabled by default now, but you can turn it on under the advanced panel of the TAP post build step. Thanks! Bruno
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Actually, there's an existing issue for filters.

            https://issues.jenkins-ci.org/browse/JENKINS-17804

            Show
            kinow Bruno P. Kinoshita added a comment - Actually, there's an existing issue for filters. https://issues.jenkins-ci.org/browse/JENKINS-17804

              People

              • Assignee:
                kinow Bruno P. Kinoshita
                Reporter:
                spurkis Steve Purkis
              • Votes:
                3 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: