Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33730

ATH needs to be made to work with 2.0

    Details

    • Similar Issues:

      Description

      Acceptance test harness needs to be checked whether it works with the 2.0 targeting changes, and if not, needs to be adapted to work with the 2.0 changes.

        Attachments

          Issue Links

            Activity

            Hide
            swashbuck1r Spike Washburn added a comment -

            Daniel expects problems need to be addressed during the beta period if possible.

            Show
            swashbuck1r Spike Washburn added a comment - Daniel expects problems need to be addressed during the beta period if possible.
            Hide
            swashbuck1r Spike Washburn added a comment -

            Root cause of this: the UI was changed, and broke many of the ATH tests. [Guess] Selenium uses page object to describe the page, and many of the tests rely on ID that have changed.

            [Suggest Tom or Keith fix up the tests]

            Show
            swashbuck1r Spike Washburn added a comment - Root cause of this: the UI was changed, and broke many of the ATH tests. [Guess] Selenium uses page object to describe the page, and many of the tests rely on ID that have changed. [Suggest Tom or Keith fix up the tests]
            Hide
            drulli Ulli Hafner added a comment - - edited

            Wouldn't it make sense to run the ATH after each UI change to ensure that nothing has been broken? I would also encourage each developer to add new ATH test cases for each of the new UI features that have been introduced...

            Show
            drulli Ulli Hafner added a comment - - edited Wouldn't it make sense to run the ATH after each UI change to ensure that nothing has been broken? I would also encourage each developer to add new ATH test cases for each of the new UI features that have been introduced...
            Hide
            danielbeck Daniel Beck added a comment -

            Ulli Hafner Definitely something we need to improve. ATH is not visible enough. We could make passing ATH in an ATH snapshot/PR a condition for UI changes. Ideally ATH could be run for all pull requests. WDYT?

            For now, I consider this a 2.0 blocker that needs fixing. So from the POV of actual Jenkins releases, we'll still be on time

            Show
            danielbeck Daniel Beck added a comment - Ulli Hafner Definitely something we need to improve. ATH is not visible enough. We could make passing ATH in an ATH snapshot/PR a condition for UI changes. Ideally ATH could be run for all pull requests. WDYT? For now, I consider this a 2.0 blocker that needs fixing. So from the POV of actual Jenkins releases, we'll still be on time
            Hide
            drulli Ulli Hafner added a comment -

            Letting the ATH run for each PR would be ideal, of course. (I'm not sure if this is feasible since one run of the ATH is quite time consuming...)

            Since all core changes require a code review it would make sense to make at least one ATH test case mandatory to get a 'bee' label in such a review (and a couple of unit tests

            Show
            drulli Ulli Hafner added a comment - Letting the ATH run for each PR would be ideal, of course. (I'm not sure if this is feasible since one run of the ATH is quite time consuming...) Since all core changes require a code review it would make sense to make at least one ATH test case mandatory to get a 'bee' label in such a review (and a couple of unit tests
            Hide
            tfennelly Tom FENNELLY added a comment -

            So there's 2 immediately obvious issues ...

            1. Install wizard needs to be disabled for all/most tests.
            2. No bundled plugins. Looks like we need to get these in there somehow.

            #2 above is the trickier one of fix. Maybe we need the acceptance test harness to get all of these plugins automatically and install them in the $JENKINS_HOME/plugins ?

            Show
            tfennelly Tom FENNELLY added a comment - So there's 2 immediately obvious issues ... Install wizard needs to be disabled for all/most tests. No bundled plugins. Looks like we need to get these in there somehow. #2 above is the trickier one of fix. Maybe we need the acceptance test harness to get all of these plugins automatically and install them in the $JENKINS_HOME/plugins ?
            Hide
            tfennelly Tom FENNELLY added a comment -

            What I've been doing ...

            1. Modified Jenkins core 2.0 to allow for an env based install-state override, allowing us to bypass the install wizard in the ATH (using LocalController.commonLaunchEnv to set the override in the test harness)
            2. Testing against alpha4 for now because the new-item page changes cause tests to fail immediately.
            Show
            tfennelly Tom FENNELLY added a comment - What I've been doing ... Modified Jenkins core 2.0 to allow for an env based install-state override, allowing us to bypass the install wizard in the ATH (using LocalController.commonLaunchEnv to set the override in the test harness) Testing against alpha4 for now because the new-item page changes cause tests to fail immediately.
            Hide
            danielbeck Daniel Beck added a comment -

            Modified Jenkins core 2.0 to allow for an env based install-state override

            Is this on top of the existing jenkins.install.runSetupWizard and hudson.Main.development System properties that control setup wizard appearance? Can they maybe be consolidated?

            Show
            danielbeck Daniel Beck added a comment - Modified Jenkins core 2.0 to allow for an env based install-state override Is this on top of the existing jenkins.install.runSetupWizard and hudson.Main.development System properties that control setup wizard appearance? Can they maybe be consolidated?
            Hide
            olivergondza Oliver Gondža added a comment -

            Tom FENNELLY, thanks for looking at it. Can you push what you have so we can collaborate?

            Show
            olivergondza Oliver Gondža added a comment - Tom FENNELLY , thanks for looking at it. Can you push what you have so we can collaborate?
            Hide
            tfennelly Tom FENNELLY added a comment - - edited

            Oliver Gondža Hey Oliver. I pushed the small few changes I have made to ...

            Oliver Gondža I'm really stuck on an issue with the selection of the "Execute shell" option from the "Add build step" dropdown. I'm seeing an issue with it on WarningsPluginTest.should_detect_warnings_of_multiple_compilers_in_console_freestyle, where it's unable to find .//A[@href][@id='Execute shell' or normalize-space(.)='Execute shell' or @title='Execute shell' or .//img[@alt='Execute shell']].

            When I run it in my IDE, setting a breakpoint where the findElement fails, I'm able to go into the dev console in the browser and execute...

            $x(".//A[@href][@id='Execute shell' or normalize-space(.)='Execute shell' or @title='Execute shell' or .//img[@alt='Execute shell']]")
            

            Sure enough, when I execute this the first time it returns an empy array. But ... when I manually click on the "Add build step" dropdown I can actually see the "Execute shell" option + when I then go back to the dev console and execute the above xpath it actually works the second time i.e. it finds it.

            See maybe it's some sort of timing issue? Still not sure how the config page changes have caused this.

            Show
            tfennelly Tom FENNELLY added a comment - - edited Oliver Gondža Hey Oliver. I pushed the small few changes I have made to ... https://github.com/jenkinsci/jenkins/pull/2196 (don't worry Daniel Beck ... we're not going to merge it ) https://github.com/jenkinsci/acceptance-test-harness/pull/87 Oliver Gondža I'm really stuck on an issue with the selection of the "Execute shell" option from the "Add build step" dropdown. I'm seeing an issue with it on WarningsPluginTest.should_detect_warnings_of_multiple_compilers_in_console_freestyle , where it's unable to find .//A [@href] [@id='Execute shell' or normalize-space(.)='Execute shell' or @title='Execute shell' or .//img [@alt='Execute shell'] ] . When I run it in my IDE, setting a breakpoint where the findElement fails, I'm able to go into the dev console in the browser and execute... $x( ". //A[@href][@id= 'Execute shell' or normalize-space(.)= 'Execute shell' or @title= 'Execute shell' or .//img[@alt= 'Execute shell' ]]" ) Sure enough, when I execute this the first time it returns an empy array. But ... when I manually click on the "Add build step" dropdown I can actually see the "Execute shell" option + when I then go back to the dev console and execute the above xpath it actually works the second time i.e. it finds it. See maybe it's some sort of timing issue? Still not sure how the config page changes have caused this.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža this is funky I think part of the problem is that the Add build step dropdown button is no longer in view after a few steps have been added.

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža this is funky I think part of the problem is that the Add build step dropdown button is no longer in view after a few steps have been added.
            Hide
            olivergondza Oliver Gondža added a comment - - edited

            I see there are some features getting into our way. Security by default seems to be the first one. Is there any way to launch Jenkins without that or inject the key somehow? Cc Daniel Beck.

            https://jenkins.ci.cloudbees.com/job/core/job/acceptance-test-harness-2.0/59/console

            Show
            olivergondza Oliver Gondža added a comment - - edited I see there are some features getting into our way. Security by default seems to be the first one. Is there any way to launch Jenkins without that or inject the key somehow? Cc Daniel Beck . https://jenkins.ci.cloudbees.com/job/core/job/acceptance-test-harness-2.0/59/console
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža I think I resolved my earlier issue with the Add build step. See 0b9dfaf. Maybe you know of the right way of doing this or if it's actually been caused by something else that can be fixed?

            As for the security error you are seeing in the CI build ... is that not because it's not running against a Jenkins 2.0 that has https://github.com/jenkinsci/jenkins/pull/2196 ??

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža I think I resolved my earlier issue with the Add build step. See 0b9dfaf . Maybe you know of the right way of doing this or if it's actually been caused by something else that can be fixed? As for the security error you are seeing in the CI build ... is that not because it's not running against a Jenkins 2.0 that has https://github.com/jenkinsci/jenkins/pull/2196 ??
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            core/src/main/java/jenkins/install/InstallUtil.java
            http://jenkins-ci.org/commit/jenkins/9999516aa826dd34986204a508a6bda7124e889d
            Log:
            Merge pull request #2196 from tfennelly/2.0-JENKINS-33730

            JENKINS-33730 Fix ATH for Jenkins 2.0

            Compare: https://github.com/jenkinsci/jenkins/compare/4d01c07db60d...9999516aa826

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: core/src/main/java/jenkins/install/InstallUtil.java http://jenkins-ci.org/commit/jenkins/9999516aa826dd34986204a508a6bda7124e889d Log: Merge pull request #2196 from tfennelly/2.0- JENKINS-33730 JENKINS-33730 Fix ATH for Jenkins 2.0 Compare: https://github.com/jenkinsci/jenkins/compare/4d01c07db60d...9999516aa826
            Hide
            olivergondza Oliver Gondža added a comment -

            This got the most of the ATH running. However, there are still things to fix. A lot of tests require presence of formerly bundled plugins. Th easiest fix seems to be just add the annotations or alternatively to make the dependency resolution mechanism aware of unbundling in similar way we do in core.

            Show
            olivergondza Oliver Gondža added a comment - This got the most of the ATH running. However, there are still things to fix. A lot of tests require presence of formerly bundled plugins. Th easiest fix seems to be just add the annotations or alternatively to make the dependency resolution mechanism aware of unbundling in similar way we do in core.
            Hide
            olivergondza Oliver Gondža added a comment -

            Tom FENNELLY, working on tools config page extraction support ...

            Show
            olivergondza Oliver Gondža added a comment - Tom FENNELLY , working on tools config page extraction support ...
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža Another PR: https://github.com/jenkinsci/acceptance-test-harness/pull/88

            Should fix some of those scrolling issues + sort out the missing dependencies.

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža Another PR: https://github.com/jenkinsci/acceptance-test-harness/pull/88 Should fix some of those scrolling issues + sort out the missing dependencies.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža hey Oliver ... any luck extracting the tools ?

            I just tried doing it myself here, but not really sure what I'm at since I don't know this code very well. I'm using the PmdPluginTest and it gets a bit further than it was getting, but stalls (and eventually times out) when doing the build just after extracting the maven zip. Will look again in the morning if you haven't had time to look into it.

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža hey Oliver ... any luck extracting the tools ? I just tried doing it myself here , but not really sure what I'm at since I don't know this code very well. I'm using the PmdPluginTest and it gets a bit further than it was getting, but stalls (and eventually times out) when doing the build just after extracting the maven zip. Will look again in the morning if you haven't had time to look into it.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža I see you sorted out the tooling. I also had a goo at that as mentioned in my last post - the problem I mentioned seemed only to be a timing issue when running in a debugger Anyway, thanks for that.

            PR 88 is running again and is about 1/2 way through it seems.

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža I see you sorted out the tooling. I also had a goo at that as mentioned in my last post - the problem I mentioned seemed only to be a timing issue when running in a debugger Anyway, thanks for that. PR 88 is running again and is about 1/2 way through it seems.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža also note that I had to add a temporary tweak to make the tooling config go to the right page since 2.0-beta is actually older than VersionNumber("2"). We can reverty to what you added after the 2.0 release.

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža also note that I had to add a temporary tweak to make the tooling config go to the right page since 2.0-beta is actually older than VersionNumber("2") . We can reverty to what you added after the 2.0 release.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/org/jenkinsci/test/acceptance/controller/LocalController.java
            src/main/java/org/jenkinsci/test/acceptance/po/ToolInstallation.java
            src/main/java/org/jenkinsci/test/acceptance/selenium/Scroller.java
            src/main/resources/org/jenkinsci/test/acceptance/selenium/scroller.js
            http://jenkins-ci.org/commit/acceptance-test-harness/c1512e4da9dc3b4dd701988dae103eaec405ec25
            Log:
            Merge pull request #88 from tfennelly/2.0-JENKINS-33730

            JENKINS-33730 More ATH tweaks for Jenkins 2.0

            Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/4de2dfc72cf2...c1512e4da9dc

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/org/jenkinsci/test/acceptance/controller/LocalController.java src/main/java/org/jenkinsci/test/acceptance/po/ToolInstallation.java src/main/java/org/jenkinsci/test/acceptance/selenium/Scroller.java src/main/resources/org/jenkinsci/test/acceptance/selenium/scroller.js http://jenkins-ci.org/commit/acceptance-test-harness/c1512e4da9dc3b4dd701988dae103eaec405ec25 Log: Merge pull request #88 from tfennelly/2.0- JENKINS-33730 JENKINS-33730 More ATH tweaks for Jenkins 2.0 Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/4de2dfc72cf2...c1512e4da9dc
            Hide
            tfennelly Tom FENNELLY added a comment -

            I should be done with this today.

            Show
            tfennelly Tom FENNELLY added a comment - I should be done with this today.
            Hide
            olivergondza Oliver Gondža added a comment -

            Tom FENNELLY, there seems to be some problem with Control#selectDropdownMenu(Unable to add build step, post build step, tool installation) but I am not able to reproduce that locally. I speculate it might take a little longer to appear in 2.0 so we are clicking the item before the menu appear but it is just a wild guess.

            Show
            olivergondza Oliver Gondža added a comment - Tom FENNELLY , there seems to be some problem with Control#selectDropdownMenu (Unable to add build step, post build step, tool installation) but I am not able to reproduce that locally. I speculate it might take a little longer to appear in 2.0 so we are clicking the item before the menu appear but it is just a wild guess.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Oliver Gondža where are you seeing the issue with Control#selectDropdownMenu ? Which test?

            Show
            tfennelly Tom FENNELLY added a comment - Oliver Gondža where are you seeing the issue with Control#selectDropdownMenu ? Which test?
            Show
            olivergondza Oliver Gondža added a comment - Tom FENNELLY for instance https://jenkins.ci.cloudbees.com/job/core/job/acceptance-test-harness-2.0/lastCompletedBuild/testReport/plugins/AnalysisCollectorPluginTest/should_compute_all_new_open_tasks/
            Hide
            tfennelly Tom FENNELLY added a comment -

            Hmmm I've run that a few times now and it passed each time.

            Show
            tfennelly Tom FENNELLY added a comment - Hmmm I've run that a few times now and it passed each time.
            Hide
            olivergondza Oliver Gondža added a comment -

            I am not able to reproduce that locally, either.

            Show
            olivergondza Oliver Gondža added a comment - I am not able to reproduce that locally, either.
            Hide
            tfennelly Tom FENNELLY added a comment -

            Closing this now. The only failures/errors I get now are tests that were already in fail/error when running against 1.x.

            Show
            tfennelly Tom FENNELLY added a comment - Closing this now. The only failures/errors I get now are tests that were already in fail/error when running against 1.x.

              People

              • Assignee:
                tfennelly Tom FENNELLY
                Reporter:
                danielbeck Daniel Beck
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: