Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40808

Factor Docker fixtures out of acceptance-test-harness for use in functional tests

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Lots of plugin functional tests using JenkinsRule assume some kind of software environment on the testing host, typically for running external processes (for example SCM tools), but this is rarely explicit. That makes it complicated to assure that both developer machines and CI builders have all required software packages installed and in the correct versions and configurations.

      The acceptance-test-harness includes a very useful DockerFixture system which addresses this for acceptance tests. But acceptance tests are slow, flaky, and external to plugin sources, so most test code is written using functional tests.

      We need to be able to reuse the same system in both kinds of tests.

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          Jenkinsfile
          pom.xml
          src/main/java/hudson/plugins/mercurial/MercurialInstallation.java
          src/main/resources/hudson/plugins/mercurial/MercurialInstallation/help-executable.html
          src/test/java/hudson/plugins/mercurial/ChangeComparatorTest.java
          src/test/java/hudson/plugins/mercurial/CustomConfigTest.java
          src/test/java/hudson/plugins/mercurial/DisableChangeLogTest.java
          src/test/java/hudson/plugins/mercurial/EnvVarTest.java
          src/test/java/hudson/plugins/mercurial/FunctionalTest.java
          src/test/java/hudson/plugins/mercurial/HgExeFunctionalTest.java
          src/test/java/hudson/plugins/mercurial/MatrixProjectTest.java
          src/test/java/hudson/plugins/mercurial/MercurialContainer.java
          src/test/java/hudson/plugins/mercurial/MercurialContainerTest.java
          src/test/java/hudson/plugins/mercurial/MercurialRule.java
          src/test/java/hudson/plugins/mercurial/MercurialSampleRepoRule.java
          src/test/java/hudson/plugins/mercurial/PipelineTest.java
          src/test/java/hudson/plugins/mercurial/SCMTestBase.java
          src/test/resources/hudson/plugins/mercurial/MercurialContainer/Dockerfile
          http://jenkins-ci.org/commit/mercurial-plugin/2903930fd367b73312f305b9c82b356162493ae0
          Log:
          Merge pull request #93 from jglick/docker-fixtures

          JENKINS-40808 Using docker-fixtures repository

          Compare: https://github.com/jenkinsci/mercurial-plugin/compare/5cc6e1c2b06b...2903930fd367

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile pom.xml src/main/java/hudson/plugins/mercurial/MercurialInstallation.java src/main/resources/hudson/plugins/mercurial/MercurialInstallation/help-executable.html src/test/java/hudson/plugins/mercurial/ChangeComparatorTest.java src/test/java/hudson/plugins/mercurial/CustomConfigTest.java src/test/java/hudson/plugins/mercurial/DisableChangeLogTest.java src/test/java/hudson/plugins/mercurial/EnvVarTest.java src/test/java/hudson/plugins/mercurial/FunctionalTest.java src/test/java/hudson/plugins/mercurial/HgExeFunctionalTest.java src/test/java/hudson/plugins/mercurial/MatrixProjectTest.java src/test/java/hudson/plugins/mercurial/MercurialContainer.java src/test/java/hudson/plugins/mercurial/MercurialContainerTest.java src/test/java/hudson/plugins/mercurial/MercurialRule.java src/test/java/hudson/plugins/mercurial/MercurialSampleRepoRule.java src/test/java/hudson/plugins/mercurial/PipelineTest.java src/test/java/hudson/plugins/mercurial/SCMTestBase.java src/test/resources/hudson/plugins/mercurial/MercurialContainer/Dockerfile http://jenkins-ci.org/commit/mercurial-plugin/2903930fd367b73312f305b9c82b356162493ae0 Log: Merge pull request #93 from jglick/docker-fixtures JENKINS-40808 Using docker-fixtures repository Compare: https://github.com/jenkinsci/mercurial-plugin/compare/5cc6e1c2b06b...2903930fd367
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/test/acceptance/docker/Docker.java
          src/main/java/org/jenkinsci/test/acceptance/docker/DockerContainer.java
          src/main/java/org/jenkinsci/test/acceptance/docker/DockerFixture.java
          src/main/java/org/jenkinsci/test/acceptance/docker/DockerImage.java
          src/main/java/org/jenkinsci/test/acceptance/docker/DynamicDockerContainer.java
          src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/JavaContainer.java
          src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer.java
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/JavaContainer/Dockerfile
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/Dockerfile
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe.pub
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key.pub
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key_passphrase.txt
          src/test/java/org/jenkinsci/test/acceptance/docker/DockerFixtureTest.java
          src/test/java/org/jenkinsci/test/acceptance/docker/DockerImageTest.java
          src/test/java/org/jenkinsci/test/acceptance/docker/DynamicDockerContainerTest.java
          src/test/resources/org/jenkinsci/test/acceptance/docker/DockerFixtureTest/TestContainer/Dockerfile
          src/test/resources/test/Dockerfile
          http://jenkins-ci.org/commit/acceptance-test-harness/d682a3da859cc67ea86749642b6b1a102256f25f
          Log:
          Merge pull request #252 from jglick/docker-fixtures

          JENKINS-40808 Using docker-fixtures repository

          Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/c26b4e0cec68...d682a3da859c

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/test/acceptance/docker/Docker.java src/main/java/org/jenkinsci/test/acceptance/docker/DockerContainer.java src/main/java/org/jenkinsci/test/acceptance/docker/DockerFixture.java src/main/java/org/jenkinsci/test/acceptance/docker/DockerImage.java src/main/java/org/jenkinsci/test/acceptance/docker/DynamicDockerContainer.java src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/JavaContainer.java src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer.java src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/JavaContainer/Dockerfile src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/Dockerfile src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe.pub src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key.pub src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshdContainer/unsafe_enc_key_passphrase.txt src/test/java/org/jenkinsci/test/acceptance/docker/DockerFixtureTest.java src/test/java/org/jenkinsci/test/acceptance/docker/DockerImageTest.java src/test/java/org/jenkinsci/test/acceptance/docker/DynamicDockerContainerTest.java src/test/resources/org/jenkinsci/test/acceptance/docker/DockerFixtureTest/TestContainer/Dockerfile src/test/resources/test/Dockerfile http://jenkins-ci.org/commit/acceptance-test-harness/d682a3da859cc67ea86749642b6b1a102256f25f Log: Merge pull request #252 from jglick/docker-fixtures JENKINS-40808 Using docker-fixtures repository Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/c26b4e0cec68...d682a3da859c
          Hide
          jglick Jesse Glick added a comment -

          Andrew Bayer

          tests that run builds running Docker have problems on OS X

          Here I am only porting something that has been in broad use for a long time in acceptance-test-harness, so if those tests passed for you then this should too.

          Show
          jglick Jesse Glick added a comment - Andrew Bayer tests that run builds running Docker have problems on OS X Here I am only porting something that has been in broad use for a long time in acceptance-test-harness , so if those tests passed for you then this should too.
          Hide
          abayer Andrew Bayer added a comment -

          It's worth noting that JenkinsRule tests that run builds running Docker have problems on OS X due to symlinking of /tmp. Not sure if that'd be a problem here, though.

          Show
          abayer Andrew Bayer added a comment - It's worth noting that JenkinsRule tests that run builds running Docker have problems on OS X due to symlinking of /tmp. Not sure if that'd be a problem here, though.
          Hide
          jglick Jesse Glick added a comment -

          I guess the best workaround is to use a tagged image from DockerHub and only add those things to Dockerfile which seem version-independent.

          Show
          jglick Jesse Glick added a comment - I guess the best workaround is to use a tagged image from DockerHub and only add those things to Dockerfile which seem version-independent.
          Hide
          jglick Jesse Glick added a comment -

          Another problem I find is that existing fixtures are not set up to define exact, reproducible software versions. In fact it seems this is impossible to do on Ubuntu, since the package repositories do not host historical versions. For a particular software product you could do an explicit build, though this seems clumsy.

          Show
          jglick Jesse Glick added a comment - Another problem I find is that existing fixtures are not set up to define exact, reproducible software versions. In fact it seems this is impossible to do on Ubuntu, since the package repositories do not host historical versions. For a particular software product you could do an explicit build, though this seems clumsy.
          Hide
          jglick Jesse Glick added a comment -

          In the case of mercurial-plugin it turns out that merely adding fixtures does not absolve you of the need to have hg installed to run most tests, since many are either designed to run on master specifically, or do or could use an agent but would still need to run hg to implement caching—and JenkinsRule cannot run the master in a container. (ATH can, but not at the discretion of the test, so it does not do much better in this regard.)

          Show
          jglick Jesse Glick added a comment - In the case of mercurial-plugin it turns out that merely adding fixtures does not absolve you of the need to have hg installed to run most tests, since many are either designed to run on master specifically, or do or could use an agent but would still need to run hg to implement caching—and JenkinsRule cannot run the master in a container. (ATH can, but not at the discretion of the test, so it does not do much better in this regard.)
          Hide
          jglick Jesse Glick added a comment -

          An alternative would be to use Arquillian Cube Standalone as in this demo by Alex Soto/Alex Soto. The upside is of course use of a third-party library which is presumably maintained by others. Downsides:

          • learning curve for Jenkins developers already familiar with DockerFixture
          • intrusive @RunWith; unclear whether that is compatible with JenkinsRule
          • more verbose configuration, including what looks like YAML inside of XML
          • does not seem to handle IP address injection (yet?)

          Closer to the spirit of DockerFixture would be junit-docker-rule or docker-junit-rule, neither of which seems to support building images on demand from Dockerfile; or docker-compose-rule, which sounds good for multi-container situations, though it imposes an additional requirement that Compose be installed.

          Also none of these seem to have any equivalent to ATH’s ability to subclass DockerContainer to define reusable, type-safe container classes.

          Show
          jglick Jesse Glick added a comment - An alternative would be to use Arquillian Cube Standalone as in this demo by Alex Soto / Alex Soto . The upside is of course use of a third-party library which is presumably maintained by others. Downsides: learning curve for Jenkins developers already familiar with DockerFixture intrusive @RunWith ; unclear whether that is compatible with JenkinsRule more verbose configuration, including what looks like YAML inside of XML does not seem to handle IP address injection (yet?) Closer to the spirit of DockerFixture would be junit-docker-rule or docker-junit-rule , neither of which seems to support building images on demand from Dockerfile ; or docker-compose-rule , which sounds good for multi-container situations, though it imposes an additional requirement that Compose be installed. Also none of these seem to have any equivalent to ATH’s ability to subclass DockerContainer to define reusable, type-safe container classes.

            People

            • Assignee:
              jglick Jesse Glick
              Reporter:
              jglick Jesse Glick
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: