Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17170

Jenkins auto install tools feature floods Nexus of HTTP request

    Details

    • Similar Issues:

      Description

      I've configured automatic tools installation (JDK, Maven, Git, ...) in Jenkins with "Extract .zip/.tar" feature pointing to a Nexus URL (like this : "http://nexus:8081/nexus/content/repositories/jenkins-ci-tools/fr/mipih/ic/tools/jdk/oracle-hotspot/1.6.0_25_64bits/oracle-hotspot-1.6.0_25_64bits-linux.zip").

      An example of JDK tools declaration in config.xml is in attachment (config-JDK-installs.txt).

      The problem is that Jenkins floods Nexus of HTTP requests to download some configured tools every minute.

      This feature has been improved by https://issues.jenkins-ci.org/browse/JENKINS-16215 in the version 1.500 of Jenkins but Nexus still faces with a flood of requests returning 304 response.

      I've activated access logs of Nexus (nexus-access-log.txt).
      Each minute, between 26 and 28 seconds, Jenkins creates more than 130 HTTP requests trying to download some tools parametered in global configuration.
      I've 13 tools configured in global configuration.

      96 HTTP requests only for oracle-hotspot-1.6.0_25_64bits-linux.zip in 3 seconds.

      Each time, the repartition of HTTP requests is the same :
      96 - oracle-hotspot-1.6.0_25_64bits-linux.zip
      22 - maven-2.2.1.zip
      5 - maven-3.0.3.zip
      3 - ibm-j9-1.5.0_sr8a_32bits.zip
      3 - git-1.7.5.1-aix.zip
      1 - maven-3.0.4.zip
      1 - oracle-hotspot-1.5.0_22_64bits-linux.zip
      1 - oracle-hotspot-1.6.0_26_32bits-windows.zip

      I've created this ticket with Critical priority because Nexus is overloaded each day.

      I've tried to look in Jenkins core but I cannot see why Jenkins is trying to download again and again these tools. Any idea ?

      Jenkins is supposed to call the auto-installer feature only when a build which need a non installed tool is triggered right ? Are there any other cases ?

      Thanks in advance!

        Attachments

          Issue Links

            Activity

            Hide
            evernat evernat added a comment -

            Is it reproduced with a recent Jenkins version?

            Show
            evernat evernat added a comment - Is it reproduced with a recent Jenkins version?
            Hide
            mpapo Michael Pailloncy added a comment -

            Yes. Current version : 1.531

            Show
            mpapo Michael Pailloncy added a comment - Yes. Current version : 1.531
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Slawomir Jodynski
            Path:
            src/main/java/hudson/plugins/groovy/SystemGroovy.java
            http://jenkins-ci.org/commit/groovy-plugin/612689f61c596a6065c833fa036b3a9c3d2ae689
            Log:
            fixes JENKINS-17171 - token macro expansion not working

            see https://issues.jenkins-ci.org/browse/JENKINS-17170

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Slawomir Jodynski Path: src/main/java/hudson/plugins/groovy/SystemGroovy.java http://jenkins-ci.org/commit/groovy-plugin/612689f61c596a6065c833fa036b3a9c3d2ae689 Log: fixes JENKINS-17171 - token macro expansion not working see https://issues.jenkins-ci.org/browse/JENKINS-17170
            Hide
            danielbeck Daniel Beck added a comment -

            Untested, but based on the code this is still unresolved in ~1.580.

            Whenever something asks for the home directory of a tool on a node, all ToolLocationTranslator are queried for getToolHome(...). InstallerTranslator is one of them. It actually runs ToolInstaller.performInstallation(...) when doing so, which in the case of ZipExtractionInstaller calls FilePath.installIfNecessaryFrom(...) and that ultimately open a URLConnection and checks whether getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED.

            Show
            danielbeck Daniel Beck added a comment - Untested, but based on the code this is still unresolved in ~1.580. Whenever something asks for the home directory of a tool on a node, all ToolLocationTranslator are queried for getToolHome(...) . InstallerTranslator is one of them. It actually runs ToolInstaller.performInstallation(...) when doing so, which in the case of ZipExtractionInstaller calls FilePath.installIfNecessaryFrom(...) and that ultimately open a URLConnection and checks whether getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED .
            Show
            integer Kanstantsin Shautsou added a comment - Weird. https://github.com/jenkinsci/jenkins/blob/70f5968160b0a1cf81dc451fd2aacf1e0f9201ea/core/src/main/java/hudson/tools/DownloadFromUrlInstaller.java#L67 has check before. Both DownloadFromUrlInstaller and ZipExtractionInstaller do the same: "download and extract". JDKInstaller also has such check https://github.com/jenkinsci/jenkins/blob/70f5968160b0a1cf81dc451fd2aacf1e0f9201ea/core/src/main/java/hudson/tools/JDKInstaller.java#L126
            Hide
            danielbeck Daniel Beck added a comment -

            Kanstantsin Shautsou: AFAIU the difference is that DownloadFromURL installer is for publicly available distributions based on files with specific assigned version numbers, while ZipExtraction can use arbitrary user-provided paths, including e.g. "currentTools.zip", a file updated whenever there's a change. So the former can rather safely assume that something downloaded once doesn't need to be redownloaded, while the latter cannot.

            Show
            danielbeck Daniel Beck added a comment - Kanstantsin Shautsou : AFAIU the difference is that DownloadFromURL installer is for publicly available distributions based on files with specific assigned version numbers, while ZipExtraction can use arbitrary user-provided paths, including e.g. "currentTools.zip", a file updated whenever there's a change. So the former can rather safely assume that something downloaded once doesn't need to be redownloaded, while the latter cannot.
            Hide
            integer Kanstantsin Shautsou added a comment -

            So we need have some checkbox to control this behavior.

            Show
            integer Kanstantsin Shautsou added a comment - So we need have some checkbox to control this behavior.
            Hide
            danielbeck Daniel Beck added a comment -

            Might be sufficient to cache any given URL's state after a check for ~10 seconds. One request per Jenkins instance and URL per 10 seconds should be reasonable.

            Show
            danielbeck Daniel Beck added a comment - Might be sufficient to cache any given URL's state after a check for ~10 seconds. One request per Jenkins instance and URL per 10 seconds should be reasonable.

              People

              • Assignee:
                Unassigned
                Reporter:
                mpapo Michael Pailloncy
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: