Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-1447

https://repo.jenkins-ci.org/ is slow and or unreachable

    Details

    • Similar Issues:

      Description

      Reported this morning by Oleg Nenashev

      {quote}

      [1:33 PM] Oleg Nenashev: @here could somebody with Jenkins INFRA access permissions take a look at https://repo.jenkins-ci.org/ performance? It blocks PCT tests which I need to perform as a part of JEP-200
      [1:34 PM] Oleg Nenashev: Merry Christmas though
      [1:34 PM] Oleg Nenashev: Likely nobody is online, CC @aheritier just in case

      {quote}

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment - - edited

          Artifactory seems to be holding up well since I posted the previous comment. We have regular Maven index updates every 100 minutes or so, and there have been no outages I could confirm (a single request of those I send once per minute failed, but that could have been my connection).

          Show
          danielbeck Daniel Beck added a comment - - edited Artifactory seems to be holding up well since I posted the previous comment. We have regular Maven index updates every 100 minutes or so, and there have been no outages I could confirm (a single request of those I send once per minute failed, but that could have been my connection).
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          It works for me well, I was able to release some plugins, more to come. Thanks a lot!

          Show
          oleg_nenashev Oleg Nenashev added a comment - It works for me well, I was able to release some plugins, more to come. Thanks a lot!
          Hide
          aheritier Arnaud Héritier added a comment -

          From JFrog

          Hi Daniel,

          Thank you for your patience. We want to share some of our findings with you. We have noticed that the load on your server has increased, and we are still looking into the exact root cause. We do know with certainty that the load is related to what appears to be a script that runs on your side, that updates/creates Permission Targets via the api/security/permissions endpoint, using a user with username 'permission-updater'. We've noticed that this script is running every 30 minutes, and this, along with repository browsing that happens constantly in the background, causes the load on your server to spike every 30 minutes. We are still looking into the root cause of the increase in load, to see if it's related to a change in one of the recent versions of Artifactory. 

          In the meantime, can you kindly elaborate on the use-case behind updating the permission targets every 30 minutes? additionally, if possible, please consider staggering the executions of this script as a short-term remediation strategy, at least until we complete our RCA.    

          Best regards,
          Uriah Levy

           

          Show
          aheritier Arnaud Héritier added a comment - From JFrog Hi Daniel, Thank you for your patience. We want to share some of our findings with you. We have noticed that the load on your server has increased, and we are still looking into the exact root cause. We do know with certainty that the load is related to what appears to be a script that runs on your side, that updates/creates Permission Targets via the api/security/permissions endpoint, using a user with username 'permission-updater'. We've noticed that this script is running every 30 minutes, and this, along with repository browsing that happens constantly in the background, causes the load on your server to spike every 30 minutes. We are still looking into the root cause of the increase in load, to see if it's related to a change in one of the recent versions of Artifactory.  In the meantime, can you kindly elaborate on the use-case behind updating the permission targets every 30 minutes? additionally, if possible, please consider staggering the executions of this script as a short-term remediation strategy, at least until we complete our RCA.     Best regards, Uriah Levy  
          Hide
          aheritier Arnaud Héritier added a comment -

          Daniel Beck it seems that our update permissions script could be a part of the issue ...

          Show
          aheritier Arnaud Héritier added a comment - Daniel Beck it seems that our update permissions script could be a part of the issue ...
          Hide
          rtyler R. Tyler Croy added a comment -

          Going to go ahead and close this since normal service appears to have resumed

          Show
          rtyler R. Tyler Croy added a comment - Going to go ahead and close this since normal service appears to have resumed

            People

            • Assignee:
              danielbeck Daniel Beck
              Reporter:
              aheritier Arnaud Héritier
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: