Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60383

Remote nodes unable to checkout code

    Details

    • Similar Issues:
    • Released As:
      0.4.1

      Description

      Moving to 0.3.7 has broken checkout on remote nodes. They fail reporting unable to checkout the code after a 10 minute timeout.

      In the log on the gerrit server are hundreds of lines saying:

      WARN com.google.gerrit.httpd.ProjectBasicAuthFilter : Authentication failed for Username for 'https: account inactive or not provisioned in Gerrit

      So it appears that the remote node is using the wrong credentials to try to do the checkout.

       

      Reverting to .03.5 and everything works fine

        Attachments

          Issue Links

            Activity

            willwagner Will Wagner created issue -
            Hide
            willwagner Will Wagner added a comment -

            The output from the pipeline job that fails on a node is:

            using credential 6e96f729-8440-40bf-8710-88689c791d3b
            Cloning the remote Git repository
            Cloning repository https://code.office.carallon.com/pharos/magpie
            > C:\Program Files\Git\bin\git.exe init c:\jenkins\workspace\magpie-verify_01_8301_2 # timeout=10
            Fetching upstream changes from https://code.office.carallon.com/pharos/magpie
            > C:\Program Files\Git\bin\git.exe --version # timeout=10
            using GIT_ASKPASS to set credentials
            > C:\Program Files\Git\bin\git.exe fetch --tags --force --progress – https://code.office.carallon.com/pharos/magpie +refs/heads/:refs/remotes/origin/ # timeout=10
            > C:\Program Files\Git\bin\git.exe config remote.origin.url https://code.office.carallon.com/pharos/magpie # timeout=10
            > C:\Program Files\Git\bin\git.exe config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
            > C:\Program Files\Git\bin\git.exe config remote.origin.url https://code.office.carallon.com/pharos/magpie # timeout=10
            Fetching upstream changes from https://code.office.carallon.com/pharos/magpie
            using GIT_ASKPASS to set credentials
            > C:\Program Files\Git\bin\git.exe fetch --tags --force --progress – https://code.office.carallon.com/pharos/magpie refs/changes/01/8301/2:refs/remotes/origin/01/8301/2 # timeout=10
            Checking out Revision 787879583e36f6e3afc5b60d35d15409d2c6a11a (refs/remotes/origin/01/8301/2)
            > C:\Program Files\Git\bin\git.exe rev-parse "01/8301/2^

            {commit}" # timeout=10
            > C:\Program Files\Git\bin\git.exe rev-parse "refs/remotes/origin/01/8301/2^{commit}

            " # timeout=10
            > C:\Program Files\Git\bin\git.exe config core.sparsecheckout # timeout=10
            > C:\Program Files\Git\bin\git.exe checkout -f 787879583e36f6e3afc5b60d35d15409d2c6a11a # timeout=10
            ERROR: Timeout after 10 minutes
             

            Show
            willwagner Will Wagner added a comment - The output from the pipeline job that fails on a node is: using credential 6e96f729-8440-40bf-8710-88689c791d3b Cloning the remote Git repository Cloning repository https://code.office.carallon.com/pharos/magpie > C:\Program Files\Git\bin\git.exe init c:\jenkins\workspace\magpie-verify_01_8301_2 # timeout=10 Fetching upstream changes from https://code.office.carallon.com/pharos/magpie > C:\Program Files\Git\bin\git.exe --version # timeout=10 using GIT_ASKPASS to set credentials > C:\Program Files\Git\bin\git.exe fetch --tags --force --progress – https://code.office.carallon.com/pharos/magpie +refs/heads/ :refs/remotes/origin/ # timeout=10 > C:\Program Files\Git\bin\git.exe config remote.origin.url https://code.office.carallon.com/pharos/magpie # timeout=10 > C:\Program Files\Git\bin\git.exe config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 > C:\Program Files\Git\bin\git.exe config remote.origin.url https://code.office.carallon.com/pharos/magpie # timeout=10 Fetching upstream changes from https://code.office.carallon.com/pharos/magpie using GIT_ASKPASS to set credentials > C:\Program Files\Git\bin\git.exe fetch --tags --force --progress – https://code.office.carallon.com/pharos/magpie refs/changes/01/8301/2:refs/remotes/origin/01/8301/2 # timeout=10 Checking out Revision 787879583e36f6e3afc5b60d35d15409d2c6a11a (refs/remotes/origin/01/8301/2) > C:\Program Files\Git\bin\git.exe rev-parse "01/8301/2^ {commit}" # timeout=10 > C:\Program Files\Git\bin\git.exe rev-parse "refs/remotes/origin/01/8301/2^{commit} " # timeout=10 > C:\Program Files\Git\bin\git.exe config core.sparsecheckout # timeout=10 > C:\Program Files\Git\bin\git.exe checkout -f 787879583e36f6e3afc5b60d35d15409d2c6a11a # timeout=10 ERROR: Timeout after 10 minutes  
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Have you checked which credentials are used and what are the ones you were expecting to use?

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Have you checked which credentials are used and what are the ones you were expecting to use?
            Hide
            willwagner Will Wagner added a comment -

            I believe I am using the correct stored credentials, they work fine if I downgrade the module to 0.3.5.

            They are https username and password that work everywhere else. Do they need to be something special?

            Show
            willwagner Will Wagner added a comment - I believe I am using the correct stored credentials, they work fine if I downgrade the module to 0.3.5. They are https username and password that work everywhere else. Do they need to be something special?
            Hide
            willwagner Will Wagner added a comment -

            So looking at this some more, I believe it is the LFS Pull step that is using the wrong credentials. For some reason it previously used to say 'Enabling Git LFS pull' before it started this and did the config core.sparsecheckout.

            Show
            willwagner Will Wagner added a comment - So looking at this some more, I believe it is the LFS Pull step that is using the wrong credentials. For some reason it previously used to say 'Enabling Git LFS pull' before it started this and did the config core.sparsecheckout.
            Hide
            nickb_carallon Nick Bulleid added a comment -

            I've uploaded a fix for this issue to GerritHub.

            https://review.gerrithub.io/c/jenkinsci/gerrit-code-review-plugin/+/477949

            Show
            nickb_carallon Nick Bulleid added a comment - I've uploaded a fix for this issue to GerritHub. https://review.gerrithub.io/c/jenkinsci/gerrit-code-review-plugin/+/477949
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Thanks Nick Bulleid looking at your fix right now.

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Thanks Nick Bulleid looking at your fix right now.
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Unfortunately, your fix is breaking existing functionality.

            Let me reproduce the problem and see if there is another way to achieve the same result.

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Unfortunately, your fix is breaking existing functionality. Let me reproduce the problem and see if there is another way to achieve the same result.
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            I just tried and it works for me. When the build is triggered on a remote host, I see:

            using GIT_ASKPASS to set credentials jenkins-at-gerritci

            In your case I see from your logs:

            using GIT_ASKPASS to set credentials

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - I just tried and it works for me. When the build is triggered on a remote host, I see: using GIT_ASKPASS to set credentials jenkins-at-gerritci In your case I see from your logs: using GIT_ASKPASS to set credentials
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Did you set the credentials id on the multi-branch pipeline job?

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Did you set the credentials id on the multi-branch pipeline job?
            willwagner Will Wagner made changes -
            Field Original Value New Value
            Attachment credentials.png [ 49910 ]
            Hide
            willwagner Will Wagner added a comment -

            We have set username/password credentials on the pipeline job, see uploaded image.

            Out of interest what functionality is broken by our change?

            Show
            willwagner Will Wagner added a comment - We have set username/password credentials on the pipeline job, see uploaded image. Out of interest what functionality is broken by our change?
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Can you try to use the Gerrit authenticated URL?

            https://gerrit.mycompany.com/a/myrepo

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Can you try to use the Gerrit authenticated URL? https://gerrit.mycompany.com/a/myrepo
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Will Wagner the SCM extensions are completely removed in your proposed change. Without those extensions, you won't be able to successfully complete a branch scanning of a medium-sized Gerrit repository.

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Will Wagner the SCM extensions are completely removed in your proposed change. Without those extensions, you won't be able to successfully complete a branch scanning of a medium-sized Gerrit repository.
            lucamilanesio Luca Domenico Milanesio made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            lucamilanesio Luca Domenico Milanesio made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As 0.4.1
            Hide
            lucamilanesio Luca Domenico Milanesio added a comment -

            Thanks again Will Wagner for reporting and fixing the problem.

            Show
            lucamilanesio Luca Domenico Milanesio added a comment - Thanks again Will Wagner for reporting and fixing the problem.
            lucamilanesio Luca Domenico Milanesio made changes -
            Link This issue is blocked by JENKINS-60362 [ JENKINS-60362 ]

              People

              • Assignee:
                lucamilanesio Luca Domenico Milanesio
                Reporter:
                willwagner Will Wagner
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: