-
Bug
-
Resolution: Not A Defect
-
Major
-
None
-
windows 10, jenkins 2.46.1, git plugin 3.2.0
I intermittently see git fetch hang on my windows boxes, which timeout causing my automatons to fail. Annoyingly though there is no obvious reason as to why. Also in any given automation i run about 8 of these windows jobs in parallel and generally all will work except one random one, so i know its nothing i'm doing wrong, otherwise why would others fail? Also it isn't specific to a windows machine. Also when it does timeout, the git pluging spits out the stdout and based on what i can see the bulk of the git fetch is done, the only bit thats missing is the output after "Resolving deltas:...".
Based on the symptoms, imo, it seems this hang could be due to the plugin trying to read output from the git process, which is maybe preventing git process from writing to the internal buffer? Pure guessing, but i've seen this symptom before in my own scripting.
Anyway this is the output i see when the hang happens;
[Pipeline] checkout
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url https://yacoub.autodesk.com/modeling-components/asm.git # timeout=10
*Fetching upstream changes from https://yacoub.autodesk.com/modeling-components/asm.git*
> git.exe --version # timeout=10
using GIT_ASKPASS to set credentials git credentials
> git.exe fetch --tags --progress https://yacoub.autodesk.com/modeling-components/asm.git +refs/heads/:refs/remotes/origin/* --depth=1 # timeout=120*
ERROR: Timeout after 120 minutes
ERROR: Error fetching remote repo 'origin'
*hudson.plugins.git.GitException: Failed to fetch from https://yacoub.autodesk.com/modeling-components/asm.git*
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:806)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --progress https://yacoub.autodesk.com/modeling-components/asm.git +refs/heads/:refs/remotes/origin/* --depth=1" returned status code -1:*
stdout:
stderr: remote: Counting objects: 1
remote: Counting objects: 144, done.
remote: Compressing objects: 0% (1/131)
remote: Compressing objects: 1% (2/131)
remote: Compressing objects: 2% (3/131)
remote: Compressing objects: 3% (4/131)
remote: Compressing objects: 4% (6/131)
remote: Compressing objects: 5% (7/131)
remote: Compressing objects: 6% (8/131)
remote: Compressing objects: 7% (10/131)
remote: Compressing objects: 8% (11/131)
remote: Compressing objects: 9% (12/131)
remote: Compressing objects: 10% (14/131)
remote: Compressing objects: 11% (15/131)
remote: Compressing objects: 12% (16/131)
remote: Compressing objects: 13% (18/131)
remote: Compressing objects: 14% (19/131)
remote: Compressing objects: 15% (20/131)
remote: Compressing objects: 16% (21/131)
remote: Compressing objects: 17% (23/131)
remote: Compressing objects: 18% (24/131)
remote: Compressing objects: 19% (25/131)
remote: Compressing objects: 20% (27/131)
remote: Compressing objects: 21% (28/131)
remote: Compressing objects: 22% (29/131)
remote: Compressing objects: 23% (31/131)
remote: Compressing objects: 24% (32/131)
remote: Compressing objects: 25% (33/131)
remote: Compressing objects: 26% (35/131)
remote: Compressing objects: 27% (36/131)
remote: Compressing objects: 28% (37/131)
remote: Compressing objects: 29% (38/131)
remote: Compressing objects: 30% (40/131)
remote: Compressing objects: 31% (41/131)
remote: Compressing objects: 32% (42/131)
remote: Compressing objects: 33% (44/131)
remote: Compressing objects: 34% (45/131)
remote: Compressing objects: 35% (46/131)
remote: Compressing objects: 36% (48/131)
remote: Compressing objects: 37% (49/131)
remote: Compressing objects: 38% (50/131)
remote: Compressing objects: 39% (52/131)
remote: Compressing objects: 40% (53/131)
remote: Compressing objects: 41% (54/131)
remote: Compressing objects: 42% (56/131)
remote: Compressing objects: 43% (57/131)
remote: Compressing objects: 44% (58/131)
remote: Compressing objects: 45% (59/131)
remote: Compressing objects: 46% (61/131)
remote: Compressing objects: 47% (62/131)
remote: Compressing objects: 48% (63/131)
remote: Compressing objects: 49% (65/131)
remote: Compressing objects: 50% (66/131)
remote: Compressing objects: 51% (67/131)
remote: Compressing objects: 52% (69/131)
remote: Compressing objects: 53% (70/131)
remote: Compressing objects: 54% (71/131)
remote: Compressing objects: 55% (73/131)
remote: Compressing objects: 56% (74/131)
remote: Compressing objects: 57% (75/131)
remote: Compressing objects: 58% (76/131)
remote: Compressing objects: 59% (78/131)
remote: Compressing objects: 60% (79/131)
remote: Compressing objects: 61% (80/131)
remote: Compressing objects: 62% (82/131)
remote: Compressing objects: 63% (83/131)
remote: Compressing objects: 64% (84/131)
remote: Compressing objects: 65% (86/131)
remote: Compressing objects: 66% (87/131)
remote: Compressing objects: 67% (88/131)
remote: Compressing objects: 68% (90/131)
remote: Compressing objects: 69% (91/131)
remote: Compressing objects: 70% (92/131)
remote: Compressing objects: 71% (94/131)
remote: Compressing objects: 72% (95/131)
remote: Compressing objects: 73% (96/131)
remote: Compressing objects: 74% (97/131)
remote: Compressing objects: 75% (99/131)
remote: Compressing objects: 76% (100/131)
remote: Compressing objects: 77% (101/131)
remote: Compressing objects: 78% (103/131)
remote: Compressing objects: 79% (104/131)
remote: Compressing objects: 80% (105/131)
remote: Compressing objects: 81% (107/131)
remote: Compressing objects: 82% (108/131)
remote: Compressing objects: 83% (109/131)
remote: Compressing objects: 84% (111/131)
remote: Compressing objects: 85% (112/131)
remote: Compressing objects: 86% (113/131)
remote: Compressing objects: 87% (114/131)
remote: Compressing objects: 88% (116/131)
remote: Compressing objects: 89% (117/131)
remote: Compressing objects: 90% (118/131)
remote: Compressing objects: 91% (120/131)
remote: Compressing objects: 92% (121/131)
remote: Compressing objects: 93% (122/131)
remote: Compressing objects: 94% (124/131)
remote: Compressing objects: 95% (125/131)
remote: Compressing objects: 96% (126/131)
remote: Compressing objects: 97% (128/131)
remote: Compressing objects: 98% (129/131)
remote: Compressing objects: 99% (130/131)
remote: Compressing objects: 100% (131/131)
remote: Compressing objects: 100% (131/131), done.
Receiving objects: 0% (1/144)
Receiving objects: 1% (2/144)
Receiving objects: 2% (3/144)
Receiving objects: 3% (5/144)
Receiving objects: 4% (6/144)
Receiving objects: 5% (8/144)
Receiving objects: 6% (9/144)
Receiving objects: 7% (11/144)
Receiving objects: 8% (12/144)
Receiving objects: 9% (13/144)
Receiving objects: 10% (15/144)
Receiving objects: 11% (16/144)
Receiving objects: 12% (18/144)
Receiving objects: 13% (19/144)
Receiving objects: 14% (21/144)
Receiving objects: 15% (22/144)
Receiving objects: 16% (24/144)
Receiving objects: 17% (25/144)
Receiving objects: 18% (26/144)
Receiving objects: 19% (28/144)
Receiving objects: 20% (29/144)
Receiving objects: 21% (31/144)
Receiving objects: 22% (32/144)
Receiving objects: 23% (34/144)
Receiving objects: 24% (35/144)
Receiving objects: 25% (36/144)
Receiving objects: 26% (38/144)
Receiving objects: 27% (39/144)
Receiving objects: 28% (41/144)
Receiving objects: 29% (42/144)
Receiving objects: 30% (44/144)
Receiving objects: 31% (45/144)
Receiving objects: 32% (47/144)
Receiving objects: 33% (48/144)
Receiving objects: 34% (49/144)
Receiving objects: 35% (51/144)
Receiving objects: 36% (52/144)
Receiving objects: 37% (54/144)
Receiving objects: 38% (55/144)
Receiving objects: 39% (57/144)
Receiving objects: 40% (58/144)
Receiving objects: 41% (60/144)
Receiving objects: 42% (61/144)
Receiving objects: 43% (62/144)
Receiving objects: 44% (64/144)
Receiving objects: 45% (65/144)
remote: Total 144 (delta 13), reused 144 (delta 13), pack-reused 0
Receiving objects: 46% (67/144)
Receiving objects: 47% (68/144)
Receiving objects: 48% (70/144)
Receiving objects: 49% (71/144)
Receiving objects: 50% (72/144)
Receiving objects: 51% (74/144)
Receiving objects: 52% (75/144)
Receiving objects: 53% (77/144)
Receiving objects: 54% (78/144)
Receiving objects: 55% (80/144)
Receiving objects: 56% (81/144)
Receiving objects: 57% (83/144)
Receiving objects: 58% (84/144)
Receiving objects: 59% (85/144)
Receiving objects: 60% (87/144)
Receiving objects: 61% (88/144)
Receiving objects: 62% (90/144)
Receiving objects: 63% (91/144)
Receiving objects: 64% (93/144)
Receiving objects: 65% (94/144)
Receiving objects: 66% (96/144)
Receiving objects: 67% (97/144)
Receiving objects: 68% (98/144)
Receiving objects: 69% (100/144)
Receiving objects: 70% (101/144)
Receiving objects: 71% (103/144)
Receiving objects: 72% (104/144)
Receiving objects: 73% (106/144)
Receiving objects: 74% (107/144)
Receiving objects: 75% (108/144)
Receiving objects: 76% (110/144)
Receiving objects: 77% (111/144)
Receiving objects: 78% (113/144)
Receiving objects: 79% (114/144)
Receiving objects: 80% (116/144)
Receiving objects: 81% (117/144)
Receiving objects: 82% (119/144)
Receiving objects: 83% (120/144)
Receiving objects: 84% (121/144)
Receiving objects: 85% (123/144)
Receiving objects: 86% (124/144)
Receiving objects: 87% (126/144)
Receiving objects: 88% (127/144)
Receiving objects: 89% (129/144)
Receiving objects: 90% (130/144)
Receiving objects: 91% (132/144)
Receiving objects: 92% (133/144)
Receiving objects: 93% (134/144)
Receiving objects: 94% (136/144)
Receiving objects: 95% (137/144)
Receiving objects: 96% (139/144)
Receiving objects: 97% (140/144)
Receiving objects: 98% (142/144)
Receiving objects: 99% (143/144)
Receiving objects: 100% (144/144)
Receiving objects: 100% (144/144), 18.28 KiB | 0 bytes/s, done.
Resolving deltas: 0% (0/13)
Resolving deltas: 7% (1/13)
Resolving deltas: 15% (2/13)
Resolving deltas: 23% (3/13)
Resolving deltas: 30% (4/13)
Resolving deltas: 38% (5/13)
Resolving deltas: 46% (6/13)
Resolving deltas: 53% (7/13)
Resolving deltas: 69% (9/13)
Resolving deltas: 76% (10/13)
Resolving deltas: 84% (11/13)
Resolving deltas: 100% (13/13)
Resolving deltas: 100% (13/13), done.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1799)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1525)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:65)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:316)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
at ......remote call to JNLP4-connect connection from cam096.ads.autodesk.com/10.146.47.136:49739(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:830)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor1017.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy99.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:804)
... 13 more
I'm ready to try any experiments/suggestions to try to get to the bottom of this, it is crippling my automations. Please help.