Perforce server has recently been upgraded to latest Solaris version, at Perforce's recommendation.

Now we get occasional lock-ups, traced to Hudson, when a 'changes' command takes some amount of time and several client -i commands get queued behind it from Hudson. These then never complete, causing the Perforce server to maintain file locks, blocking all subsequent commands.

Nothing in the Hudson log file until after the p4d is stopped, then the following errors (from Problem: onwards)

On the Perforce side, monitor output, just before the server locks up, indicates three client -i commands in flight. I noticed the 'changes' command complete just prior to these client -i entries.

Every 5.0s: p4 -u gray monitor show -a Wed Jan 5 11:37:35 2011
13359 R verify 00:01:46 client -i
13360 R gray 00:01:29 IDLE none
13361 R verify 00:01:42 IDLE none
13362 R verify 00:00:24 IDLE none
13363 R verify 00:01:46 client -i
13365 R verify 00:01:34 IDLE none
13366 R verify 00:01:29 client -i
13457 R gray 00:00:19 monitor show -a

Hudson.log info:

INFO: RMDMA-block #721 main build action completed: SUCCESS
Jan 5, 2011 11:35:48 AM hudson.model.Run run
INFO: NWL-FPGA #24 main build action completed: SUCCESS
Problem: Partner exited unexpectedly.
For Command: /proj/merlot/bin/p4 -s client -i
With Data:
===================
Client: verify.merlot-top-ovm.hudson-ci
Owner: verify
Description: Created by verify.
Root: /proj/merlot/work/verify/hudson-ci-data/jobs/merlot-top-OVM/workspace
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
SubmitOptions: submitunchanged
LineEnd:
View:
//proj/merlot/MAIN/... //verify.merlot-top-ovm.hudson-ci/merlot/...
//collateral/scripts/MAIN/... //verify.merlot-top-ovm.hudson-ci/scripts/...

===================

com.tek42.perforce.PerforceException: Partner exited unexpectedly.
For Command: /proj/merlot/bin/p4 -s client -i
With Data:
===================
Client: verify.merlot-top-ovm.hudson-ci
Owner: verify
Description: Created by verify.
Root: /proj/merlot/work/verify/hudson-ci-data/jobs/merlot-top-OVM/workspace
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
SubmitOptions: submitunchanged
LineEnd:
View:
//proj/merlot/MAIN/... //verify.merlot-top-ovm.hudson-ci/merlot/...
//collateral/scripts/MAIN/... //verify.merlot-top-ovm.hudson-ci/scripts/...

===================

at com.tek42.perforce.parse.AbstractPerforceTemplate.saveToPerforce(AbstractPerforceTemplate.java:248)
at com.tek42.perforce.parse.Workspaces.saveWorkspace(Workspaces.java:68)
at hudson.plugins.perforce.PerforceSCM.saveWorkspaceIfDirty(PerforceSCM.java:1163)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:791)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Jan 5, 2011 11:42:22 AM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling
java.lang.NumberFormatException: For input string: "exited"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:447)
at java.lang.Integer.<init>(Integer.java:620)
at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:144)
at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:897)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:793)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Jan 5, 2011 11:42:22 AM hudson.triggers.SCMTrigger$Runner run
INFO: SCM changes detected in OSD-DMA-block. Triggering #135
Jan 5, 2011 11:42:22 AM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling
java.lang.NumberFormatException: For input string: "exited"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:447)
at java.lang.Integer.<init>(Integer.java:620)
at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:144)
at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:897)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:793)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Problem: Partner exited unexpectedly.
For Command: /proj/merlot/bin/p4 -s client -i
With Data:
===================
Client: verify.monitor.hudson-ci
Owner: verify
Description: Created by verify.
Root: /proj/merlot/work/verify/hudson-ci-data/jobs/monitor-block/workspace
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
SubmitOptions: submitunchanged
LineEnd:
View:
//ip/monitor/MAIN/... //verify.monitor.hudson-ci/monitor/...

===================

com.tek42.perforce.PerforceException: Partner exited unexpectedly.
For Command: /proj/merlot/bin/p4 -s client -i
With Data:
===================
Client: verify.monitor.hudson-ci
Owner: verify
Description: Created by verify.
Root: /proj/merlot/work/verify/hudson-ci-data/jobs/monitor-block/workspace
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
SubmitOptions: submitunchanged
LineEnd:
View:
//ip/monitor/MAIN/... //verify.monitor.hudson-ci/monitor/...

===================

at com.tek42.perforce.parse.AbstractPerforceTemplate.saveToPerforce(AbstractPerforceTemplate.java:248)
at com.tek42.perforce.parse.Workspaces.saveWorkspace(Workspaces.java:68)
at hudson.plugins.perforce.PerforceSCM.saveWorkspaceIfDirty(PerforceSCM.java:1163)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:791)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Jan 5, 2011 11:42:23 AM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling
java.lang.NumberFormatException: For input string: "exited"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:447)
at java.lang.Integer.<init>(Integer.java:620)
at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:144)
at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:897)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:793)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Problem: Connect to server failed; check $P4PORT
1473738 [SCM polling for hudson.model.FreeStyleProject@3a6d8d73[RXD-DMA-block]] WARN perforce - Perforce process terminated suddenly
com.tek42.perforce.PerforceException: Connect to server failed; check $P4PORT
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:339)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1005)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:787)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
1473739 [SCM polling for hudson.model.FreeStyleProject@6411c21b[RMDMA-block]] WARN perforce - java.io.IOException: Write end dead
at java.io.PipedInputStream.read(PipedInputStream.java:294)
at java.io.PipedInputStream.read(PipedInputStream.java:361)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:297)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1005)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:787)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

1473739 [SCM polling for hudson.model.FreeStyleProject@3a6d8d73[RXD-DMA-block]] WARN perforce - java.io.IOException: Write end dead
at java.io.PipedInputStream.read(PipedInputStream.java:294)
at java.io.PipedInputStream.read(PipedInputStream.java:361)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:297)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1005)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:787)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1236)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)