We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '

'

6) lsof -p $processId | grep deleted

I can get below

java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)

java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)

java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)

java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)

BTW, for a finished job #5 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

 lsof -p 6480 | grep file-handle-leak

java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log