Issue

Recently got a warning that Health Advisor cannot upload a support bundle. Having a look at the log, the bundle generation did not seem to fail but the advisor cannot find the bundle generated:

2020-11-12 08:02:55.276+0000 [id=478625]    SEVERE  c.c.j.p.a.client.AdvisorClient#doUploadFile: Exception trying to upload support bundle. Message: [/var/jenkins_home/support/cloudbees-support_cloudbees-ci-managed-master-jenkins.example.com_mm-1_2020-11-11_10.55.12.zip (No such file or directory)], File: [/var/jenkins_home/support/cloudbees-support_cloudbees-ci-managed-master-jenkins.example.com_mm-1_2020-11-11_10.55.12.zip], Metadata: [isDirectory: [false], isHidden: [false], isReadable: [false], isRegularFile: [false], isSymbolicLink: [false], isWritable: [false]]

We noticed that the generation was hanging for a long time. At the time, there was a background generation as well. When we fixed the hung process, they both completed at around the same time so it looks like those events are related. We suspect that the Support Core bundle cleanup process removed the advisor bundle before it started to be uploaded.

Hypothesis

Jenkins Health Advisor generates bundle directly under $JENKINS_HOME/support/. Bundles under that directory however are candidate for deletion based on exponential decay every time a background generated bundle is completed.

With the wrong timing, a bundle generated by advisor may be deleted by support core which may cause upload failure.

Note: Support Core does not log anything about which bundles are being removed. Which could be improved as well.