When we added Heap Stats collection (https://issues.jenkins-ci.org/browse/JENKINS-22791) in v2.42 it appears that we inadvertently caused a major performance-stability regression if the histogram is collected regularly.
How? Well, this gathers a live heap histogram. This appears to triggers a Full GC. This is visible in GC logs because they show the following cause:
> [Full GC (Heap Inspection Initiated GC).
Now, because this is a FullGC and not a concurrent or young-gen GC, and we're generally using G1 GC, the slow Serial garbage collector is used for FullGC. This is a NON-concurrent GC mode, meaning the application is fully paused until it completes, and it is SINGLE-threaded, meaning rather than 1 GB/s per CPU of GC throughput, we get <1 GB/s total. It also cleans and compacts the entire heap rather than just part of it as with other modes.
So, with 15 GB of used heap that means a pause of up ~15s. This matches behavior observed in the wild.
I am rating this as critical because on larger-scale production masters a hang that long can cause job failures, visible UI hangs, HTTP request timeouts, and other issues – it should result in Surable Task failures for Pipelines, for example.
Proposed solution: only gather the live heap histogram when a user is explicitly requesting a support bundle (disable it by default).
|Field||Original Value||New Value|
|Status||Open [ 1 ]||In Progress [ 3 ]|
|Status||In Progress [ 3 ]||In Review [ 10005 ]|
|Remote Link||This issue links to "PR-134 (Web Link)" [ 20224 ]|
|Remote Link||This issue links to "PR-135 (Web Link)" [ 20225 ]|
|Remote Link||This issue links to "Page (Jenkins Wiki)" [ 20235 ]|
|Status||In Review [ 10005 ]||Resolved [ 5 ]|
|Resolution||Fixed [ 1 ]|
|Status||Resolved [ 5 ]||Closed [ 6 ]|