Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Component/s: core
-
Labels:None
-
Environment:Apache Software Foundation, Hudson 1.376, Ubuntu server, Sun 1.6 JDK, 4Gb RAM
Description
In the ASF Hudson installation, rebuilding the dependency graph is quite slow (minutes). As rebuilding seems to happen twice on saving project configuration (AbstractProject.doConfigSubmit()), working with the project configuration is very slow and will frequently time out.
We have a large number of jobs (>300) and from a quick look at the code, it seems like building the dependency graph is done even if there are no changes to the dependencies. Also, building the graph seems to be done by iterating over all jobs which will in turn iterator over all other jobs. This doesn't seem to scale to our size 
Here's a typical thread dump from saving a project configuration:
"Handling POST /hudson/job/ftpserver-trunk-jdk1.6-osx/configSubmit : http-8090-27" daemon prio=10 tid=0x000000004340a000 nid=0x4576 runnable [0x00007f2fa8e48000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.intern(Native Method)
at hudson.maven.ModuleDependency.<init>(ModuleDependency.java:48)
at hudson.maven.ModuleDependency.<init>(ModuleDependency.java:54)
at hudson.maven.MavenModule.asDependency(MavenModule.java:296)
at hudson.maven.MavenModule.buildDependencyGraph(MavenModule.java:385)
at hudson.maven.MavenModuleSet.buildDependencyGraph(MavenModuleSet.java:487)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:100)
at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3346)
at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:588)
at sun.reflect.GeneratedMethodAccessor1225.invoke(Unknown Source)
...
Issue Links
- is blocking
-
JENKINS-9301
Improve dependency graph calculation (for maven jobs)
-
- Resolved
-
Activity
Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/d3978478486e4030a058b05ff86a01f132c720a3
Log:
record changes for JENKINS-7535
Code changed in jenkins
User: Christoph Kutzinski
Path:
maven-plugin/src/main/java/hudson/maven/MavenModule.java
http://jenkins-ci.org/commit/jenkins/cccb301fbe28ff8b33b044f81f1d02fcd7ea02ee
Log:
JENKINS-7535 don't remember per-moduleset modules to prevent OOMEs
Code changed in jenkins
User: Olivier Lamy
Path:
src/main/java/hudson/maven/MavenModule.java
src/main/java/hudson/maven/ModuleDependency.java
http://jenkins-ci.org/commit/maven-plugin/004eadacbe5bb56011e3134e7b904bb4f08c9f3f
Log:
JENKINS-7535 Rebuilding dependency graph slow on large installations
Submitted by evernat.
Originally-Committed-As: 7d46b22599c6001eaeff7f47715fa80491b3fdd5
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/MavenModule.java
src/main/java/hudson/maven/MavenModuleSet.java
http://jenkins-ci.org/commit/maven-plugin/70da285e456a34febb9705199a4aa6287506b38e
Log:
[FIXED JENKINS-7535] building up the dependency graph for maven jobs
takes too long
Originally-Committed-As: 6a5330fd936fb949da74d738b82dcf5333282b61
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/MavenModule.java
http://jenkins-ci.org/commit/maven-plugin/28e769c1549bea397ce17081e50d1477489735df
Log:
JENKINS-7535 don't remember per-moduleset modules to prevent OOMEs
Originally-Committed-As: cccb301fbe28ff8b33b044f81f1d02fcd7ea02ee
It would be lovely if the dependency computation were to use a more clever algorithm so that it is very fast, but there are some quick wins available:
a) don't do it if nothing changed
b) don't do it if another thread is already doing a global recomputation. At the least, just have a single thread that does this dependency computation and have save mark the graph is needing an update. That way nobody is delayed and the worst that happens is one thread does the updates back-to-back forever.
c) only change as much as is needed by the recently changed dependencies. This is an extension of (a) to make the entire computation incremental. Basically, you can keep an update time on each node and on the graph in general. In doing the dependency computation, you can skip part of the computation as soon as you note that the node under consideration hasn't changed more recently than the last graph update.