It seems that whenever we perform a nightly automated upgrade of the jenkins package (using "cron-apt"), the debian package actually fails to stop the previous Jenkins instance and starts a new one.

We therefore end up with two java processes running at the same time, example:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4970 jenkins 20 0 1492m 990m 5032 S 0,5 32,5 4823:00 java
22025 jenkins 20 0 1459m 152m 2200 S 0,0 5,0 8345:05 java

Notice how process #4970 is slowly eating all available memory... This process was started during the upgrade, which last took place on march 11th:

root@jenkins0:/home/anarcat# ps ax -O start | grep java
4969 Mar 11 S ? 00:00:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid – /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1
4970 Mar 11 S ? 3-08:24:41 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1
9674 09:35:37 S pts/1 00:00:00 grep java
22024 Mar 04 S ? 00:00:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid – /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1
22025 Mar 04 S ? 5-19:05:05 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1

Also notice how that new process is not having control over the socket:

root@jenkins0:/home/anarcat# netstat -plunt | grep java
tcp6 0 0 :::34755 :::* LISTEN 4970/java
tcp6 0 0 :::8080 :::* LISTEN 4970/java
udp6 0 0 :::33848 :::* 4970/java
udp6 0 0 :::5353 :::* 4970/java

So the upgrade "succeeded", but takes up all memory, and leaves the old java process lying around.

We first noticed this problem on January 27th, but it may have started before that.

Suggestions?

The workaround is to stop jenkins, and killall -9 java, then restart jenkins.