Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11256

Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave

      Description

      linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave
      build fails with

      — maven-install-plugin:2.3.1:install (default-install) @ xxxxxxxx —
      Waiting for Jenkins to finish collecting data
      mavenExecutionResult exceptions not empty
      message : Internal error: java.util.ConcurrentModificationException
      cause : null
      Stack trace :
      org.apache.maven.InternalErrorException: Internal error: java.util.ConcurrentModificationException
      at org.apache.maven.lifecycle.internal.BuilderCommon.handleBuildError(BuilderCommon.java:128)[INFO] Installing xxxxxxxx.jar to xxxxxxxx

      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:95)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:164)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
      at java.util.concurrent.FutureTask.run(FutureTask.java:123)[INFO] Compiling 1 source file to xxxxxxxx

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.util.ConcurrentModificationException
      at java.util.ArrayList.writeObject(ArrayList.java:572)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.ArrayList.writeObject(ArrayList.java:569)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.HashMap.writeObject(HashMap.java:1039)
      at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      at hudson.remoting.Channel.callAsync(Channel.java:708)
      at hudson.maven.Maven3Builder$FilterImpl.executeAsync(Maven3Builder.java:229)
      at hudson.maven.reporters.MavenFingerprinter.postBuild(MavenFingerprinter.java:96)
      at hudson.maven.Maven3Builder$MavenExecutionListener.recordProjectSucceeded(Maven3Builder.java:431)
      at hudson.maven.Maven3Builder$MavenExecutionListener.projectSucceeded(Maven3Builder.java:405)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:74)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:91)
      ... 7 more
      [INFO]
      [INFO] — maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ xxxxxxxx —
      channel stopped
      Archiviere Artefakte
      Finished: FAILURE

        Activity

        Hide
        kutzi kutzi added a comment -

        Does it happen frequently or only once?

        Show
        kutzi kutzi added a comment - Does it happen frequently or only once?
        Hide
        kisa kisa added a comment -

        Every 10-15 builds ... bad enough

        Show
        kisa kisa added a comment - Every 10-15 builds ... bad enough
        Hide
        paux Stephan Pauxberger added a comment -

        All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places.

        Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation.

        I will create a pull request for this.

        Show
        paux Stephan Pauxberger added a comment - All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places. Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation. I will create a pull request for this.
        Hide
        paux Stephan Pauxberger added a comment - - edited

        Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e, however it landed in the open pull request concerning Parent POM fingerprinting.

        Kutzi, could you look at both of them together or should I split it out again?

        Show
        paux Stephan Pauxberger added a comment - - edited Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e , however it landed in the open pull request concerning Parent POM fingerprinting. Kutzi, could you look at both of them together or should I split it out again?
        Hide
        kutzi kutzi added a comment - - edited

        Thanks for the pull request.
        Yes, please create a separate pull request for each issue.

        Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem?

        I think I'd rather put a synchronized block inside the BuildCallable in postBuild()

        Show
        kutzi kutzi added a comment - - edited Thanks for the pull request. Yes, please create a separate pull request for each issue. Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem? I think I'd rather put a synchronized block inside the BuildCallable in postBuild()
        Hide
        paux Stephan Pauxberger added a comment - - edited

        Old problem with race conditions - if you need them, they don't show up.

        I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module.

        As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something).

        Which leads back to the question: where does ist happen.

        The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace)

        However, I did not found anyone modifying one of the valuelists.

        Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises.

        I will further look into this.

        UPDATE:

        Its an ArrayList in an ArrayList in the reporters map could be some plugin.

        Show
        paux Stephan Pauxberger added a comment - - edited Old problem with race conditions - if you need them, they don't show up. I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module. As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something). Which leads back to the question: where does ist happen. The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace) However, I did not found anyone modifying one of the valuelists. Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises. I will further look into this. UPDATE: Its an ArrayList in an ArrayList in the reporters map could be some plugin.
        Hide
        kutzi kutzi added a comment - - edited

        According to Olivier this might be a bug in Maven 3.0.3:
        http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html

        Show
        kutzi kutzi added a comment - - edited According to Olivier this might be a bug in Maven 3.0.3: http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html
        Hide
        paux Stephan Pauxberger added a comment -

        Short update:

        I could not reproduce the problem using a test installation, I will try another test using the exact plugin configuration of the actual system.

        Show
        paux Stephan Pauxberger added a comment - Short update: I could not reproduce the problem using a test installation, I will try another test using the exact plugin configuration of the actual system.
        Hide
        paux Stephan Pauxberger added a comment -

        Found the culprit (no plugin):

        hudson.maven.reporters.SurefireArchiver.result is a TestResult object containing an ArrayList (suites). This list is updated whenever a module containing tests completes. Since The Async invoke serializes the Maven3Builder and therefore all its reporters (along with the SurefireArchivers). If the fingerprinter runs during the update of the test result, this error occurs.

        Solution: Make the field hudson.tasks.junit.TestResult.suites a synchronized list (Collections.synchronizedList)

        Show
        paux Stephan Pauxberger added a comment - Found the culprit (no plugin): hudson.maven.reporters.SurefireArchiver.result is a TestResult object containing an ArrayList (suites). This list is updated whenever a module containing tests completes. Since The Async invoke serializes the Maven3Builder and therefore all its reporters (along with the SurefireArchivers). If the fingerprinter runs during the update of the test result, this error occurs. Solution: Make the field hudson.tasks.junit.TestResult.suites a synchronized list (Collections.synchronizedList)
        Hide
        paux Stephan Pauxberger added a comment - - edited

        @kutzi: My development environment seems broken. Could you do that change?

        Thx

        Show
        paux Stephan Pauxberger added a comment - - edited @kutzi: My development environment seems broken. Could you do that change? Thx
        Hide
        kutzi kutzi added a comment - - edited

        Ah, forget what I previously wrote here, I missed an important part of your comments.

        Did you manage to reproduce it reliably? If so, do you have a test case?

        Show
        kutzi kutzi added a comment - - edited Ah, forget what I previously wrote here, I missed an important part of your comments. Did you manage to reproduce it reliably? If so, do you have a test case?
        Hide
        kutzi kutzi added a comment -

        While it might make sense to make TestResult thread-safe, I'd also ask if it's actually needed that SurefireArchiver#result is serialized.
        Maybe that could be changed, too?

        Show
        kutzi kutzi added a comment - While it might make sense to make TestResult thread-safe, I'd also ask if it's actually needed that SurefireArchiver#result is serialized. Maybe that could be changed, too?
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Christoph Kutzinski
        Path:
        maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
        http://jenkins-ci.org/commit/jenkins/1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
        Log:
        [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter

        Compare: https://github.com/jenkinsci/jenkins/compare/796918d...1476b2e

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4 Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Compare: https://github.com/jenkinsci/jenkins/compare/796918d...1476b2e
        Hide
        kutzi kutzi added a comment -

        I've removed the serialization of 'result' and local tests indicate that this fixes the issue.
        As a positive side effect, Jenkins has to serialize less stuff on each remote call, which should reduce the Maven job overhead a bit

        Show
        kutzi kutzi added a comment - I've removed the serialization of 'result' and local tests indicate that this fixes the issue. As a positive side effect, Jenkins has to serialize less stuff on each remote call, which should reduce the Maven job overhead a bit
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Christoph Kutzinski
        Path:
        changelog.html
        http://jenkins-ci.org/commit/jenkins/b835fb15497cf31a5069804a9920a747053be3c4
        Log:
        Changelog entry for JENKINS-11256

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html http://jenkins-ci.org/commit/jenkins/b835fb15497cf31a5069804a9920a747053be3c4 Log: Changelog entry for JENKINS-11256
        Hide
        dogfood dogfood added a comment -

        Integrated in jenkins_main_trunk #1303
        [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
        Changelog entry for JENKINS-11256

        Christoph Kutzinski : 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
        Files :

        • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java

        Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4
        Files :

        • changelog.html
        Show
        dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1303 [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Changelog entry for JENKINS-11256 Christoph Kutzinski : 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4 Files : maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4 Files : changelog.html
        Hide
        swolk swolk added a comment -

        When will this make it into an LTS build? I really need this fix. Thanks

        Show
        swolk swolk added a comment - When will this make it into an LTS build? I really need this fix. Thanks
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Christoph Kutzinski
        Path:
        maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
        http://jenkins-ci.org/commit/jenkins/46629db8d493f67c5b8ce1c14465b3c6a117e847
        Log:
        [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter(cherry picked from commit 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4)

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/46629db8d493f67c5b8ce1c14465b3c6a117e847 Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter(cherry picked from commit 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4)
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Christoph Kutzinski
        Path:
        changelog.html
        http://jenkins-ci.org/commit/jenkins/b39a310c5165234dab6d7acd3ed82bd399d255eb
        Log:
        Changelog entry for JENKINS-11256(cherry picked from commit b835fb15497cf31a5069804a9920a747053be3c4)

        Conflicts:

        changelog.html

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html http://jenkins-ci.org/commit/jenkins/b39a310c5165234dab6d7acd3ed82bd399d255eb Log: Changelog entry for JENKINS-11256 (cherry picked from commit b835fb15497cf31a5069804a9920a747053be3c4) Conflicts: changelog.html
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Christoph Kutzinski
        Path:
        src/main/java/hudson/maven/reporters/SurefireArchiver.java
        http://jenkins-ci.org/commit/maven-plugin/00f1af0b4e71f8c691e36a1bb741eeff7bd512ee
        Log:
        [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
        Originally-Committed-As: 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/maven-plugin/00f1af0b4e71f8c691e36a1bb741eeff7bd512ee Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Originally-Committed-As: 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4

          People

          • Assignee:
            kutzi kutzi
            Reporter:
            kisa kisa
          • Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: