Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10148

adb daemon killed after job complete breaks other running emulators

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      SDK Tools r11, Fedora Core 14 i686 (slave), number of executors per slave 4. Jenkins 1.414, Android Emulator 1.15
    • Similar Issues:

      Description

      When running multiple emulator instances on a single node each within their own Xvnc session, at the end of the job the plugin seems to end up killing the adb daemon. The other emulators lose connection to the daemon, and even though a background task may start adb again, emulators may not successfully reconnect to adb. This is particularly problematic when running large multi-configuration android-emulator jobs on a single node with a large degree of parallelism. 2 concurrent executors seems to occasionally cause problems but with 4 the results are not even worthwhile to review.

      Reducing the number of executors on the node to 1 (or otherwise weighting the job to consume an entire system) does work-around the problem.

        Attachments

          Activity

          Hide
          orrc Christopher Orr added a comment -

          The plugin never tries to kill the adb daemon.
          I also run jobs with many executors on Linux (SDK Tools r11, multiple Ubuntu amd64 slaves, 4 executors per slave) without having this problem.

          However, I have seen this issue in the past where adb crashes at the end of a job. Since JENKINS-7354, I haven't had any further reports of it though, and adb does seem to be a bit more well behaved these days.

          But as I still don't have any way of reproducing this, and don't have any indication of why adb fails (or any sort of gdb stacktrace from when it does), there's not really much I can do about this...

          For reference, does this happen for every job, or only sporadically? Has it always happened, or only since upgrading the Android SDK, for example? Is there anything particular given in the build log output once it's finished and crashes adb?

          Show
          orrc Christopher Orr added a comment - The plugin never tries to kill the adb daemon. I also run jobs with many executors on Linux (SDK Tools r11, multiple Ubuntu amd64 slaves, 4 executors per slave) without having this problem. However, I have seen this issue in the past where adb crashes at the end of a job. Since JENKINS-7354 , I haven't had any further reports of it though, and adb does seem to be a bit more well behaved these days. But as I still don't have any way of reproducing this, and don't have any indication of why adb fails (or any sort of gdb stacktrace from when it does), there's not really much I can do about this... For reference, does this happen for every job, or only sporadically? Has it always happened, or only since upgrading the Android SDK, for example? Is there anything particular given in the build log output once it's finished and crashes adb?
          Hide
          hokatichen Mike Elkin added a comment -

          I am digging into this quite a bit more as a result of your comment. Right now it seems like it is sporadically, these are 32bit FC14 systems with Oracle JDK 1.6u24 + ant 1.8.2 + SDKr11 running multi-configuration on a variety of device configurations. I was able to instigate a crash but now after running some more tests it is not crashing. I will continue to investigate and hopefully come up with some useful logs for review.

          Show
          hokatichen Mike Elkin added a comment - I am digging into this quite a bit more as a result of your comment. Right now it seems like it is sporadically, these are 32bit FC14 systems with Oracle JDK 1.6u24 + ant 1.8.2 + SDKr11 running multi-configuration on a variety of device configurations. I was able to instigate a crash but now after running some more tests it is not crashing. I will continue to investigate and hopefully come up with some useful logs for review.
          Hide
          hokatichen Mike Elkin added a comment -

          Looking at this more you are a bit right if ADB has already been started. However when ADB has to be started up the stability seems to go way down. To replicate you'd need to kill the adb daemon, setup a many-executor node, and kick off a multi-config emulator job that will consume all of the executors. I've noticed that a lot of the jobs seem to hang and in some cases the adb server will die off.

          Show
          hokatichen Mike Elkin added a comment - Looking at this more you are a bit right if ADB has already been started. However when ADB has to be started up the stability seems to go way down. To replicate you'd need to kill the adb daemon, setup a many-executor node, and kick off a multi-config emulator job that will consume all of the executors. I've noticed that a lot of the jobs seem to hang and in some cases the adb server will die off.
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          I've submitted a proposed fix for this here:
          https://github.com/jorgenpt/android-emulator-plugin/commit/0702320d18b152d360b80d77f189342d2c8296cf

          It's a part of this pull request:
          https://github.com/jenkinsci/android-emulator-plugin/pull/8

          The change makes it run a private adb for each emulator, which should fix all problems like these.

          Show
          jorgenpt Jørgen Tjernø added a comment - I've submitted a proposed fix for this here: https://github.com/jorgenpt/android-emulator-plugin/commit/0702320d18b152d360b80d77f189342d2c8296cf It's a part of this pull request: https://github.com/jenkinsci/android-emulator-plugin/pull/8 The change makes it run a private adb for each emulator, which should fix all problems like these.
          Hide
          orrc Christopher Orr added a comment -

          Version 2.0 has now been released, which includes Jørgen's feature.

          One instance of ADB crashing no longer affects other builds running in parallel.

          Show
          orrc Christopher Orr added a comment - Version 2.0 has now been released, which includes Jørgen's feature. One instance of ADB crashing no longer affects other builds running in parallel.

            People

            • Assignee:
              orrc Christopher Orr
              Reporter:
              hokatichen Mike Elkin
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: