Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10421

Emulator occasionally hangs indefinitely on startup

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Ubuntu 10.04, linux 2.6.32, Jenkins 1.421.
      Android SDK r10
    • Similar Issues:

      Description

      When starting up the emulator, adb shell getprop dev.bootcomplete will sometimes hang indefinitely. This will cause the entire job to hang. Below is a log w/ timestamps where I have a 15 minute build timeout kick in.
      Since adb is prone to hanging, every adb command (or at least these first ones) should have a watchdog thread killing them if they take too long.
      I'm guessing an adb disconnect/connect combo after the hanging command might be needed too.

      08:17:35 Building on master
      08:17:35 [android] Using Android SDK: /usr/local/android
      08:17:35 $ /usr/local/android/tools/emulator -snapshot-list -no-window -avd hudson_en-US_160_WVGA_android-4
      08:17:35 [android] Starting Android emulator from snapshot
      08:17:35 $ /usr/local/android/tools/emulator -no-boot-anim -ports 47225,56719 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-4 -snapshot jenkins -no-snapshot-save -no-window
      08:17:35 emulator: warning: opening audio output failed
      08:17:35
      08:17:40 $ /usr/local/android/platform-tools/adb connect emu:47225,56719
      08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 logcat -v time
      08:17:40 [android] Waiting for emulator to finish booting...
      08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 shell getprop dev.bootcomplete
      08:32:35 Build timed out (after 15 minutes). Marking the build as aborted.
      08:32:35 [android] Interrupted while waiting for emulator to finish booting.
      08:32:35 [android] Timed-out after waiting 120 seconds for emulator
      08:32:35 [android] Stopping Android emulator
      08:32:35 [android] Archiving emulator log
      08:32:35 Notifying upstream projects of job completion
      08:32:35 Finished: NOT_BUILT

        Attachments

          Activity

          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Out of the last 18 builds, 4 of them have hung like this. That's more than 1 out of every 5 builds!

          Show
          jorgenpt Jørgen Tjernø added a comment - Out of the last 18 builds, 4 of them have hung like this. That's more than 1 out of every 5 builds!
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          I've applied this patch to our installation, and I'm going to see if it remedies the issue - or if it'll just cause later commands to fail.

          At least it'll time out the build after 120 seconds instead of the job-global timeout - but maybe there needs to be some other "cleanup" done when the adb shell command fails.

          Show
          jorgenpt Jørgen Tjernø added a comment - I've applied this patch to our installation, and I'm going to see if it remedies the issue - or if it'll just cause later commands to fail. At least it'll time out the build after 120 seconds instead of the job-global timeout - but maybe there needs to be some other "cleanup" done when the adb shell command fails.
          Hide
          orrc Christopher Orr added a comment -

          Interesting. I'm not sure I've ever actually seen adb hang at that point, even although I run pretty much the same setup as you. Most hangs (while rare for me) seem to be during adb install.

          Anyway, thanks for the patch. Keep us updated how it works on your installation and I'll look at getting it in the next release.

          Show
          orrc Christopher Orr added a comment - Interesting. I'm not sure I've ever actually seen adb hang at that point, even although I run pretty much the same setup as you. Most hangs (while rare for me) seem to be during adb install . Anyway, thanks for the patch. Keep us updated how it works on your installation and I'll look at getting it in the next release.
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          I'll let you know if the error reoccurs. I've been automating emulators in our own testing infrastructure before I switched to using your plugin, and I had a lot of issues with adb hanging on arbitrary commands. Had to add a lot of timeouts-killserver-start-server-retry blocks to them to make it more robust. :|

          Show
          jorgenpt Jørgen Tjernø added a comment - I'll let you know if the error reoccurs. I've been automating emulators in our own testing infrastructure before I switched to using your plugin, and I had a lot of issues with adb hanging on arbitrary commands. Had to add a lot of timeouts-killserver-start-server-retry blocks to them to make it more robust. :|
          Hide
          orrc Christopher Orr added a comment -

          Do you see this while running multiple emulators in parallel, or quite often just with a single emulator?

          If you can get any stack traces or other info, it would (possibly) be helpful to add your voice to this bug: http://b.android.com/10255

          Show
          orrc Christopher Orr added a comment - Do you see this while running multiple emulators in parallel, or quite often just with a single emulator? If you can get any stack traces or other info, it would (possibly) be helpful to add your voice to this bug: http://b.android.com/10255
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Mostly with just a single emulator, yeah, which is why killserver-startserver is a usable fix. I'm guessing adb disconnect+adb connect would have similar effect. Starred the issue, but I haven't really dug into the issue, just worked around it

          Show
          jorgenpt Jørgen Tjernø added a comment - Mostly with just a single emulator, yeah, which is why killserver-startserver is a usable fix. I'm guessing adb disconnect+adb connect would have similar effect. Starred the issue, but I haven't really dug into the issue, just worked around it
          Hide
          orrc Christopher Orr added a comment -

          Have you still been running with this patch? Does it look good?

          Show
          orrc Christopher Orr added a comment - Have you still been running with this patch? Does it look good?
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Yes, since I applied it we haven't had any long-running hangs of adb getprop. No other issues with it AFAIK.

          Show
          jorgenpt Jørgen Tjernø added a comment - Yes, since I applied it we haven't had any long-running hangs of adb getprop. No other issues with it AFAIK.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jørgen Tjernø
          Path:
          src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e
          Log:
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jørgen Tjernø Path: src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e Log: [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.
          Hide
          dogfood dogfood added a comment -

          Integrated in plugins_android-emulator #3
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Christopher Orr :
          Files :

          • src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          Show
          dogfood dogfood added a comment - Integrated in plugins_android-emulator #3 [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs. Christopher Orr : Files : src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          Hide
          orrc Christopher Orr added a comment -

          Version 1.18 has now been released, which includes this update.

          Show
          orrc Christopher Orr added a comment - Version 1.18 has now been released, which includes this update.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jørgen Tjernø
          Path:
          src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e
          Log:
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jørgen Tjernø Path: src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e Log: [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

            People

            • Assignee:
              orrc Christopher Orr
              Reporter:
              jorgenpt Jørgen Tjernø
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: