Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10421

Emulator occasionally hangs indefinitely on startup

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Ubuntu 10.04, linux 2.6.32, Jenkins 1.421.
      Android SDK r10
    • Similar Issues:

      Description

      When starting up the emulator, adb shell getprop dev.bootcomplete will sometimes hang indefinitely. This will cause the entire job to hang. Below is a log w/ timestamps where I have a 15 minute build timeout kick in.
      Since adb is prone to hanging, every adb command (or at least these first ones) should have a watchdog thread killing them if they take too long.
      I'm guessing an adb disconnect/connect combo after the hanging command might be needed too.

      08:17:35 Building on master
      08:17:35 [android] Using Android SDK: /usr/local/android
      08:17:35 $ /usr/local/android/tools/emulator -snapshot-list -no-window -avd hudson_en-US_160_WVGA_android-4
      08:17:35 [android] Starting Android emulator from snapshot
      08:17:35 $ /usr/local/android/tools/emulator -no-boot-anim -ports 47225,56719 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-4 -snapshot jenkins -no-snapshot-save -no-window
      08:17:35 emulator: warning: opening audio output failed
      08:17:35
      08:17:40 $ /usr/local/android/platform-tools/adb connect emu:47225,56719
      08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 logcat -v time
      08:17:40 [android] Waiting for emulator to finish booting...
      08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 shell getprop dev.bootcomplete
      08:32:35 Build timed out (after 15 minutes). Marking the build as aborted.
      08:32:35 [android] Interrupted while waiting for emulator to finish booting.
      08:32:35 [android] Timed-out after waiting 120 seconds for emulator
      08:32:35 [android] Stopping Android emulator
      08:32:35 [android] Archiving emulator log
      08:32:35 Notifying upstream projects of job completion
      08:32:35 Finished: NOT_BUILT

        Attachments

          Activity

          jorgenpt Jørgen Tjernø created issue -
          jorgenpt Jørgen Tjernø made changes -
          Field Original Value New Value
          Description When starting up the emulator, {{adb shell getprop dev.bootcomplete}} will sometimes hang indefinitely. This will cause the entire job to hang.
          The below log lacks timestamps, so it's not visible, but the execution took over 30 minutes before I aborted the job, at which point it told me it had timed out after 120 seconds. Since adb is prone to hanging, every adb command (or at least these first ones) should have a watchdog thread killing them if they take too long.
          I'm guessing an adb disconnect/connect combo after the hanging command might be needed too.

          {quote}[android] Using Android SDK: /usr/local/android
          $ /usr/local/android/tools/emulator -snapshot-list -no-window -avd hudson_en-US_160_WVGA_android-4
          [android] Starting Android emulator from snapshot
          $ /usr/local/android/tools/emulator -no-boot-anim -ports 49280,42013 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-4 -snapshot jenkins -no-snapshot-save -no-window
          emulator: warning: opening audio output failed

          $ /usr/local/android/platform-tools/adb connect emu:49280,42013
          $ /usr/local/android/platform-tools/adb -s emulator-49280 logcat -v time
          [android] Waiting for emulator to finish booting...
          $ /usr/local/android/platform-tools/adb -s emulator-49280 shell getprop dev.bootcomplete
          [android] Interrupted while waiting for emulator to finish booting.
          [android] Timed-out after waiting 120 seconds for emulator
          [android] Stopping Android emulator
          [android] Archiving emulator log
          Notifying upstream projects of job completion
          Finished: NOT_BUILT{quote}
          When starting up the emulator, {{adb shell getprop dev.bootcomplete}} will sometimes hang indefinitely. This will cause the entire job to hang. Below is a log w/ timestamps where I have a 15 minute build timeout kick in.
          Since adb is prone to hanging, every adb command (or at least these first ones) should have a watchdog thread killing them if they take too long.
          I'm guessing an adb disconnect/connect combo after the hanging command might be needed too.

          {quote}08:17:35 Building on master
          08:17:35 [android] Using Android SDK: /usr/local/android
          08:17:35 $ /usr/local/android/tools/emulator -snapshot-list -no-window -avd hudson_en-US_160_WVGA_android-4
          08:17:35 [android] Starting Android emulator from snapshot
          08:17:35 $ /usr/local/android/tools/emulator -no-boot-anim -ports 47225,56719 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-4 -snapshot jenkins -no-snapshot-save -no-window
          08:17:35 emulator: warning: opening audio output failed
          08:17:35
          08:17:40 $ /usr/local/android/platform-tools/adb connect emu:47225,56719
          08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 logcat -v time
          08:17:40 [android] Waiting for emulator to finish booting...
          08:17:40 $ /usr/local/android/platform-tools/adb -s emulator-47225 shell getprop dev.bootcomplete
          08:32:35 Build timed out (after 15 minutes). Marking the build as aborted.
          08:32:35 [android] Interrupted while waiting for emulator to finish booting.
          08:32:35 [android] Timed-out after waiting 120 seconds for emulator
          08:32:35 [android] Stopping Android emulator
          08:32:35 [android] Archiving emulator log
          08:32:35 Notifying upstream projects of job completion
          08:32:35 Finished: NOT_BUILT{quote}
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Out of the last 18 builds, 4 of them have hung like this. That's more than 1 out of every 5 builds!

          Show
          jorgenpt Jørgen Tjernø added a comment - Out of the last 18 builds, 4 of them have hung like this. That's more than 1 out of every 5 builds!
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          I've applied this patch to our installation, and I'm going to see if it remedies the issue - or if it'll just cause later commands to fail.

          At least it'll time out the build after 120 seconds instead of the job-global timeout - but maybe there needs to be some other "cleanup" done when the adb shell command fails.

          Show
          jorgenpt Jørgen Tjernø added a comment - I've applied this patch to our installation, and I'm going to see if it remedies the issue - or if it'll just cause later commands to fail. At least it'll time out the build after 120 seconds instead of the job-global timeout - but maybe there needs to be some other "cleanup" done when the adb shell command fails.
          jorgenpt Jørgen Tjernø made changes -
          Hide
          orrc Christopher Orr added a comment -

          Interesting. I'm not sure I've ever actually seen adb hang at that point, even although I run pretty much the same setup as you. Most hangs (while rare for me) seem to be during adb install.

          Anyway, thanks for the patch. Keep us updated how it works on your installation and I'll look at getting it in the next release.

          Show
          orrc Christopher Orr added a comment - Interesting. I'm not sure I've ever actually seen adb hang at that point, even although I run pretty much the same setup as you. Most hangs (while rare for me) seem to be during adb install . Anyway, thanks for the patch. Keep us updated how it works on your installation and I'll look at getting it in the next release.
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          I'll let you know if the error reoccurs. I've been automating emulators in our own testing infrastructure before I switched to using your plugin, and I had a lot of issues with adb hanging on arbitrary commands. Had to add a lot of timeouts-killserver-start-server-retry blocks to them to make it more robust. :|

          Show
          jorgenpt Jørgen Tjernø added a comment - I'll let you know if the error reoccurs. I've been automating emulators in our own testing infrastructure before I switched to using your plugin, and I had a lot of issues with adb hanging on arbitrary commands. Had to add a lot of timeouts-killserver-start-server-retry blocks to them to make it more robust. :|
          Hide
          orrc Christopher Orr added a comment -

          Do you see this while running multiple emulators in parallel, or quite often just with a single emulator?

          If you can get any stack traces or other info, it would (possibly) be helpful to add your voice to this bug: http://b.android.com/10255

          Show
          orrc Christopher Orr added a comment - Do you see this while running multiple emulators in parallel, or quite often just with a single emulator? If you can get any stack traces or other info, it would (possibly) be helpful to add your voice to this bug: http://b.android.com/10255
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Mostly with just a single emulator, yeah, which is why killserver-startserver is a usable fix. I'm guessing adb disconnect+adb connect would have similar effect. Starred the issue, but I haven't really dug into the issue, just worked around it

          Show
          jorgenpt Jørgen Tjernø added a comment - Mostly with just a single emulator, yeah, which is why killserver-startserver is a usable fix. I'm guessing adb disconnect+adb connect would have similar effect. Starred the issue, but I haven't really dug into the issue, just worked around it
          Hide
          orrc Christopher Orr added a comment -

          Have you still been running with this patch? Does it look good?

          Show
          orrc Christopher Orr added a comment - Have you still been running with this patch? Does it look good?
          Hide
          jorgenpt Jørgen Tjernø added a comment -

          Yes, since I applied it we haven't had any long-running hangs of adb getprop. No other issues with it AFAIK.

          Show
          jorgenpt Jørgen Tjernø added a comment - Yes, since I applied it we haven't had any long-running hangs of adb getprop. No other issues with it AFAIK.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jørgen Tjernø
          Path:
          src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e
          Log:
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jørgen Tjernø Path: src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e Log: [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.
          scm_issue_link SCM/JIRA link daemon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          dogfood dogfood added a comment -

          Integrated in plugins_android-emulator #3
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Christopher Orr :
          Files :

          • src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          Show
          dogfood dogfood added a comment - Integrated in plugins_android-emulator #3 [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs. Christopher Orr : Files : src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          Hide
          orrc Christopher Orr added a comment -

          Version 1.18 has now been released, which includes this update.

          Show
          orrc Christopher Orr added a comment - Version 1.18 has now been released, which includes this update.
          orrc Christopher Orr made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jørgen Tjernø
          Path:
          src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java
          http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e
          Log:
          [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jørgen Tjernø Path: src/main/java/hudson/plugins/android_emulator/AndroidEmulator.java http://jenkins-ci.org/commit/android-emulator-plugin/4cd68f5bbd13ab6800c050d4caa483e53723488e Log: [FIXED JENKINS-10421] Use Proc.joinWithTimeout to prevent adb hangs.
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 140613 ] JNJira + In-Review [ 205235 ]

            People

            • Assignee:
              orrc Christopher Orr
              Reporter:
              jorgenpt Jørgen Tjernø
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: