Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59216

LinkageError in ec2 spot fleet while performing jenkins UserRequest

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Environment:
    • Similar Issues:

      Description

      Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.

       

      [Environment]

      jenkins 2.176.3 with all plugins in latest version
      java version (both jenkins master and spot fleet) 1.8.0
      – plugins –
      ec2-fleet-plugin 1.10.2
      git-client 2.8.2
      junit 1.28
      --------------

      Jenkins master on ec2 r5.large, Amazon linux
      Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

       

      [Scenario]

      I use jenkins with spot-fleet plugin for CI

      CI all pass in spot-fleet with initial ec2 mechine

      While ec2 scale out, CI fail in new instance with "remote call on  i-xxxxxx failed"

      Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

      remoting_log.txt 

       

      [Reproduce]

      3 CI job with identical codebase run at the same time
      a ec2-spot-fleet with init 2 instance

      In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.

      • Result : CI pass in one job and fail in other two

      [More information]

      1. Below is the jenkinsfile with sensitive information replaced

      Jenkinsfile

      2. I face the problem in two scenario, because docker build and testing in different spot fleet

      scenario A:  

      building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

      scenario B:

      is the reproduce experiment, happen in spot-fleet used in testing stage
      new ec2 (scaled out) fail in CI testing while initial instances success
      you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

       

      3. Slave log viewed in jenkins
      ErrorInScaledOutInstance.pdf

        Attachments

        1. ec2_1_success.png
          ec2_1_success.png
          156 kB
        2. ec2_2_fail.png
          ec2_2_fail.png
          517 kB
        3. ec2_3_fail.png
          ec2_3_fail.png
          872 kB
        4. ec2_fail_build.png
          ec2_fail_build.png
          105 kB
        5. ErrorInScaledOutInstance.pdf
          1.13 MB
        6. Jenkinsfile
          14 kB
        7. Jenkinsfile.txt
          14 kB
        8. remoting_log.txt
          35 kB

          Activity

          Hide
          holis TzuChin Hsu added a comment -

          Hi, this problem have been "SOLVED"
          That's because "yum update" when jenkins slave have already run in machine,
             and some java lib might be affected, so cause "LinakageError", after update AMI this problem has not be seen anymore.

          Show
          holis TzuChin Hsu added a comment - Hi, this problem have been "SOLVED" That's because "yum update" when jenkins slave have already run in machine,    and some java lib might be affected, so cause "LinakageError", after update AMI this problem has not be seen anymore.

            People

            • Assignee:
              schmutze Chad Schmutzer
              Reporter:
              holis TzuChin Hsu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: