Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32915

Spot instance launched one after another until capacity reached for single task in queue

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: ec2-plugin
    • Environment:
      Jenkins ver. 1.647 java7
      slaves java8 from oracle
      EC2 plugin 1.31
      Launched in VPC in AWS along with Slaves. Open ports 443, 22, 9090 (for slave communication) communication in VPC allowed only using private ip addresses.
    • Similar Issues:

      Description

      For single element in queue with not slaves attached and master executors set to 0 plugin as designed starts launching spot instances. It does not stop physically until capacity is reached. It still tries until task from queue is picked up by worker that launches after ~3m

      logs ALL:

      Feb 12, 2016 12:17:38 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Launching ami-6c645106 for template Jenkins Build Agent
      Feb 12, 2016 12:17:39 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Spot instance id in provision: sir-02g4tk2d
      Feb 12, 2016 12:17:39 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 0
      Feb 12, 2016 12:17:39 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      Started provisioning Jenkins Build Agent (ami-6c645106) from ec2-Amazon profile with 1 executors. Remaining excess workload: -0.282
      Feb 12, 2016 12:17:39 PM INFO hudson.plugins.ec2.EC2Cloud$1 call
      Expected - Spot instance null failed to connect on initial provision
      Feb 12, 2016 12:17:47 PM INFO hudson.slaves.NodeProvisioner$2 run
      Jenkins Build Agent (ami-6c645106) provisioning successfully completed. We have now 3 computer(s)
      Feb 12, 2016 12:17:48 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Launching ami-6c645106 for template Jenkins Build Agent
      Feb 12, 2016 12:17:49 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Spot instance id in provision: sir-02g3vkvj
      Feb 12, 2016 12:17:49 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 0
      Feb 12, 2016 12:17:49 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      Started provisioning Jenkins Build Agent (ami-6c645106) from ec2-Amazon profile with 1 executors. Remaining excess workload: -0.254
      Feb 12, 2016 12:17:49 PM INFO hudson.plugins.ec2.EC2Cloud$1 call
      Expected - Spot instance null failed to connect on initial provision
      Feb 12, 2016 12:17:57 PM INFO hudson.slaves.NodeProvisioner$2 run
      Jenkins Build Agent (ami-6c645106) provisioning successfully completed. We have now 4 computer(s)
      Feb 12, 2016 12:17:58 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Launching ami-6c645106 for template Jenkins Build Agent
      Feb 12, 2016 12:17:59 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Spot instance id in provision: sir-02g254a9
      Feb 12, 2016 12:17:59 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 0
      Feb 12, 2016 12:17:59 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      Started provisioning Jenkins Build Agent (ami-6c645106) from ec2-Amazon profile with 1 executors. Remaining excess workload: -0.229
      Feb 12, 2016 12:17:59 PM INFO hudson.plugins.ec2.EC2Cloud$1 call
      Expected - Spot instance null failed to connect on initial provision
      Feb 12, 2016 12:18:07 PM INFO hudson.slaves.NodeProvisioner$2 run
      Jenkins Build Agent (ami-6c645106) provisioning successfully completed. We have now 5 computer(s)
      Feb 12, 2016 12:18:08 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Launching ami-6c645106 for template Jenkins Build Agent
      Feb 12, 2016 12:18:09 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Spot instance id in provision: sir-02g5r01b
      Feb 12, 2016 12:18:09 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 0
      Feb 12, 2016 12:18:09 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      Started provisioning Jenkins Build Agent (ami-6c645106) from ec2-Amazon profile with 1 executors. Remaining excess workload: -0.206
      Feb 12, 2016 12:18:09 PM INFO hudson.plugins.ec2.EC2Cloud$1 call
      Expected - Spot instance null failed to connect on initial provision
      Feb 12, 2016 12:18:17 PM INFO hudson.slaves.NodeProvisioner$2 run
      Jenkins Build Agent (ami-6c645106) provisioning successfully completed. We have now 6 computer(s)
      Feb 12, 2016 12:18:19 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:18:29 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveIfPossible
      Cannot provision - no capacity for instances: -3
      Feb 12, 2016 12:18:29 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:18:39 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveIfPossible
      Cannot provision - no capacity for instances: -3
      Feb 12, 2016 12:18:39 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:18:49 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:18:59 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:19:09 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:19:17 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Started EC2 alive slaves monitor
      Feb 12, 2016 12:19:19 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:19:19 PM INFO hudson.plugins.ec2.EC2SlaveMonitor execute
      EC2 instance is dead: null
      Feb 12, 2016 12:19:19 PM INFO hudson.plugins.ec2.EC2SpotSlave terminate
      Canceled Spot request: sir-02g5r01b
      Feb 12, 2016 12:19:20 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished EC2 alive slaves monitor. 2,391 ms
      Feb 12, 2016 12:19:28 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Launching ami-6c645106 for template Jenkins Build Agent
      Feb 12, 2016 12:19:29 PM INFO hudson.plugins.ec2.SlaveTemplate provisionSpot
      Spot instance id in provision: sir-02fzgvxj
      Feb 12, 2016 12:19:29 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 0
      Feb 12, 2016 12:19:29 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      Started provisioning Jenkins Build Agent (ami-6c645106) from ec2-Amazon profile with 1 executors. Remaining excess workload: -0.089
      Feb 12, 2016 12:19:29 PM INFO hudson.plugins.ec2.EC2Cloud$1 call
      Expected - Spot instance null failed to connect on initial provision
      Feb 12, 2016 12:19:37 PM INFO hudson.slaves.NodeProvisioner$2 run
      Jenkins Build Agent (ami-6c645106) provisioning successfully completed. We have now 6 computer(s)
      Feb 12, 2016 12:19:39 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:19:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #86 from /10.0.0.186:44954
      Feb 12, 2016 12:19:43 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #86 with /10.0.0.186:44954 is aborted: 381e79a9-c093-47df-9d79-0b921c35135d is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:19:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #87 from /10.0.0.186:44955
      Feb 12, 2016 12:19:43 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #87 with /10.0.0.186:44955 is aborted: 381e79a9-c093-47df-9d79-0b921c35135d is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:19:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #88 from /10.0.0.186:44956
      Feb 12, 2016 12:19:43 PM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
      Connection #88 failed
      java.io.EOFException
      	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:589)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:150)
      
      Feb 12, 2016 12:19:49 PM INFO hudson.plugins.ec2.EC2Cloud provision
      Attempting provision - finished, excess workload: 1
      Feb 12, 2016 12:19:53 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #89 from /10.0.0.88:47938
      Feb 12, 2016 12:19:54 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #90 from /10.0.0.234:45467
      Feb 12, 2016 12:19:56 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #91 from /10.0.0.9:37386
      Feb 12, 2016 12:24:41 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #92 from /10.0.0.9:37394
      Feb 12, 2016 12:24:41 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #92 with /10.0.0.9:37394 is aborted: b5c0b783-ccf3-44dd-ad2a-e89f18327dc2 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:41 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #93 from /10.0.0.9:37395
      Feb 12, 2016 12:24:41 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #93 with /10.0.0.9:37395 is aborted: b5c0b783-ccf3-44dd-ad2a-e89f18327dc2 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:41 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #94 from /10.0.0.9:37396
      Feb 12, 2016 12:24:41 PM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
      Connection #94 failed
      java.io.EOFException
      	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:589)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:150)
      
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #95 from /10.0.0.88:47946
      Feb 12, 2016 12:24:42 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #95 with /10.0.0.88:47946 is aborted: 82aeacc2-cf06-4cca-8374-9b9c0033f8e9 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #96 from /10.0.0.88:47947
      Feb 12, 2016 12:24:42 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #96 with /10.0.0.88:47947 is aborted: 82aeacc2-cf06-4cca-8374-9b9c0033f8e9 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #97 from /10.0.0.88:47948
      Feb 12, 2016 12:24:42 PM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
      Connection #97 failed
      java.io.EOFException
      	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:589)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:150)
      
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #98 from /10.0.0.186:44961
      Feb 12, 2016 12:24:42 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #98 with /10.0.0.186:44961 is aborted: 381e79a9-c093-47df-9d79-0b921c35135d is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #99 from /10.0.0.186:44962
      Feb 12, 2016 12:24:42 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #99 with /10.0.0.186:44962 is aborted: 381e79a9-c093-47df-9d79-0b921c35135d is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:42 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #100 from /10.0.0.186:44963
      Feb 12, 2016 12:24:42 PM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
      Connection #100 failed
      java.io.EOFException
      	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:589)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:150)
      
      Feb 12, 2016 12:24:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #101 from /10.0.0.234:45488
      Feb 12, 2016 12:24:43 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #101 with /10.0.0.234:45488 is aborted: 01df9f0d-6b5d-4051-b944-dffc698ae1d3 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #102 from /10.0.0.234:45489
      Feb 12, 2016 12:24:43 PM WARNING jenkins.slaves.JnlpSlaveHandshake error
      TCP slave agent connection handler #102 with /10.0.0.234:45489 is aborted: 01df9f0d-6b5d-4051-b944-dffc698ae1d3 is already connected to this master. Rejecting this connection.
      Feb 12, 2016 12:24:43 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #103 from /10.0.0.234:45490
      Feb 12, 2016 12:24:43 PM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
      Connection #103 failed
      java.io.EOFException
      	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:589)
      	at java.io.DataInputStream.readUTF(DataInputStream.java:564)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:150)
      
      Feb 12, 2016 12:24:50 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      Accepted connection #104 from /10.0.0.243:48286
      

      screenshots of the configuration are attached

      Please contact me here on via email if I can help in any way

        Attachments

          Activity

            People

            • Assignee:
              zajacmp3 Mateusz Zając
              Reporter:
              zajacmp3 Mateusz Zając
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: