Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26725

Tries spawning more and more EC2 slaves when master runs out of space

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • ec2-plugin
    • None
    • Linux li245-114 2.6.39.1-linode34 #1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux

      First we start seeing these:

      Feb 1, 2015 7:11:00 AM hudson.model.Executor run
      SEVERE: Unexpected executor death
      java.io.IOException: No space left on device
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:300)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
      at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:315)
      at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:148)
      at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
      at java.io.BufferedWriter.close(BufferedWriter.java:265)
      at hudson.util.AtomicFileWriter.close(AtomicFileWriter.java:94)
      at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:109)
      at hudson.util.TextFile.write(TextFile.java:121)
      at hudson.model.Job.saveNextBuildNumber(Job.java:274)
      at hudson.model.Job.assignBuildNumber(Job.java:332)
      at hudson.model.Run.<init>(Run.java:286)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:167)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:34)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:534)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:186)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1001)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1194)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor.run(Executor.java:213)
      Feb 1, 2015 7:13:49 AM hudson.model.AsyncPeriodicWork$1 run

      And then more instances get started:

      INFO: Excess workload after pending Spot instances: 1
      Feb 1, 2015 12:39:37 PM hudson.plugins.ec2.EC2Cloud addProvisionedSlave
      INFO: Provisioning for AMI ami-cc8871a4; Estimated number of total slaves: 15; Estimated number of slaves for ami ami-cc8871a4: 11
      Launching ami-cc8871a4 for template generic-slave-cc8871a4
      Feb 1, 2015 12:39:37 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning generic-slave-cc8871a4 (ami-cc8871a4) from ec2-us-east-1 with 1 executors. Remaining excess workload: -0.07
      Feb 1, 2015 12:39:37 PM hudson.plugins.ec2.SlaveTemplate provisionOndemand
      INFO: Launching ami-cc8871a4 for template generic-slave-cc8871a4
      Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [

      {Name: image-id,Values: [ami-cc8871a4]}

      ,

      {Name: instance.group-name,Values: [default, build_slaves]}

      ,

      {N ame: key-name,Values: [artsy_build_keypair]}

      ,

      {Name: instance-type,Values: [c1.medium]}

      ,

      {Name: tag:Name,Values: [build-slave-20140619]}

      ,

      {Name: instance-state-name,Values: [stopped, sto pping]}

      ],}
      No existing instance found - created: {InstanceId: i-988b7077,ImageId: ami-cc8871a4,State:

      Unable to find source-code formatter for language: 0,name: pending. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      

      

      ,PrivateDnsName: ,PublicDnsName: ,StateTransitionReason: ,KeyName: artsy
      
_build_keypair,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: c1.medium,LaunchTime: Sun Feb 01 12:39:38 EST 2015,Placement: 
      
{AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default}

      ,
      
KernelId: aki-88aa75e1,Monitoring: 
      
{State: disabled}

      ,StateReason: 
      

      

      Unable to find source-code formatter for language:  pending,message: pending. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      

      

      ,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings:
      
 [],VirtualizationType: paravirtual,ClientToken: ddc70314-0aff-4d43-8f84-ce4658bf01b5,Tags: [
      
{Key: Name,Value: build-slave-20140619}

      ],SecurityGroups: [
      
{GroupName: default,GroupId: sg-4fc
61326}

      , 
      
{GroupName: build_slaves,GroupId: sg-b751d4df}

      ],Hypervisor: xen,NetworkInterfaces: [],EbsOptimized: false,}
      
Feb 1, 2015 12:39:38 PM hudson.plugins.ec2.SlaveTemplate provisionOndemand
      
INFO: No existing instance found - created: {InstanceId: i-988b7077,ImageId: ami-cc8871a4,State: 
      

      

      Unable to find source-code formatter for language:  0,name: pending. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      

      

      ,PrivateDnsName: ,PublicDnsName: ,StateTransitionReason: ,KeyName:
      
 artsy_build_keypair,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: c1.medium,LaunchTime: Sun Feb 01 12:39:38 EST 2015,Placement: 
      
{AvailabilityZone: us-east-1a,GroupName: ,Tenancy: def
ault}

      ,KernelId: aki-88aa75e1,Monitoring: 
      
{State: disabled}

      ,StateReason: 
      

      

      Unable to find source-code formatter for language:  pending,message: pending. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      

      

      ,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMap
      
pings: [],VirtualizationType: paravirtual,ClientToken: ddc70314-0aff-4d43-8f84-ce4658bf01b5,Tags: [
      
{Key: Name,Value: build-slave-20140619}

      ],SecurityGroups: [
      
{GroupName: default,GroupId: 
sg-4fc61326}

      , 
      
{GroupName: build_slaves,GroupId: sg-b751d4df}

      ],Hypervisor: xen,NetworkInterfaces: [],EbsOptimized: false,}
      


      Now it will keep spawning instances again and again until EC2 stops giving it more (ours is set to a 50 instance limit for example).
      
            
      

      

        1. jenkins.log
          52 kB
        2. jenkins.log.1
          1.59 MB
          


          

            








            
                        

            
    
            
        
    
            
    
            
        

            
            



    
    francisu
            Francis Upton
            
        
    
            
            

                                            

            
    
            
        
    
            
    
            
        

            
            



    
    dblock
            Daniel Doubrovkine
            
        
    
    
            

                            
            
    
            
                        
            
        
            Votes:
            
        
            
                                1
            
                                                Vote for this issue
                    
            
    
            
                            
            
        
            Watchers:
            
        
            
                                
                                
                            2
            
                                        Start watching this issue
                    
            
    
            
        
            

              









              
            
              
    
              
                    Created:
            
              
    
              
        
                                                                                                                                 
    
              

              
            
              
    
              
                    Updated:
            
              
    
              
        
                                                                                                                                 
    
              

              
            
              
    
              
                    Resolved: