Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34216

Failed to reset the request input stream

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: s3-plugin
    • Labels:
      None
    • Environment:
      s3-plugin 0.9.2
    • Similar Issues:

      Description

      After upgrading the S3 plugin to version 0.9.2 we are seeing S3 uploads fail intermittently. Below is a stack trace (with some details redacted):

      ```
      Publish artifacts to S3 Bucket Using S3 profile: REDACTED
      Publish artifacts to S3 Bucket bucket=REDACTED, file=9fc530ed8ac7866e6cfcc5132c34614b1e5e6dbd region=us-east-1, upload from slave=false managed=false , server encryption false
      Publish artifacts to S3 Bucket bucket=REDACTED, file=master region=us-east-1, upload from slave=false managed=false , server encryption false
      ERROR: Failed to upload files
      java.io.IOException: Call fails for Destination [bucketName=REDACTED, objectName=REDACTED]: com.amazonaws.ResetException: Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int):: Failed after 1 tries.
      at hudson.plugins.s3.S3Profile.repeat(S3Profile.java:220)
      at hudson.plugins.s3.S3Profile.upload(S3Profile.java:145)
      at hudson.plugins.s3.S3BucketPublisher.perform(S3BucketPublisher.java:193)
      at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
      at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
      at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:723)
      at hudson.model.Build$BuildExecution.post2(Build.java:185)
      at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:668)
      at hudson.model.Run.execute(Run.java:1763)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:98)
      at hudson.model.Executor.run(Executor.java:410)
      Caused by: com.amazonaws.ResetException: Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
      at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:755)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
      at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
      at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3659)
      at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:2795)
      at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2780)
      at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadPartsInSeries(UploadCallable.java:255)
      at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:189)
      at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:121)
      at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
      at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Resetting to invalid mark
      at java.io.BufferedInputStream.reset(BufferedInputStream.java:448)
      at com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:139)
      at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
      at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:753)
      ... 16 more
      Build step 'Publish artifacts to S3 Bucket' changed build result to UNSTABLE
      ```

      The file being uploaded is a 450MB tarball.

        Attachments

          Activity

          joshuaspence Joshua Spence created issue -
          Hide
          jasonm Jason Milley added a comment -

          Just confirming that I'm also seeing the same thing since upgrading to 0.10.3 yesterday.

          Show
          jasonm Jason Milley added a comment - Just confirming that I'm also seeing the same thing since upgrading to 0.10.3 yesterday.
          Hide
          kcpublic Kevin Chen added a comment -

          We're having the same problem since upgrading to 0.10.3. It's intermittent.

          Show
          kcpublic Kevin Chen added a comment - We're having the same problem since upgrading to 0.10.3. It's intermittent.
          Hide
          jimilian Alexander A added a comment -

          In new version (0.10.5) should be fixed.

          Show
          jimilian Alexander A added a comment - In new version (0.10.5) should be fixed.
          jimilian Alexander A made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kcpublic Kevin Chen added a comment -

          We upgraded to 0.10.5. We're not getting missing transitive dependencies:

          Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.ManagedHttpClientConnection
          12:37:24 PDT at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1376)
          12:37:24 PDT at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
          12:37:24 PDT at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079)
          12:37:24 PDT at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

          Show
          kcpublic Kevin Chen added a comment - We upgraded to 0.10.5. We're not getting missing transitive dependencies: Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.ManagedHttpClientConnection 12:37:24 PDT at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1376) 12:37:24 PDT at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326) 12:37:24 PDT at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079) 12:37:24 PDT at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          kcpublic Kevin Chen made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          kcpublic Kevin Chen added a comment -

          Below is the full stack trace:

          12:37:24 PDT ERROR: Build step failed with exception
          12:37:24 PDT com.amazonaws.AmazonClientException: Unable to complete transfer: org/apache/http/conn/ManagedHttpClientConnection
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:277)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:261)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103)
          12:37:24 PDT at hudson.plugins.s3.Uploads.finishUploading(Uploads.java:40)
          12:37:24 PDT at hudson.plugins.s3.callable.S3WaitUploadCallable.invoke(S3WaitUploadCallable.java:19)
          12:37:24 PDT at hudson.plugins.s3.callable.S3WaitUploadCallable.invoke(S3WaitUploadCallable.java:10)
          12:37:24 PDT at hudson.plugins.s3.S3Profile.invoke(S3Profile.java:203)
          12:37:24 PDT at hudson.plugins.s3.S3Profile.waitUploads(S3Profile.java:195)
          12:37:24 PDT at hudson.plugins.s3.S3Profile.upload(S3Profile.java:174)
          12:37:24 PDT at hudson.plugins.s3.S3BucketPublisher.perform(S3BucketPublisher.java:192)
          12:37:24 PDT at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
          12:37:24 PDT at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32)
          12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
          12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:723)
          12:37:24 PDT at hudson.model.Build$BuildExecution.post2(Build.java:185)
          12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:668)
          12:37:24 PDT at hudson.model.Run.execute(Run.java:1763)
          12:37:24 PDT at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          12:37:24 PDT at hudson.model.ResourceController.execute(ResourceController.java:98)
          12:37:24 PDT at hudson.model.Executor.run(Executor.java:410)
          12:37:24 PDT Caused by: java.lang.NoClassDefFoundError: org/apache/http/conn/ManagedHttpClientConnection
          12:37:24 PDT at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:50)
          12:37:24 PDT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
          12:37:24 PDT at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
          12:37:24 PDT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
          12:37:24 PDT at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
          12:37:24 PDT at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
          12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837)
          12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
          12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
          12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
          12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
          12:37:24 PDT at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826)
          12:37:24 PDT at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:2708)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.initiateMultipartUpload(UploadCallable.java:350)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:178)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:121)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
          12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
          12:37:24 PDT at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:37:24 PDT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          12:37:24 PDT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          12:37:24 PDT at java.lang.Thread.run(Thread.java:745)
          12:37:24 PDT Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.ManagedHttpClientConnection
          12:37:24 PDT at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1376)
          12:37:24 PDT at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
          12:37:24 PDT at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079)
          12:37:24 PDT at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          12:37:24 PDT ... 22 more

          Show
          kcpublic Kevin Chen added a comment - Below is the full stack trace: 12:37:24 PDT ERROR: Build step failed with exception 12:37:24 PDT com.amazonaws.AmazonClientException: Unable to complete transfer: org/apache/http/conn/ManagedHttpClientConnection 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:277) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:261) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103) 12:37:24 PDT at hudson.plugins.s3.Uploads.finishUploading(Uploads.java:40) 12:37:24 PDT at hudson.plugins.s3.callable.S3WaitUploadCallable.invoke(S3WaitUploadCallable.java:19) 12:37:24 PDT at hudson.plugins.s3.callable.S3WaitUploadCallable.invoke(S3WaitUploadCallable.java:10) 12:37:24 PDT at hudson.plugins.s3.S3Profile.invoke(S3Profile.java:203) 12:37:24 PDT at hudson.plugins.s3.S3Profile.waitUploads(S3Profile.java:195) 12:37:24 PDT at hudson.plugins.s3.S3Profile.upload(S3Profile.java:174) 12:37:24 PDT at hudson.plugins.s3.S3BucketPublisher.perform(S3BucketPublisher.java:192) 12:37:24 PDT at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) 12:37:24 PDT at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32) 12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782) 12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:723) 12:37:24 PDT at hudson.model.Build$BuildExecution.post2(Build.java:185) 12:37:24 PDT at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:668) 12:37:24 PDT at hudson.model.Run.execute(Run.java:1763) 12:37:24 PDT at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 12:37:24 PDT at hudson.model.ResourceController.execute(ResourceController.java:98) 12:37:24 PDT at hudson.model.Executor.run(Executor.java:410) 12:37:24 PDT Caused by: java.lang.NoClassDefFoundError: org/apache/http/conn/ManagedHttpClientConnection 12:37:24 PDT at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:50) 12:37:24 PDT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) 12:37:24 PDT at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) 12:37:24 PDT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) 12:37:24 PDT at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) 12:37:24 PDT at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) 12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837) 12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) 12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) 12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) 12:37:24 PDT at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) 12:37:24 PDT at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826) 12:37:24 PDT at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:2708) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.initiateMultipartUpload(UploadCallable.java:350) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:178) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:121) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139) 12:37:24 PDT at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47) 12:37:24 PDT at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:37:24 PDT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 12:37:24 PDT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 12:37:24 PDT at java.lang.Thread.run(Thread.java:745) 12:37:24 PDT Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.ManagedHttpClientConnection 12:37:24 PDT at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1376) 12:37:24 PDT at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326) 12:37:24 PDT at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079) 12:37:24 PDT at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 12:37:24 PDT ... 22 more
          kcpublic Kevin Chen made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Hide
          kcpublic Kevin Chen added a comment -

          We had to downgrade to 0.10.3. That worked.

          We're on Jenkins 2.1. If 0.10.5 is not compatible with Jenkins 2.1, you shouldn't allow the install

          Show
          kcpublic Kevin Chen added a comment - We had to downgrade to 0.10.3. That worked. We're on Jenkins 2.1. If 0.10.5 is not compatible with Jenkins 2.1, you shouldn't allow the install
          jimilian Alexander A made changes -
          Attachment s3.hpi [ 33112 ]
          Hide
          jimilian Alexander A added a comment -

          Hi, Kevin Chen!

          I attached new version to this ticket, could you check it? If it works for you, I will upload it as 0.10.5.1.

          Unfortunately, I didn't recognize that Joshua Spence changed dependency group from `org.jenkins-ci.plugins` to `com.amazonaws` in his PR. Now I changed it back and it should work for Jenkins 2.* as well.

          Show
          jimilian Alexander A added a comment - Hi, Kevin Chen ! I attached new version to this ticket, could you check it? If it works for you, I will upload it as 0.10.5.1. Unfortunately, I didn't recognize that Joshua Spence changed dependency group from `org.jenkins-ci.plugins` to `com.amazonaws` in his PR . Now I changed it back and it should work for Jenkins 2.* as well.
          Hide
          kcpublic Kevin Chen added a comment -

          How do I install this manually? We only use the plugin mechanism via the Jenkins web page.

          Show
          kcpublic Kevin Chen added a comment - How do I install this manually? We only use the plugin mechanism via the Jenkins web page.
          Hide
          kcpublic Kevin Chen added a comment -

          And how do I backout the change if it doesn't work?

          Show
          kcpublic Kevin Chen added a comment - And how do I backout the change if it doesn't work?
          Hide
          kcpublic Kevin Chen added a comment -

          Figured it out by going to the "Advanced" tab and uploading. Running jobs now. Crossing fingers.

          The only other issue is that the plugin version is "0.10.6-SNAPSHOT (private-ea6f0d94-jim)". I hope this doesn't interfere with us receiving future versions.

          Show
          kcpublic Kevin Chen added a comment - Figured it out by going to the "Advanced" tab and uploading. Running jobs now. Crossing fingers. The only other issue is that the plugin version is "0.10.6-SNAPSHOT (private-ea6f0d94-jim)". I hope this doesn't interfere with us receiving future versions.
          Hide
          kcpublic Kevin Chen added a comment -

          Installed it and ran a build. No errors and verified our S3 artifact got deployed.

          I hope the snapshot version above doesn't hinder us.

          Show
          kcpublic Kevin Chen added a comment - Installed it and ran a build. No errors and verified our S3 artifact got deployed. I hope the snapshot version above doesn't hinder us.
          Hide
          joshuaspence Joshua Spence added a comment -

          Ah sorry about that. I'm not a Java developer and I think that this was my first PR for a Java project too.

          Show
          joshuaspence Joshua Spence added a comment - Ah sorry about that. I'm not a Java developer and I think that this was my first PR for a Java project too.
          Hide
          joshuaspence Joshua Spence added a comment -

          Strangely, version 0.10.5 works fine for us on Jenkins 2.9.

          Show
          joshuaspence Joshua Spence added a comment - Strangely, version 0.10.5 works fine for us on Jenkins 2.9.
          Hide
          jimilian Alexander A added a comment -

          Solved in 0.10.5.1

          Show
          jimilian Alexander A added a comment - Solved in 0.10.5.1
          jimilian Alexander A made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 170274 ] JNJira + In-Review [ 198827 ]

            People

            • Assignee:
              jimilian Alexander A
              Reporter:
              joshuaspence Joshua Spence
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: