Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59907

sh steps stuck indefinitely on uncommon architectures (e.g. s390x)

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: durable-task-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.201 (yum installed, master node only)
      durable-task plugin v1.31

      os.arch: s390x
      os.name: Linux (RedHat)
      os.version: 3.10.0-327.el7.s390x
    • Similar Issues:
    • Released As:
      1.33

      Description

      After upgrading to v1.31, the first sh step in a pipeline gets stuck. After few minutes Console Output shows:

      [Pipeline] sh (Get email of the author of last commit)
      process apparently never started in /data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6
      (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      Cannot contact : java.io.FileNotFoundException: File '/data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6/output.txt' does not exist

       

      Eventually, I discovered that a new binary was added in the latest version of this plugin. The script compile-binaries.sh in GitHub suggests that the binary is only built for Linux and MacOS.

       

      Sure enough, when I try to execute the binary myself on an architecture other than amd64, I get:

      -bash: /data/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64: cannot execute binary file

       

      Are other architectures or operating systems (Windows) not supported anymore?

        Attachments

          Issue Links

            Activity

            Hide
            smirky Bogomil Vasilev added a comment - - edited

            Carroll Chiou - Indeed, it is x64 Ubuntu 18.04. Since you mentioned that this ticket is not suitable for my use-case, I opened a new one, describing the problem:
            https://issues.jenkins-ci.org/browse/JENKINS-63253

            Show
            smirky Bogomil Vasilev added a comment - - edited Carroll Chiou - Indeed, it is x64 Ubuntu 18.04. Since you mentioned that this ticket is not suitable for my use-case, I opened a new one, describing the problem: https://issues.jenkins-ci.org/browse/JENKINS-63253
            Hide
            carroll Carroll Chiou added a comment -

            The issue was incorrectly reopened in the past. Closing again until we encounter a bug related to non-windows/non-unix architecture

            Show
            carroll Carroll Chiou added a comment - The issue was incorrectly reopened in the past. Closing again until we encounter a bug related to non-windows/non-unix architecture
            Hide
            carroll Carroll Chiou added a comment - - edited

            Bogomil Vasilev can you tell me what architecture your agent is running on? The thing is, if you are using 1.33 or greater, you should be running the traditional shell-based durable-task where there would not be any issues in running on a non-windows/non-unix architecture.

            It appears you are using a Unix architecture so this actually might not be the right ticket you're looking for.

            This ticket was also incorrectly reopened by a user reporting an issue with an x86 architecture that was not related to this ticket. Closing this ticket again. Will reopen if we discover new issues relating to non-unix/non-windows architectures.

            Show
            carroll Carroll Chiou added a comment - - edited Bogomil Vasilev can you tell me what architecture your agent is running on? The thing is, if you are using 1.33 or greater, you should be running the traditional shell-based durable-task where there would not be any issues in running on a non-windows/non-unix architecture. It appears you are using a Unix architecture so this actually might not be the right ticket you're looking for. This ticket was also incorrectly reopened by a user reporting an issue with an x86 architecture that was not related to this ticket. Closing this ticket again. Will reopen if we discover new issues relating to non-unix/non-windows architectures.
            Hide
            smirky Bogomil Vasilev added a comment -

            I managed to hit this using dir(). This is clearly a bug and causes people to write workarounds.

            At the second stage, using dir(), it gets stuck for around 6-7 minutes and eventually it fails with "process apparently never started in /opt@tmp/durable-5a20a76a".

            pipeline {
                agent {
                    docker {
                        label '********'
                        image '**********'
                        registryUrl '************'
                        registryCredentialsId '*******'
                        args '--user root:root'
                    }
                }
                stages {
                    stage('dir-testing') {
                        
                        stages {
                            stage('without dir') {
                                steps {
                                    sh 'cd /opt && ls -l'
                                }
                            }
                            stage('with dir') {
                                steps {
                                    dir('/opt') {
                                        sh 'ls -l'
                                    }
                                }
                            }
                        }
                        post {
                            always {
                                cleanWs()
                            }
                        }
                    }
                }
            }
            
            Started by user **********
            Running in Durability level: MAX_SURVIVABILITY
            [Pipeline] Start of Pipeline
            [Pipeline] node
            Running on ************ in /var/jenkins/workspace/test-cwd-bug
            [Pipeline] {
            [Pipeline] withEnv
            [Pipeline] {
            [Pipeline] withDockerRegistry
            Using the existing docker config file.Removing blacklisted property: auths$ docker login -u ******** -p ******** *********
            WARNING! Using --password via the CLI is insecure. Use --password-stdin.
            Login Succeeded
            [Pipeline] {
            [Pipeline] isUnix
            [Pipeline] sh
            + docker inspect -f . **********
            
            Error: No such object: ***********
            [Pipeline] isUnix
            [Pipeline] sh
            + docker inspect -f . ****************
            .
            [Pipeline] withDockerContainer
            ************* does not seem to be running inside a container
            $ docker run -t -d -u 0:0 --user root:root -w /var/jenkins/workspace/test-cwd-bug -v /var/jenkins/workspace/test-cwd-bug:/var/jenkins/workspace/test-cwd-bug:rw,z -v /var/jenkins/workspace/test-cwd-bug@tmp:/var/jenkins/workspace/test-cwd-bug@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** ********************* cat
            $ docker top 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 -eo pid,comm
            [Pipeline] {
            [Pipeline] stage
            [Pipeline] { (dir-testing)
            [Pipeline] stage
            [Pipeline] { (without dir)
            [Pipeline] sh
            + cd /opt
            + ls -l
            total 8
            drwxr-xr-x 4 root root 4096 Jul 24 15:29 artifactory-scripts
            drwxr-xr-x 1  608  500 4096 Jul 24 15:39 cv25_linux_sdk_2.5
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] stage
            [Pipeline] { (with dir)
            [Pipeline] dir
            Running in /opt
            [Pipeline] {
            [Pipeline] sh
            process apparently never started in /opt@tmp/durable-5a20a76a
            (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
            [Pipeline] }
            [Pipeline] // dir
            [Pipeline] }
            [Pipeline] // stage
            Post stage
            [Pipeline] cleanWs
            [WS-CLEANUP] Deleting project workspace...
            [WS-CLEANUP] Deferred wipeout is used...
            [WS-CLEANUP] done
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            $ docker stop --time=1 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16
            $ docker rm -f 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16
            [Pipeline] // withDockerContainer
            [Pipeline] }
            [Pipeline] // withDockerRegistry
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            ERROR: script returned exit code -2
            Finished: FAILURE
            
            Show
            smirky Bogomil Vasilev added a comment - I managed to hit this using dir(). This is clearly a bug and causes people to write workarounds. At the second stage, using dir(), it gets stuck for around 6-7 minutes and eventually it fails with "process apparently never started in /opt@tmp/durable-5a20a76a". pipeline { agent { docker { label '********' image '**********' registryUrl '************' registryCredentialsId '*******' args '--user root:root' } } stages { stage( 'dir-testing' ) { stages { stage( 'without dir' ) { steps { sh 'cd /opt && ls -l' } } stage( 'with dir' ) { steps { dir( '/opt' ) { sh 'ls -l' } } } } post { always { cleanWs() } } } } } Started by user ********** Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on ************ in / var /jenkins/workspace/test-cwd-bug [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withDockerRegistry Using the existing docker config file.Removing blacklisted property: auths$ docker login -u ******** -p ******** ********* WARNING! Using --password via the CLI is insecure. Use --password-stdin. Login Succeeded [Pipeline] { [Pipeline] isUnix [Pipeline] sh + docker inspect -f . ********** Error: No such object: *********** [Pipeline] isUnix [Pipeline] sh + docker inspect -f . **************** . [Pipeline] withDockerContainer ************* does not seem to be running inside a container $ docker run -t -d -u 0:0 --user root:root -w / var /jenkins/workspace/test-cwd-bug -v / var /jenkins/workspace/test-cwd-bug:/ var /jenkins/workspace/test-cwd-bug:rw,z -v / var /jenkins/workspace/test-cwd-bug@tmp:/ var /jenkins/workspace/test-cwd-bug@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** ********************* cat $ docker top 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 -eo pid,comm [Pipeline] { [Pipeline] stage [Pipeline] { (dir-testing) [Pipeline] stage [Pipeline] { (without dir) [Pipeline] sh + cd /opt + ls -l total 8 drwxr-xr-x 4 root root 4096 Jul 24 15:29 artifactory-scripts drwxr-xr-x 1 608 500 4096 Jul 24 15:39 cv25_linux_sdk_2.5 [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (with dir) [Pipeline] dir Running in /opt [Pipeline] { [Pipeline] sh process apparently never started in /opt@tmp/durable-5a20a76a (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS= true might make the problem clearer) [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // stage Post stage [Pipeline] cleanWs [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] done [Pipeline] } [Pipeline] // stage [Pipeline] } $ docker stop --time=1 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 $ docker rm -f 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withDockerRegistry [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE
            Hide
            pigstah Matthew Pigram added a comment -

            I'm still running into this same issue, whilst trying to run a pyinstaller docker image.

            The Sh command gives me the following:

            ```

            ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).
            Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.
            [Pipeline]

            { [Pipeline] sh process apparently never started in /var/jenkins_home/workspace/simple-python-pyinstaller-app@tmp/durable-87eb5f90 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }

            $ docker stop --time=1 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
            $ docker rm -f 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
            [Pipeline] // withDockerContainer
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] End of Pipeline
            ERROR: script returned exit code -2
            Finished: FAILURE

            ```

            Running latest jenkins/blueocean image and I'm currently using the following set up for my Jenkinsfile.

             

            ```

            pipeline {
            agent none
            options {| |skipStagesAfterUnstable()| |}
            stages {
            stage('Build') {
            agent {
            docker {| |image 'python:2-alpine'| |}
            }
            steps {| |sh 'python -m py_compile sources/add2vals.py sources/calc.py'| |}
            }
            stage('Test') {
            agent {
            docker {| |image 'qnib/pytest'| |}
            }
            steps {| |sh 'py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py'| |}
            post {
            always {| |junit 'test-reports/results.xml'| |}
            }
            }
            stage('Deliver') {
            agent {
            docker {| |image 'cdrx/pyinstaller-linux:python2'| |args 'docker run -v "/var/jenkins_home/workspace/simple-python-pyinstaller-app/sources:/src/" --name pyinstaller --entrypoint= cdrx/pyinstaller-linux:python2'| |}
            }
            steps {| |sh 'pyinstaller --onefile sources/add2vals.py'| |}
            post {
            success {| |archiveArtifacts 'dist/add2vals'| |}
            }
            }
            }

            }

            ```

             

            Show
            pigstah Matthew Pigram added a comment - I'm still running into this same issue, whilst trying to run a pyinstaller docker image. The Sh command gives me the following: ``` ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`. [Pipeline] { [Pipeline] sh process apparently never started in /var/jenkins_home/workspace/simple-python-pyinstaller-app@tmp/durable-87eb5f90 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] } $ docker stop --time=1 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7 $ docker rm -f 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE ``` Running latest jenkins/blueocean image and I'm currently using the following set up for my Jenkinsfile.   ``` pipeline { agent none options {| |skipStagesAfterUnstable()| |} stages { stage('Build') { agent { docker {| |image 'python:2-alpine'| |} } steps {| |sh 'python -m py_compile sources/add2vals.py sources/calc.py'| |} } stage('Test') { agent { docker {| |image 'qnib/pytest'| |} } steps {| |sh 'py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py'| |} post { always {| |junit 'test-reports/results.xml'| |} } } stage('Deliver') { agent { docker {| |image 'cdrx/pyinstaller-linux:python2'| |args 'docker run -v "/var/jenkins_home/workspace/simple-python-pyinstaller-app/sources:/src/" --name pyinstaller --entrypoint= cdrx/pyinstaller-linux:python2'| |} } steps {| |sh 'pyinstaller --onefile sources/add2vals.py'| |} post { success {| |archiveArtifacts 'dist/add2vals'| |} } } } } ```  

              People

              • Assignee:
                carroll Carroll Chiou
                Reporter:
                jloucky Jakub L
              • Votes:
                11 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: