Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59907

sh steps stuck indefinitely on uncommon architectures (e.g. s390x)

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: durable-task-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.201 (yum installed, master node only)
      durable-task plugin v1.31

      os.arch: s390x
      os.name: Linux (RedHat)
      os.version: 3.10.0-327.el7.s390x
    • Similar Issues:
    • Released As:
      1.33

      Description

      After upgrading to v1.31, the first sh step in a pipeline gets stuck. After few minutes Console Output shows:

      [Pipeline] sh (Get email of the author of last commit)
      process apparently never started in /data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6
      (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      Cannot contact : java.io.FileNotFoundException: File '/data/jenkins/workspace/TG2_PTG2_-_pipeline_build_master@tmp/durable-be2cf2a6/output.txt' does not exist

       

      Eventually, I discovered that a new binary was added in the latest version of this plugin. The script compile-binaries.sh in GitHub suggests that the binary is only built for Linux and MacOS.

       

      Sure enough, when I try to execute the binary myself on an architecture other than amd64, I get:

      -bash: /data/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64: cannot execute binary file

       

      Are other architectures or operating systems (Windows) not supported anymore?

        Attachments

          Issue Links

            Activity

            Hide
            pigstah Matthew Pigram added a comment -

            I'm still running into this same issue, whilst trying to run a pyinstaller docker image.

            The Sh command gives me the following:

            ```

            ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).
            Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.
            [Pipeline]

            { [Pipeline] sh process apparently never started in /var/jenkins_home/workspace/simple-python-pyinstaller-app@tmp/durable-87eb5f90 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }

            $ docker stop --time=1 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
            $ docker rm -f 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
            [Pipeline] // withDockerContainer
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] End of Pipeline
            ERROR: script returned exit code -2
            Finished: FAILURE

            ```

            Running latest jenkins/blueocean image and I'm currently using the following set up for my Jenkinsfile.

             

            ```

            pipeline {
            agent none
            options {| |skipStagesAfterUnstable()| |}
            stages {
            stage('Build') {
            agent {
            docker {| |image 'python:2-alpine'| |}
            }
            steps {| |sh 'python -m py_compile sources/add2vals.py sources/calc.py'| |}
            }
            stage('Test') {
            agent {
            docker {| |image 'qnib/pytest'| |}
            }
            steps {| |sh 'py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py'| |}
            post {
            always {| |junit 'test-reports/results.xml'| |}
            }
            }
            stage('Deliver') {
            agent {
            docker {| |image 'cdrx/pyinstaller-linux:python2'| |args 'docker run -v "/var/jenkins_home/workspace/simple-python-pyinstaller-app/sources:/src/" --name pyinstaller --entrypoint= cdrx/pyinstaller-linux:python2'| |}
            }
            steps {| |sh 'pyinstaller --onefile sources/add2vals.py'| |}
            post {
            success {| |archiveArtifacts 'dist/add2vals'| |}
            }
            }
            }

            }

            ```

             

            Show
            pigstah Matthew Pigram added a comment - I'm still running into this same issue, whilst trying to run a pyinstaller docker image. The Sh command gives me the following: ``` ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`. [Pipeline] { [Pipeline] sh process apparently never started in /var/jenkins_home/workspace/simple-python-pyinstaller-app@tmp/durable-87eb5f90 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] } $ docker stop --time=1 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7 $ docker rm -f 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE ``` Running latest jenkins/blueocean image and I'm currently using the following set up for my Jenkinsfile.   ``` pipeline { agent none options {| |skipStagesAfterUnstable()| |} stages { stage('Build') { agent { docker {| |image 'python:2-alpine'| |} } steps {| |sh 'python -m py_compile sources/add2vals.py sources/calc.py'| |} } stage('Test') { agent { docker {| |image 'qnib/pytest'| |} } steps {| |sh 'py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py'| |} post { always {| |junit 'test-reports/results.xml'| |} } } stage('Deliver') { agent { docker {| |image 'cdrx/pyinstaller-linux:python2'| |args 'docker run -v "/var/jenkins_home/workspace/simple-python-pyinstaller-app/sources:/src/" --name pyinstaller --entrypoint= cdrx/pyinstaller-linux:python2'| |} } steps {| |sh 'pyinstaller --onefile sources/add2vals.py'| |} post { success {| |archiveArtifacts 'dist/add2vals'| |} } } } } ```  
            Hide
            don_code Don L added a comment -

            Sure, I've cross-posted to that ticket.

            Show
            don_code Don L added a comment - Sure, I've cross-posted to that ticket.
            Hide
            carroll Carroll Chiou added a comment -

            Don L can we move this over to JENKINS-59903? That would be the relevant ticket. Could you also confirm that you are running v1.33? and not v1.31-32

            Show
            carroll Carroll Chiou added a comment - Don L can we move this over to JENKINS-59903 ? That would be the relevant ticket. Could you also confirm that you are running v1.33? and not v1.31-32
            Hide
            don_code Don L added a comment -

            I've found this also reproduces when using build agents in Kubernetes. The problem here is that Kubernetes launches two containers into a pod with a shared mount: a JNLP slave container, which Jenkins does have permission to write the cache directory in, and a build container (in my case kubectl, but could be any container without a Jenkins user) where it does not necessarily have the same permission, in which code actually runs. The plugin runs its test inside the JNLP container, enables the wrapper, and then exhibits the same hanging behavior when commands are run in the kubectl container.

            In the JNLP container:

            bash-4.4$ cd /home/jenkins/agent/caches
            bash-4.4$ ls -l
            total 0
            drwxr-xr-x    2 jenkins  jenkins          6 Mar  6 15:47 durable-task 

            In the kubectl container:

            I have no name!@<REDACTED>:/home/jenkins/agent/caches$ ls -l
            total 0
            drwxr-xr-x 2 1000 1000 6 Mar  6 15:47 durable-task
            
            I have no name!@<REDACTED>:/home/jenkins/agent/caches$ id
            uid=1001 gid=0(root) groups=0(root)
            Show
            don_code Don L added a comment - I've found this also reproduces when using build agents in Kubernetes. The problem here is that Kubernetes launches two containers into a pod with a shared mount: a JNLP slave container, which Jenkins does have permission to write the cache directory in, and a build container (in my case kubectl , but could be any container without a Jenkins user) where it does not necessarily have the same permission, in which code actually runs. The plugin runs its test inside the JNLP container, enables the wrapper, and then exhibits the same hanging behavior when commands are run in the kubectl container. In the JNLP container: bash-4.4$ cd /home/jenkins/agent/caches bash-4.4$ ls -l total 0 drwxr-xr-x 2 jenkins jenkins 6 Mar 6 15:47 durable-task In the kubectl container: I have no name!@<REDACTED>:/home/jenkins/agent/caches$ ls -l total 0 drwxr-xr-x 2 1000 1000 6 Mar 6 15:47 durable-task I have no name!@<REDACTED>:/home/jenkins/agent/caches$ id uid=1001 gid=0(root) groups=0(root)
            Hide
            carroll Carroll Chiou added a comment -

            Rahul Raj it appears you are using x86, and not an "uncommon architecture" like this ticket is describing. I would probably advise setting LAUNCH_DIAGNOSTICS=true as suggested in the output log and that can tell us better. The default behavior for this plugin should be using the original script wrappers. If you can't ascertain what is going there, I would probably post this to jenkinsci-users mailing list while we're still investigating.

            Show
            carroll Carroll Chiou added a comment - Rahul Raj it appears you are using x86, and not an "uncommon architecture" like this ticket is describing. I would probably advise setting LAUNCH_DIAGNOSTICS=true as suggested in the output log and that can tell us better. The default behavior for this plugin should be using the original script wrappers. If you can't ascertain what is going there, I would probably post this to jenkinsci-users mailing list while we're still investigating.

              People

              • Assignee:
                carroll Carroll Chiou
                Reporter:
                jloucky Jakub L
              • Votes:
                10 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated: