Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48502

image.inside stopped working on ECS agents

    Details

    • Similar Issues:

      Description

      Hi,

      We are running Jenkins with ECS cloud, meaning the agents are docker based agents.

      Some of our pipelines are using image.inside to run arbitrary build tools which are not part of the slave images.

      Before upgrading to amazon-ecs-agent-1.16.0 we got the following message in the log files:

       

      ecs-cloud-33394f9575f0e2 seems to be running inside container 01b572fc3c671c11087fcb8166d686792d2b725e570af941b37ff3f28986762f 
      

      After the upgrade, we get:

      ecs-cloud-dc46d9e888c4 does not seem to be running inside a container

      Looking at The DockerClient code I see that the way the plugin understands it is running within docker container, is by matching docker cgroups pattern with the default hierarchy name docker.

      The problem is that AWS ECS changed their default hierarchy to be ecs

      root@20f3571647fc:/usr/local/tomcat# cat /proc/self/cgroup
       9:perf_event:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
       8:memory:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
       7:hugetlb:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
      ...

      Because of that, image.inside step stopped working and we get the following error each build.

      sh: can't create /var/jenkins_home/workspace/...@tmp/durable-9cddf800/jenkins-log.txt: nonexistent directory

       

      I believe a quick fix would be to change the regex to match ecs as well, or add a cgroups hierarchy configuration parameter to the plugin.

      Best regards,

      Ohad

        Attachments

          Issue Links

            Activity

            Hide
            jonatan Jonatan Landsberg added a comment -

            For what it's worth I want to confirm this bug. My team has the exact same issue and resolved it by downgrading the ECS agent back to 1.15.2.

            Would really want a better solution as we want to use the latest version of the agent, and eventually will have to upgrade it.

            Show
            jonatan Jonatan Landsberg added a comment - For what it's worth I want to confirm this bug. My team has the exact same issue and resolved it by downgrading the ECS agent back to 1.15.2. Would really want a better solution as we want to use the latest version of the agent, and eventually will have to upgrade it.
            Hide
            jakobkylberg Jakob Kylberg added a comment -

            I can confirm this as well. Our team ran into this issue and had to downgrade the ECS agent to 1.15.2 in order to get our Jenkins setup to work again. I hope this issue can get prioritized as it is a showstopper that stops us from updating our EC2 instances and ECS agents.

            Show
            jakobkylberg Jakob Kylberg added a comment - I can confirm this as well. Our team ran into this issue and had to downgrade the ECS agent to 1.15.2 in order to get our Jenkins setup to work again. I hope this issue can get prioritized as it is a showstopper that stops us from updating our EC2 instances and ECS agents.
            Hide
            odavid Ohad David added a comment - - edited

            While https://issues.jenkins-ci.org/browse/JENKINS-47947 is related to kubernetes and this issue is related to ecs, they share the same pattern.

            Show
            odavid Ohad David added a comment - - edited While https://issues.jenkins-ci.org/browse/JENKINS-47947  is related to kubernetes and this issue is related to ecs, they share the same pattern.
            Hide
            odavid Ohad David added a comment -
            Show
            odavid Ohad David added a comment - I've created a PR: https://github.com/jenkinsci/docker-workflow-plugin/pull/124  

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                odavid Ohad David
              • Votes:
                2 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: