Uploaded image for project: 'Jenkins Website'
  1. Jenkins Website
  2. WEBSITE-467

Investigate/create/add a 'Configuring a webhook to trigger Jenkins builds' tutorial for the Jenkins User Documentation

    Details

    • Type: Task
    • Status: To Do (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: content
    • Labels:
      None
    • Similar Issues:

      Description

      For a while now, I've been meaning to look into creating a tutorial about how to configure a webhook (in a web-based Git server) to trigger a Pipeline project's builds in Jenkins.

      After a recent discussion with Liam Newman, I soon realized this was going to be a much more complex task than I anticipated. The main complexities in developing such a tutorial arise from the multitude of different network combinations and configurations that could easily lead to the content of this tutorial getting lost in the weeds.

      Therefore, the aim of this tutorial is to demonstrate the processes for setting up a webhook from a web-based Git server to trigger of a Pipeline project's build in Jenkins, without covering the details of different network combinations and configs, which are beyond the scope of this tutorial. This information would need to be stated up front in the tutorial.

      Essentially, the easiest/practical path to take for this tutorial would be to assume everything is done locally on the one machine. That way, we do not have to assume 

      Investigation

      For this tutorial, I looked at the following web-based Git servers (ones which can be installed locally on the reader's machine and not necessarily in Docker - which complicates matters):

      • Bitbucket Server - can download a free 30-day evaluation copy and is installed locally on macOS (which I tested), Linux and Windows. Once the evaluation period expires, however, it's no longer possible to push anything to your Bitbucket Server installation. A US$10 license can be purchased to restore this functionality again.

      • Gitea and Gogs - both free and can be installed locally on macOS (which I tested), Linux and Windows. (Gitea can also run in Docker, which I didn't test).
        • With a default setup, SSH operates on port 22 by default, so I found I had to activate Remote Login from my Mac's System Preferences > Sharing area to allow me to push from a local clone to either Gitea or Gogs. (I subsequently discovered that using HTTP instead of SSH for Git communications made this unnecessary.)
        • With my Gitea setup (not Gogs), I encountered this issue and found that implementing the DB path update documented in this issue fixed the problem.

      It took me some time to realize that running Jenkins in Docker (as we do for all the other existing tutorials so far) was complicating the situation. AFAICT, since I was running the web-based Git server apps (above) directly on my host machine (i.e. localhost), my Jenkins in Docker's localhost wasn't identifying the localhost of my Git server apps running outside of Docker (see attached screenshot demonstrating what happens when I attempted to configure a valid Bitbucket Server repository URL in Jenkins in Docker).

      I even encountered similar problems attempting to get Jenkins in Docker accessing a repository in GitLab CE in Docker (all running off the same Docker host).

      Finally success!

      When I ran Jenkins directly on my own machine (i.e. from a WAR file), I was able to get webhooks configured in Bitbucket Server, Gitea and Gogs to trigger Pipeline project builds. To do this, the following prerequisites are required:

      • Bitbucket Server - the free Post Webhooks for Bitbucket add-on must be installed in Bitbucket, along with the Bitbucket Plugin for Jenkins.
      • Gitea and Gogs - the Gogs Plugin for Jenkins is required.

      Interestingly, for Gitea, there is a Gitea plugin available, which doesn't appear to work. This Stackoverflow post has more information about configuring webhooks for both Gitea and Gogs.

      Proposed tutorial outline

      This may change after I have a discussion with Damien Duportal later today (8 Feb 2018 my time), since the following is based on the assumption that all required tools/software involved are installed locally.

      Prerequisites

      • Butbucket Server, Gitea or Gogs installed locally - link off to relevant page/s, although can provide recommended running instructions (for the purposes of this tutorial).
      • Jenkins WAR file (local installation) - can probably link off to the WAR file procedure in the Installing Jenkins docs.
      • ( Only for Bitbucket Server ) An SSH public/private key pair (e.g. for your computer's main user account) is necessary for Bitbucket Server (when all is installed locally) - just setting the public key component in Bitbucket Server is all that seems to be required. This doesn't need to be done with Gitea and Gogs.

      Process

      1. Fork relevant sample repository from GitHub and clone it locally.
      2. Create an empty repository in your Bitbucket Server or Gitea/Gogs instance.
      3. Add your locally cloned repo to your local Bitbucket Server or Gitea/Gogs instance - i.e. git remote add <git-server-remote-name> <git-server-url> followed by git push <git-server-remote-name>
        Note: For simplicity, use the http://... protocol for <git-server-url>.
      4. In Jenkins:
        1. Install the relevant plugin to allow Jenkins to recognize your web-based Git server's webhook request:
          • For Bitbucket, this is the Bitbucket Plugin.
          • For Gitea and Gogs, this is the Gogs Plugin.
        2. Create your Pipeline project and for simplicity, specify the URLs to your web-based Git server's repo using the http://... protocol. (There is no need to configure any credentials here - I suspect, at least in my case, because Jenkins would be running using your computer's main user account, which is what the web-based Git server would be using too.)
          • For Bitbucket, you need to ensure the Build when a change is pushed to BitBucket option is selected.
      5. In your web-based Git server, create your webhook:
        • For Bitbucket:
          1. Install the Post Webhooks for Bitbucket add-on.
          2. Go to your repo's Post Webhooks (not Webhooks) feature and create the webhook (i.e. using http://<jenkins-server>/bitbucket-hook/ where <jenkins-server> might be localhost:9090).
        • For Gitea or Gogs, go to your repo's Webhooks feature and create the webhook (i.e. using http://<jenkins-server>/gogs-webhook/?job=<your-jenkins-job-name>).
      6. Create your initial Jenkinsfile and commit it to your local clone, then git push <git-server-remote-name> (i.e. to your local web-based Git server).
      7. Go back to Jenkins and see your Pipeline project being built without you having triggered it manually - MAGIC!
      8. Iterate over the last two steps with Test and Deploy stages.

      Caveat: Unlike the Jenkins in Docker method of installing Jenkins locally (using the jenkinsci/blueocean image), a disadvantage of installing Jenkins from a WAR file is that Blue Ocean is not installed by default.

        Attachments

          Activity

          Hide
          ggaskell Giles Gaskell added a comment - - edited

          Hey Damien Duportal,

          Thanks so much for all your help here :-D

          FYI - On my way home this evening, I was able to get this tutorial working with both Jenkins and Gitea running in Docker! I'm so chuffed!

          The advantage of using this setup is that we'll be able to showcase automated builds in Blue Ocean (i.e. without the need to install the Blue Ocean plugins on the WAR installation of Jenkins).

          My next step was to create the docker-compose.yml file but you beat me to it again Those additional points you mentioned above were very helpful though and it probably would've taken me a while to find out about them myself.

          I'll give this a try on Monday and'll let you know how I go.

          I think docker-compose is going to be great way to shorten all the tutorials in the jenkins.io docs. Doing this probably means adding a section in the main Installing Jenkins documentation about Docker Compose. It'll be nice to have a section of content that covers both the docker run ... and its equivalent in a docker-compose.yml file.

          Cheers!
          Giles.

          Show
          ggaskell Giles Gaskell added a comment - - edited Hey Damien Duportal , Thanks so much for all your help here :-D FYI - On my way home this evening, I was able to get this tutorial working with both Jenkins and Gitea running in Docker! I'm so chuffed! The advantage of using this setup is that we'll be able to showcase automated builds in Blue Ocean (i.e. without the need to install the Blue Ocean plugins on the WAR installation of Jenkins). My next step was to create the docker-compose.yml file but you beat me to it again Those additional points you mentioned above were very helpful though and it probably would've taken me a while to find out about them myself. I'll give this a try on Monday and'll let you know how I go. I think docker-compose is going to be great way to shorten all the tutorials in the jenkins.io docs. Doing this probably means adding a section in the main Installing Jenkins documentation about Docker Compose. It'll be nice to have a section of content that covers both the docker run ... and its equivalent in a docker-compose.yml file. Cheers! Giles.
          Hide
          dduportal Damien Duportal added a comment -

          Here is the docker-compose version: https://gist.github.com/dduportal/e280af7e63a28a1ee423c752783b9a06

          Create a file named docker-compose.yml, and use the command docker-compose up -d in the same folder containing the YAML file.
          It will bring a stack and it will take care of everything for you, and you should be able to run tutorials.

          Note the following changes from the original docker run commands:

          • Removed the jenkins-data:/var/jenkins_home. Docker will take care fo this, by creating an anonymous data volume. Effect will be the same even with the Docker on docker "stuff". Less code to take care of.
          • Removed the volume sharing of "$HOME": we will use Gitea a git server, so no need to share any folder with Jenkins.
          • Changed the "222" port of gitea to "2222", because ports < 1024 are generally not used on local mode (a lot of reasons but mainly convention here).
          Show
          dduportal Damien Duportal added a comment - Here is the docker-compose version: https://gist.github.com/dduportal/e280af7e63a28a1ee423c752783b9a06 Create a file named docker-compose.yml , and use the command docker-compose up -d in the same folder containing the YAML file. It will bring a stack and it will take care of everything for you, and you should be able to run tutorials. Note the following changes from the original docker run commands: Removed the jenkins-data:/var/jenkins_home . Docker will take care fo this, by creating an anonymous data volume. Effect will be the same even with the Docker on docker "stuff". Less code to take care of. Removed the volume sharing of "$HOME": we will use Gitea a git server, so no need to share any folder with Jenkins. Changed the "222" port of gitea to "2222", because ports < 1024 are generally not used on local mode (a lot of reasons but mainly convention here).
          Hide
          dduportal Damien Duportal added a comment - - edited

          Hello Giles Gaskell, you are almost there!

          The goal of using the "--network" flag on docker run is to have a network with a DNS server managed by Docker. So the 2 containers will communicate trough this network by using their names as "hostnames", instead of localhost.

          By default, container names are dynamically generated and are changing. So the goal is to have a fixed named using the flag "--name" for docker run commands:

          • Create the network
          • Run the Jenkins container:
            docker run \
              --rm \
              --name jenkins \
              -u root \
              -p 8080:8080 \
              -v jenkins-data:/var/jenkins_home \
              -v /var/run/docker.sock:/var/run/docker.sock \
              -v "$HOME":/home \
              --network jenkins-tutorials \
              jenkinsci/blueocean
            
          • Run the Git Server container:
            docker run \
              --rm \
              --name gitserver \
              -p 3000:3000 \
              -p 222:22 \
              -v gitea:/data \
              --network jenkins-tutorials \
              gitea/gitea:latest
            

          With those example, Jenkins will be able to fetch the git repositories using http://gitserver:3000/(...) and the git server will be able to send webhooks to jenkins using http://jenkins:8080/(...).

          Show
          dduportal Damien Duportal added a comment - - edited Hello Giles Gaskell , you are almost there! The goal of using the "--network" flag on docker run is to have a network with a DNS server managed by Docker. So the 2 containers will communicate trough this network by using their names as "hostnames", instead of localhost . By default, container names are dynamically generated and are changing. So the goal is to have a fixed named using the flag "--name" for docker run commands: Create the network Run the Jenkins container: docker run \ --rm \ --name jenkins \ -u root \ -p 8080:8080 \ -v jenkins-data:/ var /jenkins_home \ -v / var /run/docker.sock:/ var /run/docker.sock \ -v "$HOME" :/home \ --network jenkins-tutorials \ jenkinsci/blueocean Run the Git Server container: docker run \ --rm \ --name gitserver \ -p 3000:3000 \ -p 222:22 \ -v gitea:/data \ --network jenkins-tutorials \ gitea/gitea:latest With those example, Jenkins will be able to fetch the git repositories using http://gitserver:3000/( ...) and the git server will be able to send webhooks to jenkins using http://jenkins:8080/( ...).
          Hide
          ggaskell Giles Gaskell added a comment - - edited

          Hi again Damien Duportal,

          Thanks again for the chat yesterday - great to catch up after such a long time and very enlightening stuff about Docker!

          Following on from our chat last night, I had a go at running the following 3 docker commands:

          1) CREATE NETWORK:

          docker network create jenkins-tutorials

          2) RUN JENKINS IN DOCKER:

          docker run \
            --rm \
            -u root \
            -p 8080:8080 \
            -v jenkins-data:/var/jenkins_home \
            -v /var/run/docker.sock:/var/run/docker.sock \
            -v "$HOME":/home \
            --network jenkins-tutorials \
            jenkinsci/blueocean
          

          3) RUN GITEA IN DOCKER:

          docker run \
            --rm \
            -p 3000:3000 \
            -p 222:22 \
            -v gitea:/data \
            --network jenkins-tutorials \
            gitea/gitea:latest
          

          However, I wasn't able to get my Jenkins recognising Gitea through localhost within Docker.

          I still received a similar error to what you see in the attached screenshot above.

          I even tried verifying that my both my Jenkins and Gitea containers were indeed a part of my jenkins-tutorials network using a command like:

          docker network connect jenkins-tutorials adoring_varahamihira

          (where adoring_varahamihira was the randomly assigned name of my Jenkins container that Docker gave me) and I got back:

          Error response from daemon: endpoint with name adoring_varahamihira already exists in network jenkins-tutorials

          So I'm thinking that I must be missing something here.

          Show
          ggaskell Giles Gaskell added a comment - - edited Hi again Damien Duportal , Thanks again for the chat yesterday - great to catch up after such a long time and very enlightening stuff about Docker! Following on from our chat last night, I had a go at running the following 3 docker commands: 1) CREATE NETWORK: docker network create jenkins-tutorials 2) RUN JENKINS IN DOCKER: docker run \ --rm \ -u root \ -p 8080:8080 \ -v jenkins-data:/ var /jenkins_home \ -v / var /run/docker.sock:/ var /run/docker.sock \ -v "$HOME" :/home \ --network jenkins-tutorials \ jenkinsci/blueocean 3) RUN GITEA IN DOCKER: docker run \ --rm \ -p 3000:3000 \ -p 222:22 \ -v gitea:/data \ --network jenkins-tutorials \ gitea/gitea:latest However, I wasn't able to get my Jenkins recognising Gitea through localhost within Docker. I still received a similar error to what you see in the attached screenshot above. I even tried verifying that my both my Jenkins and Gitea containers were indeed a part of my jenkins-tutorials network using a command like: docker network connect jenkins-tutorials adoring_varahamihira (where adoring_varahamihira was the randomly assigned name of my Jenkins container that Docker gave me) and I got back: Error response from daemon: endpoint with name adoring_varahamihira already exists in network jenkins-tutorials So I'm thinking that I must be missing something here.
          Hide
          ggaskell Giles Gaskell added a comment - - edited

          Hey Damien Duportal,

          I've added the Proposed tutorial outline to the description section above, which shows the prerequisites and process I'm proposing. As we discussed yesterday, this is all based on the assumption that Jenkins and your web-based Git server are all installed locally on your own computer (to minimize complication).

          Having said that, I'm willing to change this setup and process based on our discussion in a few hours. I've provided this information to demonstrate the scope of what's involved in this tutorial.

          Things to discuss:

          • Is it possible to run through this tutorial on web-hosted/cloud-based infrastructure - e.g. Katacoda for Jenkins and perhaps GitHub and/or Bitbucket Cloud as the web-based Git server.
            • There is still going to be some setting up required - i.e. configuring SSH keys. (Not sure if this would need to be part of the tutorial.)
          • As an alternative, if instead, we were to run Jenkins in Docker as well as an appropriate web-based Git server on Docker (i.e. Gitea or now Gogs and Bitbucket Server as I've just discovered), how do we get these 2 applications (i.e. Jenkins and the Git server) to communicate to one another in Docker?
            • I'm asking this more out of my Docker ignorance.
          Show
          ggaskell Giles Gaskell added a comment - - edited Hey Damien Duportal , I've added the Proposed tutorial outline to the description section above, which shows the prerequisites and process I'm proposing. As we discussed yesterday, this is all based on the assumption that Jenkins and your web-based Git server are all installed locally on your own computer (to minimize complication). Having said that, I'm willing to change this setup and process based on our discussion in a few hours. I've provided this information to demonstrate the scope of what's involved in this tutorial. Things to discuss: Is it possible to run through this tutorial on web-hosted/cloud-based infrastructure - e.g. Katacoda for Jenkins and perhaps GitHub and/or Bitbucket Cloud as the web-based Git server. There is still going to be some setting up required - i.e. configuring SSH keys. (Not sure if this would need to be part of the tutorial.) As an alternative, if instead, we were to run Jenkins in Docker as well as an appropriate web-based Git server on Docker (i.e. Gitea or now Gogs and Bitbucket Server as I've just discovered), how do we get these 2 applications (i.e. Jenkins and the Git server) to communicate to one another in Docker? I'm asking this more out of my Docker ignorance.

            People

            • Assignee:
              ggaskell Giles Gaskell
              Reporter:
              ggaskell Giles Gaskell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: