Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48960

Org folders repository folder never runs scans on repository

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: branch-api-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.101, Folders Plugin 6.3, GitHub Branch Source Plugin 2.3.2
    • Similar Issues:

      Description

      Recently updated Jenkins and all available plugins, and the Repository folder within our Organization folder stopped scanning completely. 

       

      I went so far as to delete our entire organization structure and re-create it. The organization does scan correctly on changes. The only time the repository level folder scans automatically is upon its initial creation from the creation of the organization. 

       

      Also makes little sense that the repository folder has a non-configurable interval of 1 day, while the organization folder was setup with an interval of 5 minutes. This is how the repo's trigger is defined in its config.xml:

      <spec>H H * * *</spec>
      <interval>86400000</interval>
      

      I manually changed that to match the Org folders config trigger, e.g. 

      <spec>H/12 * * * *</spec>
      <interval>300000</interval>

      and then gave Jenkins a restart. After ~5 minutes, sure enough, it's now scanning correctly. 

       

      So why on earth is the interval timer being set to such a long duration, and so dramatically different than the org's interval timer? 

        Attachments

          Issue Links

            Activity

            Hide
            jimklimov Jim Klimov added a comment -

            Then there's also a problem of the corporate networks. Our build farm may poll github, but won't be permitted to get webhooks or any other messages initiated from the internet, directly or through tunnels.

            Hmm... a case for email-hooks... rss-hooks... out of scope here though.

            Show
            jimklimov Jim Klimov added a comment - Then there's also a problem of the corporate networks. Our build farm may poll github, but won't be permitted to get webhooks or any other messages initiated from the internet, directly or through tunnels. Hmm... a case for email-hooks... rss-hooks... out of scope here though.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Jim Klimov so the correct solution to that would be something like the GitHub SNS plugin (if it was updated to replicate events correctly... last time I checked it wasn't )

            Andy Airey The point is that those are not issues for the branch-api plugin itself, they are issues with the SCM sources... or the SCMs themselves... or your environment. Polling is exceedingly expensive in terms of overhead to check and also in terms of delay of notification. Especially when you have an entire organization of repositories to scan. For something like GitHub or Bitbucket Cloud, because of API limits, polling may not even be in any way practicable.

            If you are really stuck, then there are other paths you can take, for example the Apache Software Foundation uses its own custom org folder and event forwarding... https://github.com/stephenc/asf-gitpubsub-jenkins-plugin (also see the video series where I show how this was developed https://www.youtube.com/watch?v=W8BRtJmq_2Y&list=PLvBBnHmZuNQLXcL6CBZPKbX9OstMNIhTV ... note that the video is real time development of the plugin. The only code I wrote was during the video recordings)

            Show
            stephenconnolly Stephen Connolly added a comment - Jim Klimov so the correct solution to that would be something like the GitHub SNS plugin (if it was updated to replicate events correctly... last time I checked it wasn't ) Andy Airey The point is that those are not issues for the branch-api plugin itself, they are issues with the SCM sources... or the SCMs themselves... or your environment. Polling is exceedingly expensive in terms of overhead to check and also in terms of delay of notification. Especially when you have an entire organization of repositories to scan. For something like GitHub or Bitbucket Cloud, because of API limits, polling may not even be in any way practicable. If you are really stuck, then there are other paths you can take, for example the Apache Software Foundation uses its own custom org folder and event forwarding... https://github.com/stephenc/asf-gitpubsub-jenkins-plugin (also see the video series where I show how this was developed https://www.youtube.com/watch?v=W8BRtJmq_2Y&list=PLvBBnHmZuNQLXcL6CBZPKbX9OstMNIhTV ... note that the video is real time development of the plugin. The only code I wrote was during the video recordings)
            Hide
            aairey Andy Airey added a comment -

            Sure Stephen Connolly, it's just that you were listing a limited set of use-cases. Wanted to share mine as well since you were trying to create the bigger picture here (which is appreciated).

            I agree with all your points, I wish already for more than a year now that I could use webhooks, but I simply cannot because it does not exist or is not worked on. (These are good reasons to change CI system btw ...).

            I cannot install my own custom plugins on Bitbucket server because I am not in control of that, corporate is and they don't allow us to. They only support the native webhooks.
            So yeah, you can argue that this is a problem within my corporate atmosphere - but I am fairly certain there are more businesses out there using Jenkins that are in a similar situation (be it with firewalls or what have you).

            I understand that you cannot support each and every environment but it comes down to "polling is dead"  - we all wish it were, but it isn't.

            Show
            aairey Andy Airey added a comment - Sure Stephen Connolly , it's just that you were listing a limited set of use-cases. Wanted to share mine as well since you were trying to create the bigger picture here (which is appreciated). I agree with all your points, I wish already for more than a year now that I could use webhooks, but I simply cannot because it does not exist or is not worked on. (These are good reasons to change CI system btw ...). I cannot install my own custom plugins on Bitbucket server because I am not in control of that, corporate is and they don't allow us to. They only support the native webhooks. So yeah, you can argue that this is a problem within my corporate atmosphere - but I am fairly certain there are more businesses out there using Jenkins that are in a similar situation (be it with firewalls or what have you). I understand that you cannot support each and every environment but it comes down to "polling is dead"  - we all wish it were, but it isn't.
            Hide
            hardwickj James Hardwick added a comment -

            Yea, we're in the same boat of keeping our Jenkins instance firewall'd off from the outside world, hence the polling. It's not a big deal for us since the Organization being polled has a fairly limited set of repos. 

            Every company I've ever been with has used Jenkins this way, and usually has had very legitimate reasons for doing so. I think it'd be a mistake to have a default worldview of "webhooks everywhere". Just my $.02. 

            Show
            hardwickj James Hardwick added a comment - Yea, we're in the same boat of keeping our Jenkins instance firewall'd off from the outside world, hence the polling. It's not a big deal for us since the Organization being polled has a fairly limited set of repos.  Every company I've ever been with has used Jenkins this way, and usually has had very legitimate reasons for doing so. I think it'd be a mistake to have a default worldview of "webhooks everywhere". Just my $.02. 
            Hide
            bryceman Bryce Schober added a comment -

            Count me as +1 org with a firewalled Jenkins instance that can't currently accept webhooks.

            On that point, does the Jenkins community have a guide for best-security-practices forwarding proxy setup for specific webhooks that orgs could follow?

            Show
            bryceman Bryce Schober added a comment - Count me as +1 org with a firewalled Jenkins instance that can't currently accept webhooks. On that point, does the Jenkins community have a guide for best-security-practices forwarding proxy setup for specific webhooks that orgs could follow?

              People

              • Assignee:
                Unassigned
                Reporter:
                hardwickj James Hardwick
              • Votes:
                25 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated: