Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47077

"Periodically if not otherwise run" interval does not trigger index scan

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      6.7

      Description

      I have set up a multibranch pipeline with Bitbucket Cloud repository.
      I am running jenkins locally, so I am not using webhooks.

      I've set "Periodically if not otherwise run" interval to 2 minutes, but repository is not being scanned for changes. Scanning can only be triggered manualy by "Scan Multibranch pipeline now".

      I've tried to set this up from blue ocean, then from old UI and no luck in both.
      Also using regular Git branch source instead of Bitbucket doesn't work.

      So I guess the problem is with multibranch pipeline itself.

        Attachments

          Issue Links

            Activity

            Hide
            stephenconnolly Stephen Connolly added a comment -

            Does it work for 5 minutes?

            (Aside: Keep in mind that such aggressive scanning will get you rate limited by Bitbucket Cloud, so you probably want to set up webhook notification)

            If it works for 5 minutes but not for 2 then that could well be a bug in the cloudbees-folders plugin.

            Show
            stephenconnolly Stephen Connolly added a comment - Does it work for 5 minutes? (Aside: Keep in mind that such aggressive scanning will get you rate limited by Bitbucket Cloud, so you probably want to set up webhook notification) If it works for 5 minutes but not for 2 then that could well be a bug in the cloudbees-folders plugin.
            Hide
            wojciechrak Wojciech Rak added a comment -

            I've tried also with 5 and 10 minutes, neither works for me.
            Scan Multibranch Pipeline Log is not being updated, shows only information from last scan.

            Can you tell me more about rate limit? Isn't this just fetching refs?

            If this scanning is so aggressive, is there any other setting to just fetch known branches? All I really want to achieve, is automated build of master branch after commit. It should be possible to do without webhooks, I guess.

            Show
            wojciechrak Wojciech Rak added a comment - I've tried also with 5 and 10 minutes, neither works for me. Scan Multibranch Pipeline Log is not being updated, shows only information from last scan. Can you tell me more about rate limit? Isn't this just fetching refs? If this scanning is so aggressive, is there any other setting to just fetch known branches? All I really want to achieve, is automated build of master branch after commit. It should be possible to do without webhooks, I guess.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            So I just tried on my own server,

            And as you can see the periodically runs, the initial scan was 9:32 and then when I check again it runs at 9:34... and then again at 9:36

            now there is a bug, if I recall correctly, where the 1 minute interval gets checked against 60 seconds and hence will almost always be 60 seconds since last check and so only every other minute will it run... but if I understand correctly that should only apply to the 1 minute interval and not affect the larger intervals (it's due to the cron tasks running once per minute)

            But the tasks are running for me.

            I'd check your master to see if something is blocking either the Jenkins cron thread or one of the jenkins.util.Timer threads.

            Additionally, there is a limit of 5 concurrent indexing/scanning tasks: https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/computed/ThrottleComputationQueueTaskDispatcher.java#L49 so if you have other jobs that are say indexing GitHub and they have hit the GitHub rate limit, your indexing may just be blocked waiting for those to complete

            Show
            stephenconnolly Stephen Connolly added a comment - So I just tried on my own server, And as you can see the periodically runs, the initial scan was 9:32 and then when I check again it runs at 9:34... and then again at 9:36 now there is a bug, if I recall correctly, where the 1 minute interval gets checked against 60 seconds and hence will almost always be 60 seconds since last check and so only every other minute will it run... but if I understand correctly that should only apply to the 1 minute interval and not affect the larger intervals (it's due to the cron tasks running once per minute) But the tasks are running for me. I'd check your master to see if something is blocking either the Jenkins cron thread or one of the jenkins.util.Timer threads. Additionally, there is a limit of 5 concurrent indexing/scanning tasks: https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/computed/ThrottleComputationQueueTaskDispatcher.java#L49 so if you have other jobs that are say indexing GitHub and they have hit the GitHub rate limit, your indexing may just be blocked waiting for those to complete
            Hide
            wojciechrak Wojciech Rak added a comment - - edited

            I did manage to get the schedule somehow working on Jenkins 2.60.3 on docker image.

            However it's running every ~12 minutes, despite being set to 5 minute interval.

             

             

            I also have 2.79 on my mac and as docker image, and the schedule isn't working on either of them.

             

            UPDATE

            With 2 minute interval, indexing is done every 3 minutes.

            Show
            wojciechrak Wojciech Rak added a comment - - edited I did manage to get the schedule somehow working on Jenkins 2.60.3 on docker image. However it's running every ~12 minutes, despite being set to 5 minute interval.     I also have 2.79 on my mac and as docker image, and the schedule isn't working on either of them.   UPDATE With 2 minute interval, indexing is done every 3 minutes.
            Hide
            jglick Jesse Glick added a comment -

            Maybe related to https://github.com/jenkinsci/cloudbees-folder-plugin/pull/108? Can you see if that patch works?

            Show
            jglick Jesse Glick added a comment - Maybe related to https://github.com/jenkinsci/cloudbees-folder-plugin/pull/108?  Can you see if that patch works?
            Hide
            hardwickj James Hardwick added a comment -

            After updating all of my plugins early last week, I'm now also seeing what appears to be at least a variant of this issue. 

             

            We have a GitHub Organization folder setup, which correctly scans our entire org. Only one of our repositories at the moment contains Jenkins files, and it used to correctly scan all branches within that repository and pickup them up. After updating last week, the branches are no longer scanned. When I check the organization scan log, I see this:

             

             

            Examining
            our_org_name/our_repo_name
            Checking branches... Getting remote branches... Checking branch
            bug/branch_name_here
            ‘Jenkinsfile’ found Met criteria 1 branches were processed (query completed) 1 branches were processed Finished examining our_org_name/our_repo_name
            

             

             

            ...but we have probably 15 branches with a Jenkinsfile in them. It just so happens to be that the only branch it scanned is alphabetically the first in our repo. Going into the repository and looking at it's scan log, the scan does not happen at all. I don't recall in this setup if it is the organization or repository level scan that normally would have kicked off our build. I tested making a commit to the one branch that the org scan does detect, and that did not initiate a repository level scan. 

             

            Jesse Glick if you have any suggestions of things to test/try/log or what not I'm all ears. This is having a big impact on our teams workflow.  I'll keep digging, but I suspect this is the same issue as described above as the repository level of the GitHub Org folder appears to be the same thing as a multibranch pipeline project. 

             

            Also worth nothing, when viewing the config at the repository level it has.a scan interval set at 1 day. Since this config is created by the GitHub Org folder, we can't configure it. But despite that, it's not actually running once a day. I'm looking at it right now and the last scan was when we manually kicked it off 2 days ago. 

            Show
            hardwickj James Hardwick added a comment - After updating all of my plugins early last week, I'm now also seeing what appears to be at least a variant of this issue.    We have a GitHub Organization folder setup, which correctly scans our entire org. Only one of our repositories at the moment contains Jenkins files, and it used to correctly scan all branches within that repository and pickup them up. After updating last week, the branches are no longer scanned. When I check the organization scan log, I see this:     Examining our_org_name/our_repo_name Checking branches... Getting remote branches... Checking branch bug/branch_name_here ‘Jenkinsfile’ found Met criteria 1 branches were processed (query completed) 1 branches were processed Finished examining our_org_name/our_repo_name     ...but we have probably 15 branches with a Jenkinsfile in them. It just so happens to be that the only branch it scanned is alphabetically the first in our repo. Going into the repository and looking at it's scan log, the scan does not happen at all. I don't recall in this setup if it is the organization or repository level scan that normally would have kicked off our build. I tested making a commit to the one branch that the org scan does detect, and that did not initiate a repository level scan.    Jesse Glick if you have any suggestions of things to test/try/log or what not I'm all ears. This is having a big impact on our teams workflow.  I'll keep digging, but I suspect this is the same issue as described above as the repository level of the GitHub Org folder appears to be the same thing as a multibranch pipeline project.    Also worth nothing, when viewing the config at the repository level it has.a scan interval set at 1 day. Since this config is created by the GitHub Org folder, we can't configure it. But despite that, it's not actually running once a day. I'm looking at it right now and the last scan was when we manually kicked it off 2 days ago. 
            Hide
            hardwickj James Hardwick added a comment -

            Actually, while monitoring the log I saw this when the scan ran:

             

            WARNING: JENKINS-45892: reference to jenkins.branch.OrganizationFolder@34cac973[SC] being saved from unexpected /var/lib/jenkins/jobs/SC/computation/computation.xml
            java.lang.IllegalStateException
             at hudson.XmlFile.replaceIfNotAtTopLevel(XmlFile.java:223)
             ...
             at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.save(FolderComputation.java:208)
             at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:192)
             at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:911)
             at hudson.model.ResourceController.execute(ResourceController.java:97)
             at hudson.model.Executor.run(Executor.java:429)

             

            Not sure yet where that puts my issue in relation to the originally reported one, but still commenting in the hope it helps...

            Show
            hardwickj James Hardwick added a comment - Actually, while monitoring the log I saw this when the scan ran:   WARNING: JENKINS-45892: reference to jenkins.branch.OrganizationFolder@34cac973[SC] being saved from unexpected /var/lib/jenkins/jobs/SC/computation/computation.xml java.lang.IllegalStateException at hudson.XmlFile.replaceIfNotAtTopLevel(XmlFile.java:223) ... at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.save(FolderComputation.java:208) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:192) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:911) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)   Not sure yet where that puts my issue in relation to the originally reported one, but still commenting in the hope it helps...
            Hide
            stephenconnolly Stephen Connolly added a comment -

            James Hardwick

            So I found this in one of my computation.xml files:

            <?xml version='1.0' encoding='UTF-8'?>
            <jenkins.branch.OrganizationFolder_-OrganizationScan plugin="branch-api@2.0.17">
            <actions>
            <hudson.model.CauseAction>
            <causeBag class="linked-hash-map">
            <entry>
            <hudson.triggers.TimerTrigger_-TimerTriggerCause/>
            <int>1</int>
            </entry>
            </causeBag>
            </hudson.model.CauseAction>
            <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction plugin="blueocean-rest-impl@1.3.5">
            <blueOceanUrlObject class="io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl">
            <mappedUrl>blue/organizations/jenkins/pipelines/</mappedUrl>
            <modelObject class="hudson.model.Queue$WaitingItem">
            <actions>
            <hudson.model.CauseAction reference="../../../../../hudson.model.CauseAction"/>
            <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction reference="../../../.."/>
            </actions>
            <id>15319</id>
            <task class="jenkins.branch.OrganizationFolder" resolves-to="hudson.model.AbstractItem$Replacer" plugin="branch-api@2.0.17">
            <fullName>cloudbeers</fullName>
            </task>
            <inQueueSince>1515907380276</inQueueSince>
            <timestamp>
            <time>1515907380276</time>
            <timezone>Etc/UTC</timezone>
            </timestamp>
            </modelObject>
            </blueOceanUrlObject>
            </io.jenkins.blueocean.service.embedded.BlueOceanUrlAction>
            <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction reference="../io.jenkins.blueocean.service.embedded.BlueOceanUrlAction"/>
            </actions>
            <result>
            <name>SUCCESS</name>
            <ordinal>0</ordinal>
            <color>BLUE</color>
            <completeBuild>true</completeBuild>
            </result>
            <durations>
            <long>21123</long>
            </durations>
            <timestamp>1515907380282</timestamp>
            <duration>21123</duration>
            </jenkins.branch.OrganizationFolder_-OrganizationScan>

            That smells very fishy... I go digging further and find: https://issues.jenkins-ci.org/browse/JENKINS-48322 so James Hardwick that looks to be the issue you had and upgrading to BlueOcean 1.4 should fix the strange log messages for you

            Show
            stephenconnolly Stephen Connolly added a comment - James Hardwick So I found this in one of my computation.xml files: <?xml version= '1.0' encoding= 'UTF-8' ?> <jenkins.branch.OrganizationFolder_-OrganizationScan plugin= "branch-api@2.0.17" > <actions> <hudson.model.CauseAction> <causeBag class= "linked-hash-map" > <entry> <hudson.triggers.TimerTrigger_-TimerTriggerCause/> < int >1</ int > </entry> </causeBag> </hudson.model.CauseAction> <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction plugin= "blueocean- rest -impl@1.3.5" > <blueOceanUrlObject class= "io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl" > <mappedUrl>blue/organizations/jenkins/pipelines/</mappedUrl> <modelObject class= "hudson.model.Queue$WaitingItem" > <actions> <hudson.model.CauseAction reference= "../../../../../hudson.model.CauseAction" /> <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction reference= "../../../.." /> </actions> <id>15319</id> <task class= "jenkins.branch.OrganizationFolder" resolves-to= "hudson.model.AbstractItem$Replacer" plugin= "branch-api@2.0.17" > <fullName>cloudbeers</fullName> </task> <inQueueSince>1515907380276</inQueueSince> <timestamp> <time>1515907380276</time> <timezone>Etc/UTC</timezone> </timestamp> </modelObject> </blueOceanUrlObject> </io.jenkins.blueocean.service.embedded.BlueOceanUrlAction> <io.jenkins.blueocean.service.embedded.BlueOceanUrlAction reference= "../io.jenkins.blueocean.service.embedded.BlueOceanUrlAction" /> </actions> <result> <name>SUCCESS</name> <ordinal>0</ordinal> <color>BLUE</color> <completeBuild> true </completeBuild> </result> <durations> < long >21123</ long > </durations> <timestamp>1515907380282</timestamp> <duration>21123</duration> </jenkins.branch.OrganizationFolder_-OrganizationScan> That smells very fishy... I go digging further and find: https://issues.jenkins-ci.org/browse/JENKINS-48322  so James Hardwick that looks to be the issue you had and upgrading to BlueOcean 1.4 should fix the strange log messages for you
            Hide
            stephenconnolly Stephen Connolly added a comment -

            James Hardwick

            When I check the organization scan log, I see this:

            Examining
            our_org_name/our_repo_name
            Checking branches... Getting remote branches... Checking branch
            bug/branch_name_here
            ‘Jenkinsfile’ found Met criteria 1 branches were processed (query completed) 1 branches were processed Finished examining our_org_name/our_repo_name
            

            ...but we have probably 15 branches with a Jenkinsfile in them. It just so happens to be that the only branch it scanned is alphabetically the first in our repo.

            So the Organization Folder just needs to know that there is at least one branch in the repository in order to create the child multibranch project. Once the child multibranch project has been created the responsibility for building belongs to the child.

            The organization folder is designed not to burn excessive rate limit checking unnecessary branches (it could be even smarter, e.g. by peeking inside an existing multibranch project and checking the branches in order of most recent build rather than alpha order, but that's a separate issue) 

            Now recently we fixed a bug in Organization Folder, namely there was a side-effect where every scan of an organization folder would always trigger a scan of all the child Multibranch projects (the fix was in JENKINS-48536). The fixed behaviour only triggers the child Multibranch project if the multibranch project was disabled (because there are no more branches with marker files) and has been re-enabled (e.g. at least one branch with a marker file has been introduced). The multibranch project being disabled will ignore all push events and not run periodic scans, so the organization folder needs to take over that responsibility. The bug was that previously it was being over-zealous.

            Now the question is whether the per-repository scans are taking place or not... but that is really a question for a new issue.

            In short James Hardwick could you create a new JIRA issue for the issue you are seeing with the multibranch projects in an org folder not being periodically scanned.

            Show
            stephenconnolly Stephen Connolly added a comment - James Hardwick When I check the organization scan log, I see this: Examining our_org_name/our_repo_name Checking branches... Getting remote branches... Checking branch bug/branch_name_here ‘Jenkinsfile’ found Met criteria 1 branches were processed (query completed) 1 branches were processed Finished examining our_org_name/our_repo_name ...but we have probably 15 branches with a Jenkinsfile in them. It just so happens to be that the only branch it scanned is alphabetically the first in our repo. So the Organization Folder just needs to know that there is at least one branch in the repository in order to create the child multibranch project. Once the child multibranch project has been created the responsibility for building belongs to the child. The organization folder is designed not to burn excessive rate limit checking unnecessary branches (it could be even smarter, e.g. by peeking inside an existing multibranch project and checking the branches in order of most recent build rather than alpha order, but that's a separate issue)  Now recently we fixed a bug in Organization Folder, namely there was a side-effect where every scan of an organization folder would always trigger a scan of all the child Multibranch projects (the fix was in JENKINS-48536 ). The fixed behaviour only triggers the child Multibranch project if the multibranch project was disabled (because there are no more branches with marker files) and has been re-enabled (e.g. at least one branch with a marker file has been introduced). The multibranch project being disabled will ignore all push events and not run periodic scans, so the organization folder needs to take over that responsibility. The bug was that previously it was being over-zealous. Now the question is whether the per-repository scans are taking place or not... but that is really a question for a new issue. In short  James Hardwick could you create a new JIRA issue for the issue you are seeing with the multibranch projects in an org folder not being periodically scanned.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            FYI I just checked my own personal Jenkins instance, here's an org folder and it's most recent scan:

            And here's a child multibranch project and it's most recent scan:

            Notice that the times are completely unrelated (which is correct and proper because Periodically unless otherwise ran explicitly does not let you pick the time and instead randomizes the time to level the load)

            Thus James Hardwick the only real question is was that 1 day or 2 days since either of those ran... if it is 1 day then I think we can close this issue as fixed (after confirming that 1 minute does scan every minute... need to use a git repo for that though, otherwise the rate limit will get in the way)

            Show
            stephenconnolly Stephen Connolly added a comment - FYI I just checked my own personal Jenkins instance, here's an org folder and it's most recent scan: And here's a child multibranch project and it's most recent scan: Notice that the times are completely unrelated (which is correct and proper because Periodically unless otherwise ran explicitly does not let you pick the time and instead randomizes the time to level the load) Thus James Hardwick the only real question is was that 1 day or 2 days since either of those ran... if it is 1 day then I think we can close this issue as fixed (after confirming that 1 minute does scan every minute... need to use a git repo for that though, otherwise the rate limit will get in the way)
            Hide
            hardwickj James Hardwick added a comment -

            The original one I was referring to last had a commit on a Friday later afternoon. I was reporting it here on Sunday early afternoon, so almost 2 days. 

            New ticket for my issue created at JENKINS-48960

            Show
            hardwickj James Hardwick added a comment - The original one I was referring to last had a commit on a Friday later afternoon. I was reporting it here on Sunday early afternoon, so almost 2 days.  New ticket for my issue created at  JENKINS-48960
            Hide
            samupl Jakub Szafranski added a comment -

            We're seeing the same issue on our Jenkins installation, version 2.89.4. After updating Jenkins and all plugins to latest versions this week (it was actually an old installation), the periodic organization scan stopped triggering at all.

             

            I can still manually kickstart this scan, and it works as expected then, but for some reason it doesn't run on its own - and I have no idea how to debug it. If instructed, I will be happy to provide any debugging information.

            Show
            samupl Jakub Szafranski added a comment - We're seeing the same issue on our Jenkins installation, version 2.89.4. After updating Jenkins and all plugins to latest versions this week (it was actually an old installation), the periodic organization scan stopped triggering at all.   I can still manually kickstart this scan, and it works as expected then, but for some reason it doesn't run on its own - and I have no idea how to debug it. If instructed, I will be happy to provide any debugging information.
            Hide
            billoneil Bill O'Neil added a comment -

            Also seeing this issue on Jenkins ver. 2.89.4 from a clean docker install. All plugins are up to date. The timer never seems to trigger the scan but manual scans work fine.

            Show
            billoneil Bill O'Neil added a comment - Also seeing this issue on Jenkins ver. 2.89.4 from a clean docker install. All plugins are up to date. The timer never seems to trigger the scan but manual scans work fine.
            Hide
            stephenconnolly Stephen Connolly added a comment - - edited

            So I think what has happened is that we have uncovered a bug in fixing another bug.

            Up until "recently" the branch-api plugin's OrganizationFolder functionality would end up triggering a rescan of all child multibranch projects whenever it actually did a rescan. This was a bug, because the intent would be, if you setup the org scan as daily and the multibranch as daily (ignore that you cannot configure the multibranch's period at present) then you don't want all the scans at the same time in the day, rather you want them leveled throughout the day so that if you are using an SCM with an API rate limit, the rate limit usage is leveled across the whole day. By the organization folder triggering the scans of all child multibranch projects, effectively the rate limit usage would be spiked at one time in the day, which is actually a bad thing.

            So what has made this bug surface is this change which fixed the bug whereby the Organization folder would always trigger a rescan of the child multibranch rather than only triggering a rescan if the scan configuration has changed.

            Because the Organization folder was no longer driving periodic scans, we now see the bug - which I suspect was always present - of the Multibranch's periodic setting not actually being applied.

            On my own Jenkins the periodic setting is applied on at least some projects, so this is somewhat inconsistent, but it is incredibly annoying.

            Show
            stephenconnolly Stephen Connolly added a comment - - edited So I think what has happened is that we have uncovered a bug in fixing another bug. Up until "recently" the branch-api plugin's OrganizationFolder functionality would end up triggering a rescan of all child multibranch projects whenever it actually did a rescan. This was a bug, because the intent would be, if you setup the org scan as daily and the multibranch as daily (ignore that you cannot configure the multibranch's period at present) then you don't want all the scans at the same time in the day, rather you want them leveled throughout the day so that if you are using an SCM with an API rate limit, the rate limit usage is leveled across the whole day. By the organization folder triggering the scans of all child multibranch projects, effectively the rate limit usage would be spiked at one time in the day, which is actually a bad thing. So what has made this bug surface is this change which fixed the bug whereby the Organization folder would always trigger a rescan of the child multibranch rather than only triggering a rescan if the scan configuration has changed. Because the Organization folder was no longer driving periodic scans, we now see the bug - which I suspect was always present - of the Multibranch's periodic setting not actually being applied. On my own Jenkins the periodic setting is applied on at least some projects, so this is somewhat inconsistent, but it is incredibly annoying.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Hmmm so https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/triggers/Trigger.java#L226 is responsible to call Trigger.checkTriggers periodically.

            https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/triggers/Trigger.java#L269 runs the triggers... but only on ParameterizedJobMixIn.ParameterizedJob (at least since this change) so how is the org folder getting triggered?

            Ahhh folders takes care of it itself

            Ok... so can people affected by this issue try running the following script in the system groovy console:

            for (def p in Jenkins.getInstance().allItems(com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.class)) {
                for (def t in p.getTriggers().values()) {
                  if (t.tabs == null) println("${p.fullName} trigger ${t.class.name} not started");
                }
            }
            

            If you get a report of any of the triggers not being started then that would point to an NPE in FolderCron being the root cause.

            NOTE: I do not expect the above to be the cause. My Jenkins is on 2.89.4 and on a newly created org folder it is fine as well as on an existing org folder, e.g. see the following three screenshots that illustrate the org folder scan being daily and the multibranch scans being daily - but at different points in time - and I had not logged into my Jenkins in over a week before taking these screenshots!

            Show
            stephenconnolly Stephen Connolly added a comment - Hmmm so https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/triggers/Trigger.java#L226 is responsible to call Trigger.checkTriggers periodically. https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/triggers/Trigger.java#L269 runs the triggers... but only on ParameterizedJobMixIn.ParameterizedJob (at least since this change ) so how is the org folder getting triggered? Ahhh folders takes care of it itself Ok... so can people affected by this issue try running the following script in the system groovy console: for (def p in Jenkins.getInstance().allItems(com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.class)) { for (def t in p.getTriggers().values()) { if (t.tabs == null ) println( "${p.fullName} trigger ${t. class. name} not started" ); } } If you get a report of any of the triggers not being started then that would point to an NPE in FolderCron being the root cause. NOTE: I do not expect the above to be the cause. My Jenkins is on 2.89.4 and on a newly created org folder it is fine as well as on an existing org folder, e.g. see the following three screenshots that illustrate the org folder scan being daily and the multibranch scans being daily - but at different points in time - and I had not logged into my Jenkins in over a week before taking these screenshots!
            Hide
            stephenconnolly Stephen Connolly added a comment -

            What I do find interesting is this one:

            There is more than 24h since that one ran. It should run once every 24h, but it seems to be running every 48h.

            Hmmm H H * * * is the crontab... so if that picks any time before 19:49 then yesterday it would have been less than 24h since the last run and so it would only pick up the run today... presumably it will run at 19:49 today but the last triggered field will be out my some number of milliseconds and so effectively it runs every second interval

            Show
            stephenconnolly Stephen Connolly added a comment - What I do find interesting is this one: There is more than 24h since that one ran. It should run once every 24h, but it seems to be running every 48h. Hmmm H H * * * is the crontab... so if that picks any time before 19:49 then yesterday it would have been less than 24h since the last run and so it would only pick up the run today... presumably it will run at 19:49 today but the last triggered field will be out my some number of milliseconds and so effectively it runs every second interval
            Hide
            stephenconnolly Stephen Connolly added a comment -

            https://github.com/jenkinsci/cloudbees-folder-plugin/pull/119 might fix the incidental issue of "every minute" not running every minute... but it would be really good to understand how others are not getting the child multibranch scan being triggered at least once every two days?

            OR is it being triggered once every two days, but the org folder is set to a higher interval and thus "longer than every 2 days" is being translated into "never" because people are manually triggering as a workaround to being unable to configure the child triggers

            Show
            stephenconnolly Stephen Connolly added a comment - https://github.com/jenkinsci/cloudbees-folder-plugin/pull/119 might fix the incidental issue of "every minute" not running every minute... but it would be really good to understand how others are not getting the child multibranch scan being triggered at least once every two days? OR is it being triggered once every two days, but the org folder is set to a higher interval and thus "longer than every 2 days" is being translated into "never" because people are manually triggering as a workaround to being unable to configure the child triggers
            Hide
            aairey Andy Airey added a comment -

            We have to scan all our multibranch pipeline repositories separately and cannot configure them (yet) as stated above...
            Otherwise the branch and PR jobs never get triggered.

            Also the new Bitbucket webhooks are not supported (JENKINS-47891) yet so we are kind of stuck  as we cannot install extra plugins on Bitbucket Server.
            This is really annoying.

            Show
            aairey Andy Airey added a comment - We have to scan all our multibranch pipeline repositories separately and cannot configure them (yet) as stated above... Otherwise the branch and PR jobs never get triggered. Also the new Bitbucket webhooks are not supported ( JENKINS-47891 ) yet so we are kind of stuck  as we cannot install extra plugins on Bitbucket Server. This is really annoying.
            Hide
            mslattery Michael Slattery added a comment - - edited

            As a workaround, we created a pipeline job as follows:

            import org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject;
            node {
                Jenkins.instance.items
                .findAll { it instanceof WorkflowMultiBranchProject }
                .each {
                    echo it.name
                    it.indexing.run()
                }
            }
            

            I set it to trigger to build periodically, every two minutes: H/2 * * * *

            But, I had to do script approvals.  This is temporary for us until this bug gets fixed, after which I'll remove this job and the script approvals.

            Show
            mslattery Michael Slattery added a comment - - edited As a workaround, we created a pipeline job as follows: import org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject; node { Jenkins.instance.items .findAll { it instanceof WorkflowMultiBranchProject } .each { echo it.name it.indexing.run() } } I set it to trigger to build periodically, every two minutes: H/2 * * * * But, I had to do script approvals.  This is temporary for us until this bug gets fixed, after which I'll remove this job and the script approvals.
            Hide
            dsudduth Derek Sudduth added a comment -

            Stephen Connolly, I noticed this was marked as resolved. There are no comments regarding the fix or release. Do you know when we can expect to pick up the new plugin version? And what that version will be?

            Thanks in advance!

            Derek

            Show
            dsudduth Derek Sudduth added a comment - Stephen Connolly , I noticed this was marked as resolved . There are no comments regarding the fix or release. Do you know when we can expect to pick up the new plugin version? And what that version will be? Thanks in advance! Derek
            Hide
            aurimasnav Aurimas Navardauskas added a comment -
            Show
            aurimasnav Aurimas Navardauskas added a comment - Derek Sudduth I think it is released as 6.7 -  https://plugins.jenkins.io/cloudbees-folder
            Hide
            dsudduth Derek Sudduth added a comment -

            Thanks Aurimas, looks like it's the subdirectories (child jobs) that still exhibit the issue. That's what I'm experiencing after reading Stephen's comments further.

            Show
            dsudduth Derek Sudduth added a comment - Thanks Aurimas, looks like it's the subdirectories (child jobs) that still exhibit the issue. That's what I'm experiencing after reading Stephen's comments further.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Subfolders have their own period of scanning that is not the period of the parent

            Show
            stephenconnolly Stephen Connolly added a comment - Subfolders have their own period of scanning that is not the period of the parent
            Hide
            dsudduth Derek Sudduth added a comment -

            Stephen Connolly - You're correct. There are settings generated for each repo where a Jenkinsfile is detected. However, there is an issue with permissions when dynamically generating jobs. The settings are viewable, but permissions are denied. Separate issue, I know. But, it does make it challenging to modify the scan interval.

            Looks like more of a global issue outside the scope of this plugin. It is also seen in the Github plugin.

            https://stackoverflow.com/questions/48654556/jenkins-how-to-set-the-periodic-scan-interval-of-jobs-under-github-organization

            Show
            dsudduth Derek Sudduth added a comment - Stephen Connolly - You're correct. There are settings generated for each repo where a Jenkinsfile is detected. However, there is an issue with permissions when dynamically generating jobs. The settings are viewable, but permissions are denied. Separate issue, I know. But, it does make it challenging to modify the scan interval. Looks like more of a global issue outside the scope of this plugin. It is also seen in the Github plugin. https://stackoverflow.com/questions/48654556/jenkins-how-to-set-the-periodic-scan-interval-of-jobs-under-github-organization

              People

              • Assignee:
                stephenconnolly Stephen Connolly
                Reporter:
                wojciechrak Wojciech Rak
              • Votes:
                9 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: