Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31801

Pipeline Support in Throttle Concurrent Builds

    Details

    • Similar Issues:

      Description

      I am looking to use Throttle Concurrent Builds in a WorkFlow. I have jobs that perform automated, scheduled testing, as well as user-initiated jobs. All these jobs compete for the same set of hardware resources to run tests against. I'm currently using BuildFlows for all this and Throttle Concurrent Builds works fine. My problem is that I want to use the WorkFlow plugin for the automated jobs but TCB does not natively support WorkFlows.

      I've read that people dedicate a Node per unit of hardware to solve this problem. That's not a good solution for me since I can have 10-20 hardware units. I've also ready about people using a WorkFlow Stage, but this also doesn't work because of the user-initiated jobs that are outside of the WorkFlow.

      I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin, even if the plugin does not have native WorkFlow support. Is this possible for TCB and/or is there TCB support planned for WorkFlow?

      Thanks,

      Dave

        Attachments

          Issue Links

            Activity

            daveml Dave Lawrence created issue -
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Labels concurrent jenkins throttle workflow community-bee concurrent jenkins throttle workflow
            Hide
            abayer Andrew Bayer added a comment -
            Show
            abayer Andrew Bayer added a comment - Looks like https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/32 is pertinent - diving in there.
            Hide
            abayer Andrew Bayer added a comment -

            So that change does make it possible to use TCB in its normal way with Workflow jobs - i.e., you can't use TCB within the Workflow itself, but you can use it to throttle groups of Workflow jobs, etc. I'm guessing that's not all you want, but is that a start?

            Show
            abayer Andrew Bayer added a comment - So that change does make it possible to use TCB in its normal way with Workflow jobs - i.e., you can't use TCB within the Workflow itself, but you can use it to throttle groups of Workflow jobs, etc. I'm guessing that's not all you want, but is that a start?
            rtyler R. Tyler Croy made changes -
            Rank Ranked higher
            rtyler R. Tyler Croy made changes -
            Rank Ranked lower
            Hide
            abayer Andrew Bayer added a comment -

            Ping?

            Show
            abayer Andrew Bayer added a comment - Ping?
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Dave Lawrence. Yes, this is a known case (was also a blocker at my previous company). Actually I would say that workflow's implementation is not very suitable to remote hardware management due to the failover robustness (which cannot be disabled IIRC). If Jenkins or its node restarts, WF will continue the execution and will restore ONLY it's own state. Currently the state of the remote stuff cannot be restored from Workflow => jobs may behave unreliably.

            > I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin

            I doubt it's correct. Could you refer this presentation? If it is misguiding, we will need to fix it.

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited Dave Lawrence . Yes, this is a known case (was also a blocker at my previous company). Actually I would say that workflow's implementation is not very suitable to remote hardware management due to the failover robustness (which cannot be disabled IIRC). If Jenkins or its node restarts, WF will continue the execution and will restore ONLY it's own state. Currently the state of the remote stuff cannot be restored from Workflow => jobs may behave unreliably. > I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin I doubt it's correct. Could you refer this presentation? If it is misguiding, we will need to fix it.
            Hide
            abayer Andrew Bayer added a comment -

            Jesse Glick - pinging you on this to warn you I wanna talk this over with you tomorrow. =)

            Show
            abayer Andrew Bayer added a comment - Jesse Glick - pinging you on this to warn you I wanna talk this over with you tomorrow. =)
            Hide
            daveml Dave Lawrence added a comment -

            Thanks to all who have responded. I appreciate the effort.

            To Andrew's point "...but you can use it to throttle groups of Workflow jobs..."
            Using TCB within the Workflow is where the value is. Applying a throttle over tan entire Workflow causes other problems as I'm sure you can imagine.

            Oleg:
            I'm sorry but I could not find the presentation. I know Cyrille gave it and it was older, perhaps Jan 2015. As for the Workflow failover, I understand the point, but it's not a concern for me. Hopefully TCB support can get worked into Workflow.

            Dave

            Show
            daveml Dave Lawrence added a comment - Thanks to all who have responded. I appreciate the effort. To Andrew's point "...but you can use it to throttle groups of Workflow jobs..." Using TCB within the Workflow is where the value is. Applying a throttle over tan entire Workflow causes other problems as I'm sure you can imagine. Oleg: I'm sorry but I could not find the presentation. I know Cyrille gave it and it was older, perhaps Jan 2015. As for the Workflow failover, I understand the point, but it's not a concern for me. Hopefully TCB support can get worked into Workflow. Dave
            abayer Andrew Bayer made changes -
            Link This issue is related to JENKINS-26125 [ JENKINS-26125 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            COMPATIBILITY.md
            http://jenkins-ci.org/commit/workflow-plugin/5578cc034790402d514557648f076db44143ed02
            Log:
            Add throttle-concurrent-builds plugin to the list

            https://issues.jenkins-ci.org/browse/JENKINS-31801

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: COMPATIBILITY.md http://jenkins-ci.org/commit/workflow-plugin/5578cc034790402d514557648f076db44143ed02 Log: Add throttle-concurrent-builds plugin to the list https://issues.jenkins-ci.org/browse/JENKINS-31801
            Hide
            jglick Jesse Glick added a comment -

            The easy integration would be to make the JobProperty applicable to Job, and call SubTask.getOwnerTask. The nicer integration would be a step that you could call on a particular node.

            Show
            jglick Jesse Glick added a comment - The easy integration would be to make the JobProperty applicable to Job , and call SubTask.getOwnerTask . The nicer integration would be a step that you could call on a particular node .
            hrmpw Patrick Wolf made changes -
            Epic Link JENKINS-34657 [ 170293 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary Using Throttle Concurrent Builds in a WorkFlow Using Throttle Concurrent Builds in a Pipeline
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 167248 ] JNJira + In-Review [ 182654 ]
            abayer Andrew Bayer made changes -
            Labels community-bee concurrent jenkins throttle workflow community-bee concurrent jenkins pipeline throttle workflow
            abayer Andrew Bayer made changes -
            Labels community-bee concurrent jenkins pipeline throttle workflow community-bee concurrent jenkins pipeline throttle
            Hide
            mcrooney mcrooney added a comment -

            I suppose a work-around in the meantime could be to create a new node running on the master (ssh localhost or run slave.jar), that has one executor and a label, and then require that label in your job. You can decrease the number of executors on your master by one if you want it to be identical from a bandwidth perspective!

            Show
            mcrooney mcrooney added a comment - I suppose a work-around in the meantime could be to create a new node running on the master (ssh localhost or run slave.jar), that has one executor and a label, and then require that label in your job. You can decrease the number of executors on your master by one if you want it to be identical from a bandwidth perspective!
            Hide
            abayer Andrew Bayer added a comment -

            fwiw, I'm going to try to take another look at this soon.

            Show
            abayer Andrew Bayer added a comment - fwiw, I'm going to try to take another look at this soon.
            Hide
            mcrooney mcrooney added a comment -

            Thanks Andrew, that'd be awesome! For now I am using the executor hack I described above, but it isn't very elegant so it'd be awesome to have native support for this.

            Show
            mcrooney mcrooney added a comment - Thanks Andrew, that'd be awesome! For now I am using the executor hack I described above, but it isn't very elegant so it'd be awesome to have native support for this.
            Hide
            nfrydenholm Niels Frydenholm added a comment -

            Any news/progress on the ability to use the Throttle from Pipelines? It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

            Show
            nfrydenholm Niels Frydenholm added a comment - Any news/progress on the ability to use the Throttle from Pipelines? It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-37809 [ JENKINS-37809 ]
            oleg_nenashev Oleg Nenashev made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary Using Throttle Concurrent Builds in a Pipeline Pipeline Support in Throttle Concurrent Builds
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java
            src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java
            src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottledStepInfo.java
            src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/af877cd2ebd90c669a2bc1f87bcc0b3814a8a108
            Log:
            JENKINS-31801 Initial work on throttle(category) step - needs tests

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottledStepInfo.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/af877cd2ebd90c669a2bc1f87bcc0b3814a8a108 Log: JENKINS-31801 Initial work on throttle(category) step - needs tests
            Hide
            abayer Andrew Bayer added a comment -

            Very initial work-in-progress PR up at https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/46 to add a throttle("some-category") step.

            Show
            abayer Andrew Bayer added a comment - Very initial work-in-progress PR up at https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/46  to add a throttle("some-category") step.
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #46 (Web Link)" [ 15639 ]
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            abayer Andrew Bayer made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            jglick Jesse Glick added a comment -

            It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

            You can do this already with the lock step.

            Show
            jglick Jesse Glick added a comment - It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time. You can do this already with the lock step.
            Hide
            orrc Christopher Orr added a comment -

            You can do this already with the lock step.

            AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).

            Show
            orrc Christopher Orr added a comment - You can do this already with the lock step. AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).
            Hide
            abayer Andrew Bayer added a comment -

            Jesse Glick Not exactly - the use case that Throttle was written for originally was as follows:

            • Multiple {{Node}}s with multiple executors on each of them.
            • Two different jobs (or types of jobs) - let's say backend tests and frontend tests.
            • You can only run one backend test and one frontend test at a time on each node, but you can run one of each.
            • So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time"
            • The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada.

            I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch"). I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.

            Show
            abayer Andrew Bayer added a comment - Jesse Glick Not exactly - the use case that Throttle was written for originally was as follows: Multiple {{Node}}s with multiple executors on each of them. Two different jobs (or types of jobs) - let's say backend tests and frontend tests. You can only run one backend test and one frontend test at a time on each node, but you can run one of each. So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time" The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada. I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch") . I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.
            Hide
            jglick Jesse Glick added a comment -

            lock is running before the node step

            Well, you can give each node extra executors and

            node('backend-or-frontend') {
              lock("frontend-$NODE_NAME") {
                sh './run-frontend-tests'
              }
            }
            

            Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction).

            Show
            jglick Jesse Glick added a comment - lock is running before the node step Well, you can give each node extra executors and node( 'backend-or-frontend' ) { lock( "frontend-$NODE_NAME" ) { sh './run-frontend-tests' } } Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction ).
            Hide
            bmanuel Benjamin Manuel added a comment -

            Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.

            Show
            bmanuel Benjamin Manuel added a comment - Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java
            src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java
            src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html
            src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties
            src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly
            src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html
            src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java
            src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d
            Log:
            JENKINS-31801 Add Pipeline throttle(category) step (#46)

            • JENKINS-31801 Initial work on throttle(category) step - needs tests
            • Reworked to no longer rely on StepExecutions.
            • Add a trailing newline to messages.
            • Make findbugs happy.
            • Whoops, this needs to take a block
            • Initial test, actually working

            Needed to bump to newer dependency versions, most notably to get
            PlaceholderTask.getNode(). Still a work in progress, mind you.

            • Cleanup, commenting, javadoc
            • Test across all nodes

            Also discovered that Run<?,?> is a very bad Map key.

            • Add interop with freestyle test
            • Add snippet generator support.
            • Add snippetizer support and test
            • Review comments
            • Check for pending PlaceholderTasks as well.
            • Allow multiple comma-separated categories
            • Minor review responses, moving step UI to correct dir
            • Go away, empty category names!
            • Fixed up snippetizer, switched to a list of strings for the throttle step
            • Check for and respond to duplicate or non-existent category names
            • Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline
            • unmodifiableList
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d Log: JENKINS-31801 Add Pipeline throttle(category) step (#46) JENKINS-31801 Initial work on throttle(category) step - needs tests Reworked to no longer rely on StepExecutions. Add a trailing newline to messages. Make findbugs happy. Whoops, this needs to take a block Initial test, actually working Needed to bump to newer dependency versions, most notably to get PlaceholderTask.getNode(). Still a work in progress, mind you. Cleanup, commenting, javadoc Test across all nodes Also discovered that Run<?,?> is a very bad Map key. Add interop with freestyle test Add snippet generator support. Add snippetizer support and test Review comments Check for pending PlaceholderTasks as well. Allow multiple comma-separated categories Minor review responses, moving step UI to correct dir Go away, empty category names! Fixed up snippetizer, switched to a list of strings for the throttle step Check for and respond to duplicate or non-existent category names Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline unmodifiableList
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            CHANGELOG.md
            LICENSE.txt
            README
            README.md
            doc/images/abstractProject_jobProperty.png
            doc/images/abstractProject_matrixFlags.png
            doc/images/global_categoryConfig.png
            pom.xml
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa
            Log:
            JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47)

            • Add License
            • Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801
            • Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"
            • Update Parent POM
            • Update documentation to reflect the current state of Pipeline support
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md LICENSE.txt README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png pom.xml http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa Log: JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47) Add License Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801 Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon" Update Parent POM Update documentation to reflect the current state of Pipeline support
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            README
            README.md
            doc/images/abstractProject_jobProperty.png
            doc/images/abstractProject_matrixFlags.png
            doc/images/global_categoryConfig.png
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38
            Log:
            Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38 Log: Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            CHANGELOG.md
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6
            Log:
            Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6 Log: Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - The feature has been released in the TCB plugin 2.0. https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/CHANGELOG.md#20
            oleg_nenashev Oleg Nenashev made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2037 (Web Link)" [ 18454 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2007 (Web Link)" [ 18462 ]

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                daveml Dave Lawrence
              • Votes:
                35 Vote for this issue
                Watchers:
                52 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: