Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34268

Lock multiple resources using the Pipeline lock step

    Details

    • Similar Issues:

      Description

      The current implementation of Pipeline lock step allows to block a single resource.

      It should be extended to cover all the functionality of the plugin (applicable to non-freestyle jobs) such as blocking resources by label or request a lock for N resources.

      The DSL must be something like this:

      lock (resources: ['resource1', 'resource2']) {
        ... execution block ...
      }
      

      or

      lock (label: 'my-resources') {
        ... execution block ...
      }
      

      The behavior of the label parameter would be equivalent to:

      lock (resources: ['resource3', 'resource4']) { // if both resource3 and resource4 are labeled as 'my-resources'
        ... execution block ...
      }
      

        Attachments

          Issue Links

            Activity

            amuniz Antonio Muñiz created issue -
            Hide
            aeon512 Florian Hug added a comment -

            Besides the actual label functionality, also the possibility to request a certain amount of resources from this label would be essential.

            I would personally prefer something like

            lock(label: 'fooBar', quantity: 2) {
              // whatever
            }
            

            Actually this is what keeps us from using pipelines.

            Show
            aeon512 Florian Hug added a comment - Besides the actual label functionality, also the possibility to request a certain amount of resources from this label would be essential. I would personally prefer something like lock(label: 'fooBar' , quantity: 2) { // whatever } Actually this is what keeps us from using pipelines.
            Hide
            amuniz Antonio Muñiz added a comment -

            Florian Hug Could you explain a real use case where the quantity configuration makes sense please? It's not clear to me why one would want to acquire 2 resources but do not caring about which ones specifically.

            Show
            amuniz Antonio Muñiz added a comment - Florian Hug Could you explain a real use case where the quantity configuration makes sense please? It's not clear to me why one would want to acquire 2 resources but do not caring about which ones specifically.
            Hide
            aeon512 Florian Hug added a comment - - edited

            Background:
            We dynamically create virtual machines which acts as jenkins slave (using the swarm-plugin) to perform the actual build steps. Thereby, if an error occurs we can archive the whole VM such that a developer can investigate and solve the problem. Using linked snapshots the whole purpose of creating such a new VM takes simply seconds and works very well.

            Problem solved by lockable resources:
            Since the overall amount of memory and cores on the host machine is limited, we used the lockable resources plugin to manage the distribution of memory (slices) and cpu cores.

            For example, let's say we have configured 16 resources (Memory Slice 1, Memory Slice 2, ..., Memory Slice 3), each representing e.g. 4 GB Memory. that is a total of 64 GB being available for virtual machines. When a job starts and wants to create a VM with e.g. 16 GB virtual memory, the job simply tries to request 4 slices ( =quantity ) of 4 GB memory

            lock(label: 'memory', quantity: 4) {
               // actual build steps
            }
            

            If this succeeds there is enough memory available, and the job can continue. If not, the job blocks until the necessary amount of memory is available again (since other jobs have finished and shut down their VMs).

            The same principle applies to CPU (cores) as well.

            Does this explanation help?

            Show
            aeon512 Florian Hug added a comment - - edited Background: We dynamically create virtual machines which acts as jenkins slave (using the swarm-plugin ) to perform the actual build steps. Thereby, if an error occurs we can archive the whole VM such that a developer can investigate and solve the problem. Using linked snapshots the whole purpose of creating such a new VM takes simply seconds and works very well. Problem solved by lockable resources: Since the overall amount of memory and cores on the host machine is limited, we used the lockable resources plugin to manage the distribution of memory (slices) and cpu cores. For example, let's say we have configured 16 resources (Memory Slice 1, Memory Slice 2, ..., Memory Slice 3), each representing e.g. 4 GB Memory. that is a total of 64 GB being available for virtual machines. When a job starts and wants to create a VM with e.g. 16 GB virtual memory, the job simply tries to request 4 slices ( =quantity ) of 4 GB memory lock(label: 'memory' , quantity: 4) { // actual build steps } If this succeeds there is enough memory available, and the job can continue. If not, the job blocks until the necessary amount of memory is available again (since other jobs have finished and shut down their VMs). The same principle applies to CPU (cores) as well. Does this explanation help?
            Hide
            amuniz Antonio Muñiz added a comment -

            Thanks for take the time to explain!

            Makes sense, although nothing is preventing the steps inside the lock block to consume more CPU/RAM than virtually acquired, right?

            Show
            amuniz Antonio Muñiz added a comment - Thanks for take the time to explain! Makes sense, although nothing is preventing the steps inside the lock block to consume more CPU/RAM than virtually acquired, right?
            Hide
            aeon512 Florian Hug added a comment -

            Yes and no
            No: Once the VM is configured to a certain amount of CPU/RAM, that amount is fixed and no process inside the VM can use more. If a process inside such a VM want's to consume more memory it simply receives an OutOfMemory exception.
            Yes: However, in theory, you can lock a different amount than you configure the VM.

            However, in our pipeline scripts we define a variable, e.g. required_memory_slices and use this variable to specify the necessary quantity as well as use this variable when setting up the virtual machine. Hence, in this case, it is guaranteed that we lock the same amount that we configure the VM to use, and hence no process can consume more CPU/RAM.

            Show
            aeon512 Florian Hug added a comment - Yes and no No: Once the VM is configured to a certain amount of CPU/RAM, that amount is fixed and no process inside the VM can use more. If a process inside such a VM want's to consume more memory it simply receives an OutOfMemory exception. Yes: However, in theory, you can lock a different amount than you configure the VM. However, in our pipeline scripts we define a variable, e.g. required_memory_slices and use this variable to specify the necessary quantity as well as use this variable when setting up the virtual machine. Hence, in this case, it is guaranteed that we lock the same amount that we configure the VM to use, and hence no process can consume more CPU/RAM.
            amuniz Antonio Muñiz made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-30269 [ JENKINS-30269 ]
            amuniz Antonio Muñiz made changes -
            Link This issue is related to JENKINS-34273 [ JENKINS-34273 ]
            Hide
            amuniz Antonio Muñiz added a comment -

            Ok, looks good. I've created a separate issue (JENKINS-34273) for quantity support in Pipeline as it can be handled independently of this one.

            BTW you have there a nice CI configuration

            Show
            amuniz Antonio Muñiz added a comment - Ok, looks good. I've created a separate issue ( JENKINS-34273 ) for quantity support in Pipeline as it can be handled independently of this one. BTW you have there a nice CI configuration
            Hide
            aeon512 Florian Hug added a comment -

            Thanks And we are keen on switching to the new Pipeline - but JENKINS-34273 is absolutely necessary for us.

            Show
            aeon512 Florian Hug added a comment - Thanks And we are keen on switching to the new Pipeline - but JENKINS-34273 is absolutely necessary for us.
            Hide
            aeon512 Florian Hug added a comment -

            Antonio Muñiz, maybe you could have a look at PR26 where I tried to implement the functionality for label and quantity. Since this is my first Jenkins plugin contribution (and my first actual Java implementation) I'm open for any kind of feedback. Would be great if we get this merged in.

            lock(label: "MyLabelName", quantity: 3) {
                // body
            }
            

            Note however, that for closing this Issue the functionality to specify a list of resources is still missing. JENKINS-34273 however could be closed completely.
            I couldn't figure out how the DataBoundSetter for this would need to be modified within the actual LockStep. The actual implementation for dealing with several required resources is already available.

            Show
            aeon512 Florian Hug added a comment - Antonio Muñiz , maybe you could have a look at PR26 where I tried to implement the functionality for label and quantity . Since this is my first Jenkins plugin contribution (and my first actual Java implementation) I'm open for any kind of feedback. Would be great if we get this merged in. lock(label: "MyLabelName" , quantity: 3) { // body } Note however, that for closing this Issue the functionality to specify a list of resources is still missing. JENKINS-34273 however could be closed completely. I couldn't figure out how the DataBoundSetter for this would need to be modified within the actual LockStep . The actual implementation for dealing with several required resources is already available.
            Hide
            haney David Haney added a comment -

            Should the fix for this include exposing the names of the acquired resource within the body? For instance if I have something like:

            lock (label: 'my-resources') {
              ... execution block ...
            }
            

            It would be useful to know whether resource1 or resource2 was acquired by the lock.

            Show
            haney David Haney added a comment - Should the fix for this include exposing the names of the acquired resource within the body? For instance if I have something like: lock (label: 'my-resources' ) { ... execution block ... } It would be useful to know whether resource1 or resource2 was acquired by the lock.
            Hide
            tjanez Tadej Janež added a comment -

            David Haney, I think the problem you mention would be solved by pursuing merging PR #20 tracked by JENKINS-31437.

            Show
            tjanez Tadej Janež added a comment - David Haney , I think the problem you mention would be solved by pursuing merging PR #20 tracked by JENKINS-31437 .
            Hide
            amuniz Antonio Muñiz added a comment -

            the problem you mention would be solved by pursuing merging PR #20

            No. That PR is only valid for freestyle jobs and it's about exposing arbitrary properties not the name of locked resources.

            Show
            amuniz Antonio Muñiz added a comment - the problem you mention would be solved by pursuing merging PR #20 No. That PR is only valid for freestyle jobs and it's about exposing arbitrary properties not the name of locked resources.
            Hide
            tjanez Tadej Janež added a comment -

            Antonio Muñiz, oh, I see. Still, it would be great if one could set properties/environement variables for each resource and they would then be accessible within the lock execution block. Or it that possible already?

            Show
            tjanez Tadej Janež added a comment - Antonio Muñiz , oh, I see. Still, it would be great if one could set properties/environement variables for each resource and they would then be accessible within the lock execution block. Or it that possible already?
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 170331 ] JNJira + In-Review [ 183859 ]
            nwegner Niels Wegner made changes -
            Link This issue is duplicated by JENKINS-38165 [ JENKINS-38165 ]
            Hide
            nwegner Niels Wegner added a comment -

            May be the locked resource might be return via given variable as requested in JENKINS-30269. This would be a similar way as if configured thru a job config.

            Show
            nwegner Niels Wegner added a comment - May be the locked resource might be return via given variable as requested in JENKINS-30269 . This would be a similar way as if configured thru a job config.
            Hide
            rainwaj Justin Rainwater added a comment - - edited

            Just curious has anyone tried nested locking as a workaround? I'm curious if that would work.
            So instead of

                lock (resources: ['resource1', 'resource2']) {
                  ... execution block ...
               }
            

            it would be:

                lock('resource1) {
                    lock('resource2) {
                        ... execution block ...
                    }
                }
            
            Show
            rainwaj Justin Rainwater added a comment - - edited Just curious has anyone tried nested locking as a workaround? I'm curious if that would work. So instead of lock (resources: [ 'resource1' , 'resource2' ]) { ... execution block ... } it would be: lock('resource1) { lock('resource2) { ... execution block ... } }
            Hide
            epsilon_b Epsilon B added a comment -

            Implemented in PR36.
            The proposed DSL syntaxe is (each parameter is optional, some are aliases such as "label" / "labels" / "capability" / "capabilities"):

            lock(resource: 'resource1 resource2', resources: ['resource1', 'resource2'], 
                label: 'label1 label2', labels: ['label1', 'label2'],
                capability: 'label1 label2', capabilities: ['label1', 'label2'],
                quantity: 2, variable: 'MY_VAR') {
                    ... execution block ...
            }
            

            Note that "quantity" is only relevant for "labels" / "capabilities" (not "resources", since they are unique and defined by names)

            Show
            epsilon_b Epsilon B added a comment - Implemented in PR36 . The proposed DSL syntaxe is (each parameter is optional, some are aliases such as "label" / "labels" / "capability" / "capabilities"): lock(resource: 'resource1 resource2', resources: ['resource1', 'resource2'], label: 'label1 label2', labels: ['label1', 'label2'], capability: 'label1 label2', capabilities: ['label1', 'label2'], quantity: 2, variable: 'MY_VAR') { ... execution block ... } Note that "quantity" is only relevant for "labels" / "capabilities" (not "resources", since they are unique and defined by names)
            hrmpw Patrick Wolf made changes -
            Assignee Antonio Muñiz [ amuniz ] CloudBees Inc. [ cloudbees ]
            hrmpw Patrick Wolf made changes -
            Assignee CloudBees Inc. [ cloudbees ]
            Hide
            fkykko Staffan Forsell added a comment -

            Will PR36 be merged/released soon? This would resource allocation of some old ssh builders that we would like to do from pipelines. I there anything I can do to help? Is there a test build that we could run?

            Show
            fkykko Staffan Forsell added a comment - Will PR36 be merged/released soon? This would resource allocation of some old ssh builders that we would like to do from pipelines. I there anything I can do to help? Is there a test build that we could run?
            amuniz Antonio Muñiz made changes -
            Assignee Antonio Muñiz [ amuniz ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Antonio Muniz
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/BackwardCompatibility.java
            src/main/java/org/jenkins/plugins/lockableresources/LockStep.java
            src/main/java/org/jenkins/plugins/lockableresources/LockStepExecution.java
            src/main/java/org/jenkins/plugins/lockableresources/LockableResource.java
            src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java
            src/main/java/org/jenkins/plugins/lockableresources/actions/LockableResourcesRootAction.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/LockRunListener.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/LockableResourcesStruct.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/QueuedContextStruct.java
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/config.jelly
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-label.html
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-quantity.html
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-resource.html
            src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/974572db98214f7b9293a303c9fd8371877250a0
            Log:
            Merge pull request #42 from amuniz/pr-26-fix

            JENKINS-34268JENKINS-34273 Lock multiple resources with specific quantity

            Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/ba48550fa5bc...974572db9821

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Antonio Muniz Path: src/main/java/org/jenkins/plugins/lockableresources/BackwardCompatibility.java src/main/java/org/jenkins/plugins/lockableresources/LockStep.java src/main/java/org/jenkins/plugins/lockableresources/LockStepExecution.java src/main/java/org/jenkins/plugins/lockableresources/LockableResource.java src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java src/main/java/org/jenkins/plugins/lockableresources/actions/LockableResourcesRootAction.java src/main/java/org/jenkins/plugins/lockableresources/queue/LockRunListener.java src/main/java/org/jenkins/plugins/lockableresources/queue/LockableResourcesStruct.java src/main/java/org/jenkins/plugins/lockableresources/queue/QueuedContextStruct.java src/main/resources/org/jenkins/plugins/lockableresources/LockStep/config.jelly src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-label.html src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-quantity.html src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-resource.html src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java http://jenkins-ci.org/commit/lockable-resources-plugin/974572db98214f7b9293a303c9fd8371877250a0 Log: Merge pull request #42 from amuniz/pr-26-fix JENKINS-34268 JENKINS-34273 Lock multiple resources with specific quantity Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/ba48550fa5bc...974572db9821
            Hide
            brendan_shape Brendan Mannix added a comment -

            Now that the PR that fixes this ticket is merged, I'd like to request a release of this plugin as soon as is appropriate.

            Show
            brendan_shape Brendan Mannix added a comment - Now that the PR that fixes this ticket is merged, I'd like to request a release of this plugin as soon as is appropriate.
            Hide
            amuniz Antonio Muñiz added a comment -

            Released as 1.11

            Show
            amuniz Antonio Muñiz added a comment - Released as 1.11
            amuniz Antonio Muñiz made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            amuniz Antonio Muñiz made changes -
            Remote Link This issue links to "PR (Web Link)" [ 15142 ]
            Hide
            fkykko Staffan Forsell added a comment -

            Antonio Muñiz
            Hi! Nice to see this merged.
            Looking at the merged content, I can't seem to find any pipeline syntax for actually knowing what resource was locked.
            Specifically the proposed syntax from PR36 above: ", variable: 'MY_VAR'" is missing.
            Our use case is that we wan't to specify a label containing node names for legacy builders that can't run the slave.jar (java7) anymore. We wan't to lock a resources (host name) from he label and then ssh to that machine any execute some build steps. This works with the FreestyleJob functionality of #Reserved resources variable name". Any thought of adding this to pipeline?

            Show
            fkykko Staffan Forsell added a comment - Antonio Muñiz Hi! Nice to see this merged. Looking at the merged content, I can't seem to find any pipeline syntax for actually knowing what resource was locked. Specifically the proposed syntax from PR36 above: ", variable: 'MY_VAR'" is missing. Our use case is that we wan't to specify a label containing node names for legacy builders that can't run the slave.jar (java7) anymore. We wan't to lock a resources (host name) from he label and then ssh to that machine any execute some build steps. This works with the FreestyleJob functionality of #Reserved resources variable name". Any thought of adding this to pipeline?
            Hide
            koomietx Karl Schulz added a comment -

            I'm also unable to successfully use the MY_VAR option. It's awesome to be able to lock multiple resources now, but can't recreate the capability of a Freestyle job without being able to query the results.

            Show
            koomietx Karl Schulz added a comment - I'm also unable to successfully use the MY_VAR option. It's awesome to be able to lock multiple resources now, but can't recreate the capability of a Freestyle job without being able to query the results.
            Hide
            glance Anton Lundin added a comment -

            I did a really ugly workaround for the missing resourceVariable :

            lock(label: 'LABEL', quantity: 1)

            { echo org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild())[0].getName() }

            That will get you which resource you locked.

            Show
            glance Anton Lundin added a comment - I did a really ugly workaround for the missing resourceVariable : lock(label: 'LABEL', quantity: 1) { echo org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild())[0].getName() } That will get you which resource you locked.
            Hide
            dafalcon Dan Falcone added a comment - - edited

            After implementing Anton Lundin's suggestion, I received the following error in my build (followed by a stacktrace):

            org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method

            To fix it, I went to Manage Jenkins > In process Script Approval, clicked Approve on the pending signature approval, and reran the build.  I had to repeat the process 4-5 times to approve all the required signatures, but it worked great after that.

            Show
            dafalcon Dan Falcone added a comment - - edited After implementing Anton Lundin's suggestion, I received the following error in my build (followed by a stacktrace): org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method To fix it, I went to Manage Jenkins > In process Script Approval, clicked Approve on the pending signature approval, and reran the build.  I had to repeat the process 4-5 times to approve all the required signatures, but it worked great after that.
            Hide
            mrozekma Michael Mrozek added a comment - - edited

            Signatures required for the above workaround:

            method org.jenkins.plugins.lockableresources.LockableResource getName
            method org.jenkins.plugins.lockableresources.LockableResourcesManager getResourcesFromBuild hudson.model.Run
            method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
            staticMethod org.jenkins.plugins.lockableresources.LockableResourcesManager get
            Show
            mrozekma Michael Mrozek added a comment - - edited Signatures required for the above workaround : method org.jenkins.plugins.lockableresources.LockableResource getName method org.jenkins.plugins.lockableresources.LockableResourcesManager getResourcesFromBuild hudson.model.Run method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild staticMethod org.jenkins.plugins.lockableresources.LockableResourcesManager get
            Hide
            mireksz Mirek Sz added a comment -

            Anton Lundin

            I did a really ugly workaround for the missing resourceVariable :

            Did you happen to find any way of getting the resource that's locked in the current lock closure? Your workaround always returns the name of first locked resource. I'm trying to run label-based locks in parallel, so I cannot simply use getResourcesFromBuild(...)[0], but must find out the exact resource locked within current scope. Any ideas?

            Show
            mireksz Mirek Sz added a comment - Anton Lundin I did a really ugly workaround for the missing resourceVariable : Did you happen to find any way of getting the resource that's locked in the current lock closure? Your workaround always returns the name of first locked resource. I'm trying to run label-based locks in parallel, so I cannot simply use getResourcesFromBuild(...) [0] , but must find out the exact resource locked within current scope. Any ideas?
            Hide
            glance Anton Lundin added a comment -

            The hack grew quite a bit over the following days, and ended up as:

            https://gist.github.com/glance-/aaa3c037757895798d4e1be5134bb843

             

            I removed the need for extensive whitelisting by writing it as a pipeline library which later could be imported into the pipeline dsl by:
            @Library("PipelineHelpers")
            import PipelineHelpers.LockableResourcesHelper

             

            We never use different locked resources in parallel so I have never needed to figure out if you can map locked resource to which block that locked it. If you ever do, please post the solution here for others to find.

            Show
            glance Anton Lundin added a comment - The hack grew quite a bit over the following days, and ended up as: https://gist.github.com/glance-/aaa3c037757895798d4e1be5134bb843   I removed the need for extensive whitelisting by writing it as a pipeline library which later could be imported into the pipeline dsl by: @Library("PipelineHelpers") import PipelineHelpers.LockableResourcesHelper   We never use different locked resources in parallel so I have never needed to figure out if you can map locked resource to which block that locked it. If you ever do, please post the solution here for others to find.
            Hide
            armb Alan Braggins added a comment - - edited

            We use locked resources in parallel, so the getResourcesFromBuild workaround didn't work for us.

            https://github.com/jenkinsci/lockable-resources-plugin/pull/50 seems to work. (I haven't tried pull/49, but it appears to be a similar approach to the same fix.)

            Show
            armb Alan Braggins added a comment - - edited We use locked resources in parallel, so the getResourcesFromBuild workaround didn't work for us. https://github.com/jenkinsci/lockable-resources-plugin/pull/50  seems to work. (I haven't tried pull/49, but it appears to be a similar approach to the same fix.)
            Hide
            chrisdickinsonnio Chris Dickinson added a comment -

            Anyone hoping to make use of locking a dynamic list of resources will probably find this feature (if it was ever implemented) doesn't work in v2.4 of the plugin.

            So, I whipped up the following helper function which should allow you to do so:

             

            def lockResources(listOfResources, closure) {
                if (listOfResources.size() > 1) {
                    lockResources(listOfResources[1..-1], { lock(listOfResources[0]) { closure() }})
                } else {
                    lock(listOfResources[0]) { closure() }
                }
            }
            

            Example usage:

            def resources = ['foo', 'bar', 'baz']
            lockResources(myResources, { 
                print('hello, world') 
            })
            

             

             

            Show
            chrisdickinsonnio Chris Dickinson added a comment - Anyone hoping to make use of locking a dynamic list of resources will probably find this feature (if it was ever implemented) doesn't work in v2.4 of the plugin. So, I whipped up the following helper function which should allow you to do so:   def lockResources(listOfResources, closure) { if (listOfResources.size() > 1) { lockResources(listOfResources[1..-1], { lock(listOfResources[0]) { closure() }}) } else { lock(listOfResources[0]) { closure() } } } Example usage: def resources = [ 'foo' , 'bar' , 'baz' ] lockResources(myResources, { print( 'hello, world' ) })    

              People

              • Assignee:
                amuniz Antonio Muñiz
                Reporter:
                amuniz Antonio Muñiz
              • Votes:
                19 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: