Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48420

Provide lightweight checkout capability for Subversion

    Details

    • Similar Issues:
    • Released As:
      2.12.0

      Description

      JENKINS-33273 introduced a lightweight checkout capability to download a Jenkinsfile from source control without checkout out the entire repository, but it appears this hasn't been implemented for Subversion: "Lightweight checkout support not available, falling back to full checkout."

        Attachments

          Issue Links

            Activity

            mgrochowalski Matt Grochowalski created issue -
            Hide
            andre4s_m Andreas Magnusson added a comment -

            I have a working fork on GitHub with this functionality, what it the best way to get the community to validate it? Should I assign the JIRA to me and issue a PR, or something else?

            Show
            andre4s_m Andreas Magnusson added a comment - I have a working fork on GitHub with this functionality, what it the best way to get the community to validate it? Should I assign the JIRA to me and issue a PR, or something else?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Andreas Magnusson yes, it would be the best approach

            Show
            oleg_nenashev Oleg Nenashev added a comment - Andreas Magnusson yes, it would be the best approach
            Hide
            ilmarinen Mathias Helminger added a comment -

            Andreas Magnusson: any news on this? I'd be interested, too. (Link to the hinted GitHub would also be a good start)

            Show
            ilmarinen Mathias Helminger added a comment - Andreas Magnusson : any news on this? I'd be interested, too. (Link to the hinted GitHub would also be a good start)
            ilmarinen Mathias Helminger made changes -
            Field Original Value New Value
            Link This issue is blocking JENKINS-47299 [ JENKINS-47299 ]
            Hide
            andre4s_m Andreas Magnusson added a comment -

            Mathias Helminger: Life interfered, but here is the GitHub project: https://github.com/Zune01/subversion-plugin/branches

            Show
            andre4s_m Andreas Magnusson added a comment - Mathias Helminger : Life interfered, but here is the GitHub project: https://github.com/Zune01/subversion-plugin/branches
            Hide
            jmalopoy Jesús Malo Poyatos added a comment -

            Andreas Magnusson: Thanks for the link to the GitHub project. Did you send the PR?

            Show
            jmalopoy Jesús Malo Poyatos added a comment - Andreas Magnusson : Thanks for the link to the GitHub project. Did you send the PR?
            jmalopoy Jesús Malo Poyatos made changes -
            Link This issue depends on JENKINS-33273 [ JENKINS-33273 ]
            jmalopoy Jesús Malo Poyatos made changes -
            Remote Link This issue links to "Andreas Magnusson contribution (Web Link)" [ 20263 ]
            kpop kpop made changes -
            Link This issue blocks JENKINS-50490 [ JENKINS-50490 ]
            kpop kpop made changes -
            Link This issue blocks JENKINS-50490 [ JENKINS-50490 ]
            kpop kpop made changes -
            Link This issue is related to JENKINS-50490 [ JENKINS-50490 ]
            Hide
            tflanigan Thomas Flanigan added a comment -

            I am currently trying to migrate some of my organization's existing jobs to use Jenkinsfiles instead. We'd like to store the Jenkinsfiles in some of our SVN paths since new branches are regularly created which need jobs created automatically, so managing jobs with a multibranch pipeline job scanning for Jenkinsfiles is a great solution for us, but I am currently running into this SVN limitation.

            Although we allow plenty of space on our agents for checkouts, our repos are large enough that we risk running out of space on our master Jenkins machine if a fully recursive checkout is done there simply to read the Jenkinsfile. Will this feature be available relatively soon?

            Show
            tflanigan Thomas Flanigan added a comment - I am currently trying to migrate some of my organization's existing jobs to use Jenkinsfiles instead. We'd like to store the Jenkinsfiles in some of our SVN paths since new branches are regularly created which need jobs created automatically, so managing jobs with a multibranch pipeline job scanning for Jenkinsfiles is a great solution for us, but I am currently running into this SVN limitation. Although we allow plenty of space on our agents for checkouts, our repos are large enough that we risk running out of space on our master Jenkins machine if a fully recursive checkout is done there simply to read the Jenkinsfile. Will this feature be available relatively soon?
            Hide
            elmuerte Michiel Hendriks added a comment -

            This commit contains the fixes for lightweight checkouts, right? https://github.com/Zune01/subversion-plugin/commit/100d8fbefb3d68733b26439daf94cded6cd9fbb8

            (We really need this fix as we are going to run out of inodes)

            Show
            elmuerte Michiel Hendriks added a comment - This commit contains the fixes for lightweight checkouts, right? https://github.com/Zune01/subversion-plugin/commit/100d8fbefb3d68733b26439daf94cded6cd9fbb8 (We really need this fix as we are going to run out of inodes)
            Hide
            elmuerte Michiel Hendriks added a comment -

            I've tested the changes against the current plugin versions and it looks like everything works correctly. I've issued a pull request for these changes: https://github.com/jenkinsci/subversion-plugin/pull/221

            Show
            elmuerte Michiel Hendriks added a comment - I've tested the changes against the current plugin versions and it looks like everything works correctly. I've issued a pull request for these changes: https://github.com/jenkinsci/subversion-plugin/pull/221
            Hide
            elmuerte Michiel Hendriks added a comment -

            Looks like one of the added test cases fails on the buildserver (work for me locally): SubversionSCMFileSystemTest#lastModified_Smokes

            Not sure what the added value of this test is as it mostly relies on svnkit's "lastModified" response. But there is clearly a timing issue going on. i.e. buildserver not being fast enough for this test to fall within the given time window.

             

            Show
            elmuerte Michiel Hendriks added a comment - Looks like one of the added test cases fails on the buildserver (work for me locally): SubversionSCMFileSystemTest#lastModified_Smokes Not sure what the added value of this test is as it mostly relies on svnkit's "lastModified" response. But there is clearly a timing issue going on. i.e. buildserver not being fast enough for this test to fall within the given time window.  
            jsmith45 Joe Smith made changes -
            Link This issue is duplicated by JENKINS-42957 [ JENKINS-42957 ]
            Hide
            elmuerte Michiel Hendriks added a comment -

            Kevin Cathcart created a new pull request with fixed unit test: https://github.com/jenkinsci/subversion-plugin/pull/223

            It passes all tests. (So I've closed the #221 PR in favor of the #223)

            Show
            elmuerte Michiel Hendriks added a comment - Kevin Cathcart created a new pull request with fixed unit test: https://github.com/jenkinsci/subversion-plugin/pull/223 It passes all tests. (So I've closed the #221 PR in favor of the #223)
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            Assignee Ivan Fernandez Calvo [ ifernandezcalvo ]
            Resolution Fixed [ 1 ]
            Released As 2.12.0
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            Hide
            ziggystar Thomasero Vulturera added a comment - - edited

            I just did an update to the new version. It breaks my jobs in a Pipeline-Multibranch job because the sparse checkout apparently cannot follow svn:externals. There is also no option within the Pipeline-Multibranch plugin to disable sparse checkout, so I had to downgrade the plugin.

            I have 50+ permanent branches, and I want all of them to use the same Jenkinsfile. So I put the script into a separate folder and import that folder into my branch folder and have a symlink into the external:

            shared/
              Jenkinsfile
            <branch>/ (with shared/ as svn:external under shared/)
              Jenkinsfile -> shared/Jenkinsfile

            I could get rid of this setup with file externals (don't remember why this didn't work). But still, the Subversion plugin in sparse mode needs to follow externals for my usecase to work.

             

            Show
            ziggystar Thomasero Vulturera added a comment - - edited I just did an update to the new version. It breaks my jobs in a Pipeline-Multibranch job because the sparse checkout apparently cannot follow svn:externals. There is also no option within the Pipeline-Multibranch plugin to disable sparse checkout, so I had to downgrade the plugin. I have 50+ permanent branches, and I want all of them to use the same Jenkinsfile. So I put the script into a separate folder and import that folder into my branch folder and have a symlink into the external: shared/   Jenkinsfile <branch>/ (with shared/ as svn:external under shared/)   Jenkinsfile -> shared/Jenkinsfile I could get rid of this setup with file externals (don't remember why this didn't work). But still, the Subversion plugin in sparse mode needs to follow externals for my usecase to work.  
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I think it could have easy workaround with a property here https://github.com/KevinCathcart/subversion-plugin/blob/d70e95d6cfe2ea660449417fa54abdeec51b203e/src/main/java/jenkins/scm/impl/subversion/SubversionSCMFileSystem.java#L74-L82 I will make the PR and release a minor version with the workaround

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I think it could have easy workaround with a property here https://github.com/KevinCathcart/subversion-plugin/blob/d70e95d6cfe2ea660449417fa54abdeec51b203e/src/main/java/jenkins/scm/impl/subversion/SubversionSCMFileSystem.java#L74-L82  I will make the PR and release a minor version with the workaround
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Remote Link This issue links to "PR - property to disable lightweight checkout (Web Link)" [ 21822 ]
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            I just releases 2.12.1, it contains a new property to disable lightweight checkout -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=<true/false> for SCM API 2.0 so it would use the default implementation.

            By default the SCM API 2.0 is used (the new implementation here), set the property means to use the implementation on <2.12.0 versions

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited I just releases 2.12.1, it contains a new property to disable lightweight checkout -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=<true/false> for SCM API 2.0 so it would use the default implementation. By default the SCM API 2.0 is used (the new implementation here), set the property means to use the implementation on <2.12.0 versions
            Hide
            psteiner Philip Steiner added a comment - - edited

            After upgrading SVNPlugin from v2.11.1 to v2.12.1, pipeline jobs on my Jenkins v2.107.3 master started throwing an SVN exception (see example below).

            The pipeline jobs are defined as "Pipeline script from SCM" with "Lightweight Checkout" selected (default when SVN is added to job).

            Check-out Strategy is "Always check out a fresh copy". Quiet checkout is selected.

            If I deselect "Lightweight Checkout" in the job configuration, then the pipeline checkout step succeeds.

            org.tmatesoft.svn.core.SVNException: svn: E160013: '/svn/gpd/!svn/bc/59394/CI/dev/trunk/CI/GPD60/AE/jenkinsfile' path not found: 404 Not Found (https://svnyvr1.mda.ca)
            {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)}}
            {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:920)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doGet(DAVConnection.java:335)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:289)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:251)}}
            {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:82)}}
            Caused: java.io.IOException: failed to fetch file: CI/GPD60/AE/jenkinsfile
            {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:85)}}
            {{ at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)}}
            {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:110)}}
            {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)}}
            {{ at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)}}
            {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
            {{ at hudson.model.Executor.run(Executor.java:429)}}
            Finished: FAILURE

             

            Show
            psteiner Philip Steiner added a comment - - edited After upgrading SVNPlugin from v2.11.1 to v2.12.1, pipeline jobs on my Jenkins v2.107.3 master started throwing an SVN exception (see example below). The pipeline jobs are defined as "Pipeline script from SCM" with "Lightweight Checkout" selected (default when SVN is added to job). Check-out Strategy is "Always check out a fresh copy". Quiet checkout is selected. If I deselect "Lightweight Checkout" in the job configuration, then the pipeline checkout step succeeds. org.tmatesoft.svn.core.SVNException: svn: E160013: '/svn/gpd/!svn/bc/59394/CI/dev/trunk/CI/GPD60/AE/jenkinsfile' path not found: 404 Not Found ( https://svnyvr1.mda.ca ) {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)}} {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)}} {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)}} {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:920)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doGet(DAVConnection.java:335)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:289)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:251)}} {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:82)}} Caused: java.io.IOException: failed to fetch file: CI/GPD60/AE/jenkinsfile {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:85)}} {{ at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)}} {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:110)}} {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)}} {{ at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)}} {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}} {{ at hudson.model.Executor.run(Executor.java:429)}} Finished: FAILURE  
            Hide
            jpraet Jimmy Praet added a comment -

            Since upgrading to v2.12.x, our jenkins pipeline builds always showed "No changes".

            I configured -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true as a workaround.

            Show
            jpraet Jimmy Praet added a comment - Since upgrading to v2.12.x, our jenkins pipeline builds always showed "No changes". I configured -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true as a workaround.
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            Philip Steiner Is the jenkinsfile in an external? if so you will need to set -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true property.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - Philip Steiner Is the jenkinsfile in an external? if so you will need to set -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true property.
            Hide
            psteiner Philip Steiner added a comment -

            Ivan Fernandez Calvo - I don't think there are any externals. The files all reside in the same subdirectory of the repository, and as far as I know, there are no external references (I am not an SVN expert).

            I have worked around the error for now by deselecting "Lightweight checkout" on all jobs that use these pipeline files.

            Forgive this question, I am just an end-user on this Jenkins master, with no admin access - where would I ask the sysadmin to set the property you suggested? Is that in a configuration file for the Jenkins master?

            Show
            psteiner Philip Steiner added a comment - Ivan Fernandez Calvo - I don't think there are any externals. The files all reside in the same subdirectory of the repository, and as far as I know, there are no external references (I am not an SVN expert). I have worked around the error for now by deselecting "Lightweight checkout" on all jobs that use these pipeline files. Forgive this question, I am just an end-user on this Jenkins master, with no admin access - where would I ask the sysadmin to set the property you suggested? Is that in a configuration file for the Jenkins master?
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            Philip Steiner you have detailed steps to make it on this KB

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - Philip Steiner you have detailed steps to make it on this KB
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Link This issue is related to JENKINS-53919 [ JENKINS-53919 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Link This issue is related to JENKINS-53892 [ JENKINS-53892 ]
            Hide
            psteiner Philip Steiner added a comment - - edited

            Thanks Ivan Fernandez Calvo I will forward the KB to my sys admin

            Show
            psteiner Philip Steiner added a comment - - edited Thanks Ivan Fernandez Calvo I will forward the KB to my sys admin
            Hide
            rockowwc Robert Orrock added a comment -

            Philip Steiner did the java argument fix your issue?

            Show
            rockowwc Robert Orrock added a comment - Philip Steiner did the java argument fix your issue?
            Hide
            psteiner Philip Steiner added a comment -

            Robert Orrock Yes it did! 

            Show
            psteiner Philip Steiner added a comment - Robert Orrock Yes it did! 
            Hide
            richard_a Arnaud Richard added a comment -

            I'm having the same issue than Philip Steiner. I'm sure I don't have externals. What is the fix ?

            Show
            richard_a Arnaud Richard added a comment - I'm having the same issue than Philip Steiner. I'm sure I don't have externals. What is the fix ?

              People

              • Assignee:
                ifernandezcalvo Ivan Fernandez Calvo
                Reporter:
                mgrochowalski Matt Grochowalski
              • Votes:
                14 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: