Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38313

External Build Log storage for Jenkins

    Details

    • Epic Name:
      External Build Log storage for Jenkins
    • Similar Issues:

      Description

      Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

      It would be great to have such solution. Top-level requirements:

      • All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
      • Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
      • External Build Log Storage has an extensible and pluggable architecture
      • At least 2 reference implementations:
        • File-system-based storage (current implementation)
        • One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
      • Jenkins Pipeline is supported out of the box
      • Old logs should be browseable after changing the storage engine (Migration or parallel engines)

      Design document:

        Attachments

          Issue Links

            Activity

            oleg_nenashev Oleg Nenashev created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Epic Child JENKINS-38381 [ 174556 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is blocked by JENKINS-38314 [ JENKINS-38314 ]
            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution
            Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution.


            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution.


            Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

            Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

            Design document:
            * https://docs.google.com/document/d/1_bquSeA_lC7zJhQoWhxlSKJAg6b8duNbyQ15zCz-e4Y/edit#
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "Design Document draft (Web Link)" [ 15820 ]
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "PoC Demo Slides (Web Link)" [ 15821 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I am not working on these stories anymore. I have been assigned to another project in October by my company

            Show
            oleg_nenashev Oleg Nenashev added a comment - I am not working on these stories anymore. I have been assigned to another project in October by my company
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-26545 [ JENKINS-26545 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-45150 [ JENKINS-45150 ]
            jamesdumay James Dumay made changes -
            Component/s pipeline [ 21692 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels pipeline
            oleg_nenashev Oleg Nenashev made changes -
            Component/s pipeline [ 21692 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            James Dumay This EPIC includes Changes in Pipeline, e.g. JENKINS-38381. Please do not remove labels/components

            Show
            oleg_nenashev Oleg Nenashev added a comment - James Dumay This EPIC includes Changes in Pipeline, e.g. JENKINS-38381 . Please do not remove labels/components
            Hide
            postman Per Östman added a comment -

            Is there any idea when this could be ready? We've started to use pipelines and immediately ran into issues with the Jenkins system incurring a massive increase in IOPS (the master has build log storage on external filer).

            Is there any short-term recommended workaround?

             

            Show
            postman Per Östman added a comment - Is there any idea when this could be ready? We've started to use pipelines and immediately ran into issues with the Jenkins system incurring a massive increase in IOPS (the master has build log storage on external filer). Is there any short-term recommended workaround?  
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Per Östman Regarding Pipeline, see JENKINS-38381. Jesse Glick made a significant progress with the log storage improvement there (with/without external storage), but the pull request has not been landed yet afaik

            Show
            oleg_nenashev Oleg Nenashev added a comment - Per Östman Regarding Pipeline, see JENKINS-38381 . Jesse Glick made a significant progress with the log storage improvement there (with/without external storage), but the pull request has not been landed yet afaik
            svanoort Sam Van Oort made changes -
            Epic Child JENKINS-38381 [ 174556 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-38381 [ JENKINS-38381 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32444 [ JENKINS-32444 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue blocks JENKINS-49327 [ JENKINS-49327 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-50291 [ JENKINS-50291 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-51778 [ 191226 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52691 [ 192576 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52692 [ 192577 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            This EPIC is in progress again

            Show
            oleg_nenashev Oleg Nenashev added a comment - This EPIC is in progress again
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "Core API: https://github.com/jenkinsci/jenkins/pull/3557/ (Web Link)" [ 21216 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52693 [ 192578 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52695 [ 192580 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52696 [ 192581 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52697 [ 192583 ]
            jglick Jesse Glick made changes -
            Epic Child JENKINS-52729 [ 192620 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-51229 [ JENKINS-51229 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52867 [ 192803 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52873 [ 192810 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52894 [ 192847 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52914 [ 192873 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52696 [ 192581 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52697 [ 192583 ]
            ctran Cuong Tran made changes -
            Rank Ranked higher
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            And... it was aborted again for some reasons out of my control...

            Show
            oleg_nenashev Oleg Nenashev added a comment - And... it was aborted again for some reasons out of my control...
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I have abandoned all tasks in this EPIC. As stated in https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ , Carlos Sanchez and Jesse Glick can take over any tasks, PRs and JEPs related to this EPIC

            Show
            oleg_nenashev Oleg Nenashev added a comment - I have abandoned all tasks in this EPIC. As stated in https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ , Carlos Sanchez and Jesse Glick can take over any tasks, PRs and JEPs related to this EPIC

              People

              • Assignee:
                Unassigned
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                13 Vote for this issue
                Watchers:
                40 Start watching this issue

                Dates

                • Created:
                  Updated: