Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26545

Persist build(s) log(s) in MySQL (or similar)

    Details

    • Similar Issues:

      Description

      Persisting build(s) log(s) in MySQL (or similar) will make Jenkins updates much easier to manage.

      In fact, when build logs change their format we sometimes end up with a broken system which it is often unable to read logs of old builds. Software changes even break stored builds info from time to time

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            for more than 200 maven modules per single build run per job.

            Don't use the Maven job type. It has some well known, severe performance problems. Problem solved.

            Show
            danielbeck Daniel Beck added a comment - for more than 200 maven modules per single build run per job. Don't use the Maven job type. It has some well known, severe performance problems. Problem solved.
            Hide
            dserodio Daniel Serodio added a comment -

            Thanks for the info Daniel Beck, is this documented somewhere?

            Show
            dserodio Daniel Serodio added a comment - Thanks for the info Daniel Beck , is this documented somewhere?
            Hide
            danielbeck Daniel Beck added a comment -

            More learned through experience by a number of community members. I doubt it's explicitly documented somewhere. From my limited experience with Maven I'd also consider 200 modules in a single build to be rather unusual, so the performance issues you experience likely aren't experienced by most users. When loading a single build involves parsing hundreds of XML files (compared to one in the case of freestyle), performance problems are to be expected.

            Note that there are other issues with the job type, which were written up by Stephen Connolly, a contributor to Jenkins and Maven, here:
            http://javaadventure.blogspot.de/2013/11/jenkins-maven-job-type-considered-evil.html

            Show
            danielbeck Daniel Beck added a comment - More learned through experience by a number of community members. I doubt it's explicitly documented somewhere. From my limited experience with Maven I'd also consider 200 modules in a single build to be rather unusual, so the performance issues you experience likely aren't experienced by most users. When loading a single build involves parsing hundreds of XML files (compared to one in the case of freestyle), performance problems are to be expected. Note that there are other issues with the job type, which were written up by Stephen Connolly , a contributor to Jenkins and Maven, here: http://javaadventure.blogspot.de/2013/11/jenkins-maven-job-type-considered-evil.html
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Part of the issue in start-up performance for the evil job type is the mutability of build results issue...

            You can see this in the other side-effect issues: JENKINS-20731, JENKINS-25075

            Basically, back from when this job type was truly completely and utterly evil, it would run each module as a separate "job" in parallel and basically re-implement the whole Maven Reactor with a bunch of hacks... You can still enable this mode of using the evil job type, but it is at least no longer the default.

            So what you have is that you can re-run a single module only... and the build result for an evil job is the aggregate build result of the most recent build of all the modules...

            Thus if you get a foo-project build #5 broken due to a flakey test in foo-manchu-module, you just re-trigger the failing module and presto! your foo-project build #5 is now overall fixed because the foo-manchu-module build #6 was a success (note that the original build of foo-project #5 resulted in foo-manchu-module build #5)

            So when Jenkins loads an evil job build record, not only does it have to parse all latest build child module descriptors, but it has to re-evaluate all the latest build results of all modules in the current build...

            And then the weather column wants to evaluate the build stability... so it goes asking for the build result of the previous 4 builds...

            So actually I would contend (and James Nord would agree) that the root cause of the performance issues is actually the mutability of the build result in the evil job type... fix that and a lot of the evil job's performance issues can be resolved... there are other performance issues with the evil job... and it will still be evil

            (FYI: I call it evil from my PoV as a committer to the Apache Maven project because it is a job type that expressly goes completely against the core principles of Maven... one of which being that the build can be completely reproducible using an identical environment variables, user permissions pom and command-line... the evil one doesn't do this because it modifies the effective pom in ways you cannot see or deterministically determine without injecting some additional inspection code into your Maven binaries)

            Show
            stephenconnolly Stephen Connolly added a comment - Part of the issue in start-up performance for the evil job type is the mutability of build results issue... You can see this in the other side-effect issues: JENKINS-20731 , JENKINS-25075 Basically, back from when this job type was truly completely and utterly evil, it would run each module as a separate "job" in parallel and basically re-implement the whole Maven Reactor with a bunch of hacks... You can still enable this mode of using the evil job type, but it is at least no longer the default. So what you have is that you can re-run a single module only... and the build result for an evil job is the aggregate build result of the most recent build of all the modules... Thus if you get a foo-project build #5 broken due to a flakey test in foo-manchu-module, you just re-trigger the failing module and presto! your foo-project build #5 is now overall fixed because the foo-manchu-module build #6 was a success (note that the original build of foo-project #5 resulted in foo-manchu-module build #5) So when Jenkins loads an evil job build record, not only does it have to parse all latest build child module descriptors, but it has to re-evaluate all the latest build results of all modules in the current build... And then the weather column wants to evaluate the build stability... so it goes asking for the build result of the previous 4 builds... So actually I would contend (and James Nord would agree) that the root cause of the performance issues is actually the mutability of the build result in the evil job type... fix that and a lot of the evil job's performance issues can be resolved... there are other performance issues with the evil job... and it will still be evil (FYI: I call it evil from my PoV as a committer to the Apache Maven project because it is a job type that expressly goes completely against the core principles of Maven... one of which being that the build can be completely reproducible using an identical environment variables, user permissions pom and command-line... the evil one doesn't do this because it modifies the effective pom in ways you cannot see or deterministically determine without injecting some additional inspection code into your Maven binaries)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            FTR there was some design work and prototyping in JENKINS-38313

            Show
            oleg_nenashev Oleg Nenashev added a comment - FTR there was some design work and prototyping in JENKINS-38313

              People

              • Assignee:
                Unassigned
                Reporter:
                czerny Libero Scarcelli
              • Votes:
                2 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: