Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36559

File pattern (e.g. archive) field validation can run forever

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Every few days, the Jenkins UI hangs and becomes totally unresponsive, i.e. "Jenkins is not picking up HTTP requests, so all attempts to connect hang and no page is ever rendered."
      It doens't seem to be connected with a certain job getting started or anything.

      I couldn't get a thread dump from the UI, nor from jstack, only from jstack -F. I could get the jmap -heap, but not the full heap dump. Both outputs are attached. If I understand that right, we are in a GC spiral.

      Is there anything I can do to help debugging this?

        Attachments

        1. config.xml
          1 kB
        2. jconsole.png
          jconsole.png
          45 kB
        3. jhat.PNG
          jhat.PNG
          225 kB
        4. jmap.out
          1 kB
        5. jstack.out
          170 kB

          Issue Links

            Activity

            Hide
            olenz Olaf Lenz added a comment -

            I will test that. Now that you mention it, I have seen hundreds of instances of this message in the jenkins log:

            skipping symbolic link /workspaces/socvm459/coverity/ws-r8.0-coverity/opt/93000/src/hp83000/hp83000/hp83000/hp83000/hp83000/test/devices/offline/recursiveLink1/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLink2/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2 -- too many levels of symbolic links.
            
            Show
            olenz Olaf Lenz added a comment - I will test that. Now that you mention it, I have seen hundreds of instances of this message in the jenkins log: skipping symbolic link /workspaces/socvm459/coverity/ws-r8.0-coverity/opt/93000/src/hp83000/hp83000/hp83000/hp83000/hp83000/test/devices/offline/recursiveLink1/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLink2/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2 -- too many levels of symbolic links.
            Hide
            olenz Olaf Lenz added a comment -

            Setting the property on the command line doesn't help, even when setting it to 100.
            I also tested it with Jenkins 2.7.1, but it has the same problem.

            Show
            olenz Olaf Lenz added a comment - Setting the property on the command line doesn't help, even when setting it to 100. I also tested it with Jenkins 2.7.1, but it has the same problem.
            Hide
            olenz Olaf Lenz added a comment - - edited

            I have created a simple job that reproduces the behaviour and attached the config.xml. It creates a subdir that contains a recursive link, plus 1000 empty files, and archives the files. This leads to a memory overflow within a few minutes, apparently independent of hudson.FilePath.VALIDATE_ANT_FILE_MASK_BOUND.

            It looks as though it is crucial that the directory also contains a lot of files. My diagnosis would be that Jenkins tries to create a catalogue of files recursively. If there are very few files in the dir, the memory won't overflow as the limit of scanDir-calls is reached soon, but if there are files, each scanDir call is large.

            Show
            olenz Olaf Lenz added a comment - - edited I have created a simple job that reproduces the behaviour and attached the config.xml. It creates a subdir that contains a recursive link, plus 1000 empty files, and archives the files. This leads to a memory overflow within a few minutes, apparently independent of hudson.FilePath.VALIDATE_ANT_FILE_MASK_BOUND. It looks as though it is crucial that the directory also contains a lot of files. My diagnosis would be that Jenkins tries to create a catalogue of files recursively. If there are very few files in the dir, the memory won't overflow as the limit of scanDir-calls is reached soon, but if there are files, each scanDir call is large.
            Hide
            olenz Olaf Lenz added a comment -

            Note that the config that I have attached doesn't exactly reproduce the issue, as this only crashes jenkins when building the job. The actual problem is worse insofar as it crashes jenkins when trying to configure the job.

            Show
            olenz Olaf Lenz added a comment - Note that the config that I have attached doesn't exactly reproduce the issue, as this only crashes jenkins when building the job. The actual problem is worse insofar as it crashes jenkins when trying to configure the job.
            Hide
            danielbeck Daniel Beck added a comment -

            Should be enough to remove the archiving, build once to get the files, then config to get the archiving again – pattern validation will then result in OOM

            Show
            danielbeck Daniel Beck added a comment - Should be enough to remove the archiving, build once to get the files, then config to get the archiving again – pattern validation will then result in OOM

              People

              • Assignee:
                danielbeck Daniel Beck
                Reporter:
                olenz Olaf Lenz
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: