Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62454

Gcc warning number changes when using parallel builds

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: analysis-model
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      We are seeing an inconsistent number of reported warnings from warnings-ng plugin. Building the very same commit multiple times gives different results:

      • First build: GNU C Compiler (gcc): 106 warnings
      • Second build: GNU C Compiler (gcc): 105 warnings
        11 new warnings
        12 fixed warnings
      • Third build: GNU C Compiler (gcc): 107 warnings
        6 new warnings
        4 fixed warnings

      This behavior makes warnings unreliable, and we cannot detect real new warnings. We couldn't find a workaround for that. This is why I prioritized this bug as "major": loss of function without workaround.

      The console log contains the same 500 warnings. Because of the random order of compilation I couldn't compare them side-by-side, but I sorted both console logs, and then do a diff on the sorted logs. The diff only showed line where the build number or the number of warnings reported by the plugin was present.

      The warnings in question come from a single header file that is included from multiple translation units. The compilation of each translation unit will output the (almost) same warning. These warnings will differ in the context they were included from (the include trace that says: "In file included from: ..."). My guess is that the deduplication fails to find all of them as duplicates. Depending the build order of the compilation units, it will treat different warnings as duplicates, thus giving the above mentioned "11 new, 12 fixed". This theory is also supported by the fact that when the build only uses a single thread (make -j1), the reported warnings are stable. Please note, that I don't consider building on a single thread as a workaround, that would just take too long.

      I created a Git repository with a simple project that can be used to reproduce the above mentioned issue. The repository can be found at: https://github.com/tbilles/warningtest.git

       

      Please let me know if you need any more information.

        Attachments

          Activity

          Hide
          spalani Senthilkumar Palanisamy added a comment -

          Hi,

          we are also facing same issue,Is there any workaround to fix this issue.

          Show
          spalani Senthilkumar Palanisamy added a comment - Hi, we are also facing same issue,Is there any workaround to fix this issue.
          Hide
          drulli Ulli Hafner added a comment -

          Seems that due to the option of using parallel jobs the output is non-deterministic. So the Gcc parser sometimes picks the wrong lines for a warning.

          Is it possible that the output of the parallel builds are interleaving on the console? Can you pipe the logs of the individual jobs into a warnings.log file that can be parsed sequentially? Or is there a way to separate the logs for the parallel jobs?

          Workaround: do not use parallel compiles.

          Show
          drulli Ulli Hafner added a comment - Seems that due to the option of using parallel jobs the output is non-deterministic. So the Gcc parser sometimes picks the wrong lines for a warning. Is it possible that the output of the parallel builds are interleaving on the console? Can you pipe the logs of the individual jobs into a warnings.log file that can be parsed sequentially? Or is there a way to separate the logs for the parallel jobs? Workaround: do not use parallel compiles.
          Hide
          tbilles Tibor Billes added a comment -

          Sorry for the late reply, I had trouble accessing my account. Thanks for looking into it!

          I can confirm the output lines are interleaving in the demo github project. Here are some methods that I found to prevent interleaving:

          • make: as of version 4.0 make supports the --output-sync option.
          • ninja: has builtin support to prevent interleaving
          • scons: has no builtin solution nor command line switch, but can be implemented in SConstruct scripts with something like the idBuffering method in the middle of this page: https://github.com/SCons/scons/wiki/BuildLog

          However I found instances where two consecutive warning is incorrectly parsed by Jenkins:

          src/ledpwm/PWMLedManager.cpp: In member function 'void PWMLedManager::updateLightSensorValue(int32_t)':
          src/ledpwm/PWMLedManager.cpp:84:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
            if (SYSTIME_DIFF(now, lastResult) > MS2ST(SAMPLING_TIME_IN_MS)) {
          src/ledpwm/PWMLedManager.cpp:96:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
              if(S2ST(lightSamplingTime) < SYSTIME_DIFF(now, lastStoredResult)) {
          In file included from common/obudisplay/UD/internalStateStorerThread.h:4:0,
                           from src/services.h:21,
                           from src/keyboard.cpp:16:
          common/thread/abstract_thread.h:26:15: warning: 'virtual void idata::AbstractThreadBase::initialize()' was hidden [-Woverloaded-virtual]
            virtual void initialize();
                         ^~~~~~~~~~

          This is the output from two different compilation units: PWMLedManager.cpp and keyboard.cpp . These 2 warnings are not interleaved, and should be treated separately like the following:

          src/ledpwm/PWMLedManager.cpp: In member function 'void PWMLedManager::updateLightSensorValue(int32_t)':
          src/ledpwm/PWMLedManager.cpp:84:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
            if (SYSTIME_DIFF(now, lastResult) > MS2ST(SAMPLING_TIME_IN_MS)) {
          src/ledpwm/PWMLedManager.cpp:96:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
              if(S2ST(lightSamplingTime) < SYSTIME_DIFF(now, lastStoredResult)) {
          
          
          
          
          In file included from common/obudisplay/UD/internalStateStorerThread.h:4:0,
                           from src/services.h:21,
                           from src/keyboard.cpp:16:
          common/thread/abstract_thread.h:26:15: warning: 'virtual void idata::AbstractThreadBase::initialize()' was hidden [-Woverloaded-virtual]
            virtual void initialize();
                         ^~~~~~~~~~

          But instead I see that the include trace is considered to be part of the sign comparison warning, as can be seen on the following screenshot:

          This creates inconsistent warnings when the order of compilation changes.

          In scons I can workaround this by inserting an extra empty line between compilation outputs, it makes Jenkins parse warnings separately from each other. I'm not sure if this can be achieved in other build systems.

          Show
          tbilles Tibor Billes added a comment - Sorry for the late reply, I had trouble accessing my account. Thanks for looking into it! I can confirm the output lines are interleaving in the demo github project. Here are some methods that I found to prevent interleaving: make: as of version 4.0 make supports the --output-sync option. ninja: has builtin support to prevent interleaving scons: has no builtin solution nor command line switch, but can be implemented in SConstruct scripts with something like the idBuffering method in the middle of this page: https://github.com/SCons/scons/wiki/BuildLog However I found instances where two consecutive warning is incorrectly parsed by Jenkins: src/ledpwm/PWMLedManager.cpp: In member function 'void PWMLedManager::updateLightSensorValue(int32_t)' : src/ledpwm/PWMLedManager.cpp:84:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (SYSTIME_DIFF(now, lastResult) > MS2ST(SAMPLING_TIME_IN_MS)) { src/ledpwm/PWMLedManager.cpp:96:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (S2ST(lightSamplingTime) < SYSTIME_DIFF(now, lastStoredResult)) { In file included from common/obudisplay/UD/internalStateStorerThread.h:4:0, from src/services.h:21, from src/keyboard.cpp:16: common/thread/abstract_thread.h:26:15: warning: 'virtual void idata::AbstractThreadBase::initialize()' was hidden [-Woverloaded-virtual] virtual void initialize(); ^~~~~~~~~~ This is the output from two different compilation units: PWMLedManager.cpp and keyboard.cpp . These 2 warnings are not interleaved, and should be treated separately like the following: src/ledpwm/PWMLedManager.cpp: In member function 'void PWMLedManager::updateLightSensorValue(int32_t)' : src/ledpwm/PWMLedManager.cpp:84:36: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (SYSTIME_DIFF(now, lastResult) > MS2ST(SAMPLING_TIME_IN_MS)) { src/ledpwm/PWMLedManager.cpp:96:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (S2ST(lightSamplingTime) < SYSTIME_DIFF(now, lastStoredResult)) { In file included from common/obudisplay/UD/internalStateStorerThread.h:4:0, from src/services.h:21, from src/keyboard.cpp:16: common/thread/abstract_thread.h:26:15: warning: 'virtual void idata::AbstractThreadBase::initialize()' was hidden [-Woverloaded-virtual] virtual void initialize(); ^~~~~~~~~~ But instead I see that the include trace is considered to be part of the sign comparison warning, as can be seen on the following screenshot: This creates inconsistent warnings when the order of compilation changes. In scons I can workaround this by inserting an extra empty line between compilation outputs, it makes Jenkins parse warnings separately from each other. I'm not sure if this can be achieved in other build systems.

            People

            • Assignee:
              Unassigned
              Reporter:
              tbilles Tibor Billes
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: