Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9104

Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Windows XP, Windows 7 using MSBuild or devenv.exe to build MS Visual Studio Projects
    • Similar Issues:

      Description

      I run into errors when using a customized build system which uses Visual Studio's devenv.exe under the hood to compile VisualStudio 2005 projects (with VC++ compiler). When starting two parallel builds with Jenkins (on different code base) the second job will always fail with "Fatal error C1090: PDB API call failed, error code '23' : '(" in exactly the same second the first job finishes processing. Running both jobs outside Jenkins does not produce the error.
      This has also been reported for builds executed by MSBuild on the Jenkins user mailing list [1].

      I analysed this issue thoroughly and can track the problem down to the usage of mspdbsrv.exe. This program is automatically spawned when building a VisualStudio project. All Visual Studio instances normally share one common pdb-server which shutdown itself after a idle period (standard is 10 minutes). "It ensures access to .pdb files is properly serialized in parallel builds when multiple instances of the compiler try to access the same .pdb file" [2].
      I assume that Jenkins does a clean up of its build environment when a automatically started job finishes (like as described at http://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build). I checked mspbsrv.exe with ProcessExplorer and the process indeed has a variable JENKINS_COOKIE/HUDSON_COOKIE set in its environment if started through Jenkins. Killing mspdbsrv.exe while projects are still connected will break compilation.

      Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time.


      [1] http://jenkins.361315.n4.nabble.com/MSBuild-fatal-errors-when-build-triggered-by-timer-td385181.html
      [2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/b1d1bceb-06b6-47ef-a0ea-23ea752e0c4f/

        Attachments

          Issue Links

            Activity

            Hide
            vuiletgiraffe John Doe added a comment -

            I'm occasionally getting this error with the latest versions of Jenkins and all the plugins. It started in the recent months, haven't been a problem for a year before that. The problem seems to have NOT been resolved, or possibly re-emerged.

            What can I do, is there a workaround? Sporadic build failures for no reason are super annoying.

            Show
            vuiletgiraffe John Doe added a comment - I'm occasionally getting this error with the latest versions of Jenkins and all the plugins. It started in the recent months, haven't been a problem for a year before that. The problem seems to have NOT been resolved, or possibly re-emerged. What can I do, is there a workaround? Sporadic build failures for no reason are super annoying.
            Hide
            billhoo Bill Hoo added a comment -

            Same error with latest Jenkins ver. 2.150.3

            The error is aways occured when running two jobs concurrently on the same agent with VS2015:
            fatal error C1090: PDB API

            Show
            billhoo Bill Hoo added a comment - Same error with latest Jenkins ver. 2.150.3 The error is aways occured when running two jobs concurrently on the same agent with VS2015: fatal error C1090: PDB API
            Hide
            vuiletgiraffe John Doe added a comment -

            Bill Hoo, thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds.

            Show
            vuiletgiraffe John Doe added a comment - Bill Hoo , thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds.
            Hide
            billhoo Bill Hoo added a comment -

            John Doe, totaly the same, we have many different jobs which use MSVC14 as toolchain, but now we can only perform one build at a time, its a huge waste of mashine resources ;(

            Hope it can be truly solved.

            Show
            billhoo Bill Hoo added a comment - John Doe , totaly the same, we have many different jobs which use MSVC14 as toolchain, but now we can only perform one build at a time, its a huge waste of mashine resources ;( Hope it can be truly solved.
            Hide
            ext3h Andreas Ringlstetter added a comment - - edited

            Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique:

            _MSPDBSRV_ENDPOINT_=<UUID>
            TMP=<Unique Tempdir>
            TEMP=$TMP
            TMPDIR=$TMP

            Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since.

            The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured.

            Show
            ext3h Andreas Ringlstetter added a comment - - edited Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique: _MSPDBSRV_ENDPOINT_=<UUID> TMP=<Unique Tempdir> TEMP=$TMP TMPDIR=$TMP Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since. The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured.

              People

              • Assignee:
                danielweber Daniel Weber
                Reporter:
                gordin Christoph VogtlÃĪnder
              • Votes:
                71 Vote for this issue
                Watchers:
                92 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: