Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9104

Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Environment:
      Windows XP, Windows 7 using MSBuild or devenv.exe to build MS Visual Studio Projects
    • Similar Issues:

      Description

      I run into errors when using a customized build system which uses Visual Studio's devenv.exe under the hood to compile VisualStudio 2005 projects (with VC++ compiler). When starting two parallel builds with Jenkins (on different code base) the second job will always fail with "Fatal error C1090: PDB API call failed, error code '23' : '(" in exactly the same second the first job finishes processing. Running both jobs outside Jenkins does not produce the error.
      This has also been reported for builds executed by MSBuild on the Jenkins user mailing list [1].

      I analysed this issue thoroughly and can track the problem down to the usage of mspdbsrv.exe. This program is automatically spawned when building a VisualStudio project. All Visual Studio instances normally share one common pdb-server which shutdown itself after a idle period (standard is 10 minutes). "It ensures access to .pdb files is properly serialized in parallel builds when multiple instances of the compiler try to access the same .pdb file" [2].
      I assume that Jenkins does a clean up of its build environment when a automatically started job finishes (like as described at http://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build). I checked mspbsrv.exe with ProcessExplorer and the process indeed has a variable JENKINS_COOKIE/HUDSON_COOKIE set in its environment if started through Jenkins. Killing mspdbsrv.exe while projects are still connected will break compilation.

      Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time.


      [1] http://jenkins.361315.n4.nabble.com/MSBuild-fatal-errors-when-build-triggered-by-timer-td385181.html
      [2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/b1d1bceb-06b6-47ef-a0ea-23ea752e0c4f/

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            *sigh*

            1.26 is tagged in GitHub but no artifacts are uploaded. Looks like a failed release. Sorry about that.

            Note that MSBuild Plugin is almost certainly not currently maintained, as Gregory stopped working on his plugins, so if someone here wants to take over (Daniel Weber perhaps?) that should be possible.

            Show
            danielbeck Daniel Beck added a comment - *sigh* 1.26 is tagged in GitHub but no artifacts are uploaded. Looks like a failed release. Sorry about that. Note that MSBuild Plugin is almost certainly not currently maintained, as Gregory stopped working on his plugins, so if someone here wants to take over ( Daniel Weber perhaps?) that should be possible.
            Hide
            teljj001 James Telfer added a comment -

            Daniel Beck no need to apologise, I appreciate you looking at it.

            Show
            teljj001 James Telfer added a comment - Daniel Beck no need to apologise, I appreciate you looking at it.
            Hide
            josch Johannes Schmieder added a comment -

            As a workaround I have created a Jenkins Job that executes a Windows batch command on the jenkins node where Visual Studio is installed.
            The jenkins job triggers the batch command once a day and works in my environment for several years now.
            The batch command looks like this:

            set MSPDBSRV_EXE=mspdbsrv.exe
            set MSPDBSRV_PATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE
            
            set PATH=%MSPDBSRV_PATH%;%PATH%
            set ORIG_BUILD_ID=%BUILD_ID%
            set BUILD_ID=DoNotKillMe
            
            echo stop mspdbsrv.exe
            %MSPDBSRV_EXE% -stop
            
            echo wait 7 sec
            %windir%\system32\ping.exe -n 7 localhost> nul
            
            echo restart mspdbsrv.exe with a shutdowntime of 25 hours
            start /b %MSPDBSRV_EXE% -start -spawn -shutdowntime 90000
            
            set BUILD_ID=%ORIG_BUILD_ID%
            set ORIG_BUILD_ID=
            exit 0
            

            What the batch command does is:
            stop the mspdbsrv.exe to free up resources
            start mspdbsrv.exe with BUILD_ID=DoNotKillMe and a shutdowntime of 25 hours, that leaks the mspdbsrv process without getting killed and it runs for 25 hours so that other build jobs can use the already running process

            What you maybe have to do is to change the Path to mspdbsrv -> set MSPDBSRV_PATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE

            Show
            josch Johannes Schmieder added a comment - As a workaround I have created a Jenkins Job that executes a Windows batch command on the jenkins node where Visual Studio is installed. The jenkins job triggers the batch command once a day and works in my environment for several years now. The batch command looks like this: set MSPDBSRV_EXE=mspdbsrv.exe set MSPDBSRV_PATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE set PATH=%MSPDBSRV_PATH%;%PATH% set ORIG_BUILD_ID=%BUILD_ID% set BUILD_ID=DoNotKillMe echo stop mspdbsrv.exe %MSPDBSRV_EXE% -stop echo wait 7 sec %windir%\system32\ping.exe -n 7 localhost> nul echo restart mspdbsrv.exe with a shutdowntime of 25 hours start /b %MSPDBSRV_EXE% -start -spawn -shutdowntime 90000 set BUILD_ID=%ORIG_BUILD_ID% set ORIG_BUILD_ID= exit 0 What the batch command does is: stop the mspdbsrv.exe to free up resources start mspdbsrv.exe with BUILD_ID=DoNotKillMe and a shutdowntime of 25 hours, that leaks the mspdbsrv process without getting killed and it runs for 25 hours so that other build jobs can use the already running process What you maybe have to do is to change the Path to mspdbsrv -> set MSPDBSRV_PATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE
            Hide
            mbrock Michael Brock added a comment -

            Updating the msbuild plugin won't work in our situation. We run into this issue, but we don't have the plugin installed. Rather the issue comes for us in the Final Builder scripts we run via Jenkins that call msbuild.

            Show
            mbrock Michael Brock added a comment - Updating the msbuild plugin won't work in our situation. We run into this issue, but we don't have the plugin installed. Rather the issue comes for us in the Final Builder scripts we run via Jenkins that call msbuild.
            Hide
            danielbeck Daniel Beck added a comment -

            Then install it. MSBuild will veto all mspdbsrv killing.

            Show
            danielbeck Daniel Beck added a comment - Then install it. MSBuild will veto all mspdbsrv killing.

              People

              • Assignee:
                danielweber Daniel Weber
                Reporter:
                gordin Christoph Vogtländer
              • Votes:
                62 Vote for this issue
                Watchers:
                79 Start watching this issue

                Dates

                • Created:
                  Updated: