We have a chain of four multi-configuration / matrix jobs. We were we seeing runs in which all variations originally passed randomly revert to showing one or more of the variations in the 'not run' state. For example, these four green balls would revert to a mix of green and grey balls after some number of hours (see snap-1.png)

This caused downstream jobs to then fail because they couldn't find the build artifacts any more. Re-running the full sequence solved it, but it meant a lot of re-building and once in a while we couldn't get through the full sequence of jobs before something broke.

The problem appears to have been the matrix setting of 'label' here show in snap-2.png. Previously I had renamed 'label' to 'LABEL' because on Windows bash automatically uppercases these variables regardless of the setting, but Mac OS and Linux don't. After scanning configuration files I noticed an internal setting where 'label' was still lower case. Reverting to lower-case label seems to have solved the instability issues. One hypothesis is that the jobs were ok until something got unloaded from memory and then broke upon reload.

A case-sensitivity issue like that seems a little far-fetched, so it may be a rename issue and if I had originally created the job using all upper-case that it would be ok. Or perhaps even it's an incorrect case-insensitive compare somewhere and if I had renamed 'label' to 'LABELX' it would have fixed it.

Hopefully this helps track down the issue. Anyone looking into this issue is welcome to email me for more information.