I have a diamond build setup that runs across several machines. The first job runs on the parent node, then kicks off jobs on multiple child nodes, as well as on itself...then a join build is done on a grandchild node.
I have a list of predefined parameters, one of which recently failed on me unexpectedly:
TIMESTAMP=${BUILD_ID}_${NODE_NAME}

I expect these predefined parameters to evaluate on the parent node, where they are configured (this is how they are evaluated on the pass-off to its children). However, I just witnessed a grandchild build (during the join action) evaluate these environment variables from a child node (The build_id and node_names were changed to that of a child job). I'm guessing it was evaluated from the last job/node to finish, but that wouldn't exactly explain all the other times this has worked correctly.