We use Jenkins to manage testing of SW in isolated test chambers, the system consists of linux slaves outside the chambers which manage testing inside the chamber over ssh on dedicated links.

So we have jenkins SSH to slave (slave1a), then linux ssh to secondary slave (slave2a).

We suffer from an issue where if jenkins fails or a test is abandoned the jenkins to slave ssh link drops but the secondary ssh connection does not always drop, so we get orphaned tasks.

We solve that by detecting orphaned tasks as we start and tidy up - not ideal - but workable so far.

(in detail, the jenkins job runs run_test.sh, the start of that script looks for any run_test.sh running with a ID != our own)

We have just added a second chamber, so now there is slave1a and slave1b connected to Jenkins, both configured with a single executor.

My understanding is that Jenkins will only run one job at a time on a slave with one executor, and because it is a single freestyle job, even if there were 2 executors on a single slave it would still serialize the jobs.

What I am seeing is that a subsequent job is being started before the last has finished on a single slave, I can see quite clearly in the log of the later job it has detected remnants of the last run and issued the relevant kill commands. In the log of the running job I can see it being killed at the same second.

(in detail run_test.sh is terminated, control returns to the jenkins job which runs the subsequent steps).

My suspicion is this is to do with the checkbox that allows jobs to run concurrently over multiple slaves, one effect of this is that at the end of a run some plugins (eg Jira) wait for preceding jobs to finish at a checkpoint to ensure logs are in order.

Does that in some way weaken the serialization?

I raised this as a bug because I think the tests should not run in parallel in this configuration, I could not find an existing issue that looked the same, but I am well aware I may not know the vocab that would let me find it, sorry if that is the case.

I have attached the support bundle from jenkins.