From the best I can tell, there has been a recent-ish regression that changes the behaviour of the "Cancel build on update" option. Now any build that is queued for the job for any PR is replaced by the new build. Builds already executing do not appear to be executed. Even with FINEST logs enabled for the plugin, I don't see anything related to the removal from the queue.