This comment from JENKINS-23212 is still applicable "Was able to reproduce the described behavior using JNLP connection with a long post-boot delay and a short idle time to disconnect. The two features are colliding. Solutions: decrease post-boot delay to bare minimum needed, use non-JNLP slave agent connection, or increase the idle time to disconnect."

However, I cannot find this documented anywhere and it should be on the plugin page – or fixed.  Idle timeout should not start counting down until after the connect is complete.