... before we know we'll invoke ssh, directly or indirectly.

This happened in the context of mercurial check out from ssh.

Either that or allow people to disable host key verifications altogether.