When hasGitRepo is false, retrieveChanges() tries firstly to Clone repository and then fetch according to available list of repositories.
But when it clones it picks first repository name and injects ALL branches fetch https://github.com/jenkinsci/git-client-plugin/blob/2b217bd90a93462112176febbb5f30e05141c674/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L704-L712
So, when refspecs is empty - it fetch all branches (git default), when it null it fetches all branches.
The workaround is to set some refspec from branch i.e. https://github.com/jenkinsci/git-plugin/blob/a7ad6eb38a00397c4a656b53d99bfbad8ea3b2cc/src/main/java/jenkins/plugins/git/GitSCMSourceDefaults.java#L108-L115
But, later it will be fetched anyway.
Don't fetch anything during clone operation (useful part is clean, fetch etc). OR pass repositories to CloneComand and remove separate fetch iteration .