When hasGitRepo is false, retrieveChanges() tries firstly to Clone repository and then fetch according to available list of repositories.
https://github.com/jenkinsci/git-plugin/blob/e9d15cd3a390f1d1849b9d6cbee707bc85213ab0/src/main/java/hudson/plugins/git/GitSCM.java#L1104-L1110
But when it clones it picks first repository name and injects ALL branches fetch https://github.com/jenkinsci/git-client-plugin/blob/2b217bd90a93462112176febbb5f30e05141c674/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L704-L712
So, when refspecs is empty - it fetch all branches (git default), when it null it fetches all branches.

The workaround is to set some refspec from branch i.e. https://github.com/jenkinsci/git-plugin/blob/a7ad6eb38a00397c4a656b53d99bfbad8ea3b2cc/src/main/java/jenkins/plugins/git/GitSCMSourceDefaults.java#L108-L115
But, later it will be fetched anyway.

Solution:
Don't fetch anything during clone operation (useful part is clean, fetch etc). OR pass repositories to CloneComand and remove separate fetch iteration .