For a standard source code repository, this is not significant performance wise and we have not noticed this issue on our source code repositories.
However, in general, it is not good that it is fetching references and therefore objects that are not required by the build. It is just inefficient. However, as stated, for source code repositories it is not significant enough that we have noticed.
I have created a new issue that targets the scenario for our specific issue JENKINS-14572.
Where we have noticed it is in the Git repositories we are using to deploy our web applications which are large repositories of binary files.
We create each deployment image as an orphan commit, tag it, and push the tag not the branch back to our Git server. This way we can delete the tags after a holding period causing those commits to be unreferenced and eligible for garbage collection.
We have the Jenkins plug-in configured to pull down an 'empty' branch by specifying a refspec. When all of the tags are fetched, then a large number of unneeded objects are pulled down from the repository greatly increasing the size of the repositories on the Jenkins slaves.
Having said all of that, if you want to build a specific tag, why not specify it in the refspec? The refspec can use build parameters also. Why fetch all tags when you only need one?