Since commit 853603cccd4434b116ef9b8e094c3f5b815aa75a, we set GIT_LFS_SKIP_SMUDGE=1 to improve performance, and require users to add the "Git LFS pull after checkout" additional behaviour to resolve pointers.
This was mostly done because git clone performance was pretty bad with the smudge filter enabled, and in some configurations, (ssh clone URLs), git lfs pull didn't work.
Since then, a lot has improved:
Clone performance through the smudge filter has improved since https://github.com/git-lfs/git-lfs/pull/2511.
Backends should support the git-lfs-authenticate dance as described in https://github.com/git-lfs/git-lfs/blob/master/docs/api/authentication.md#ssh, which retrieves a temporary authorization token to do the clone over https.
I propose to just do a git clone without setting GIT_LFS_SKIP_SMUDGE=1. Most systems should have done a git lfs install before, so they will resolve the lfs pointers during a clone. We can keep the "Git LFS pull after checkout" additional behaviour, for systems that don't have setup global smudge filters via git lfs install.