It seemed to be my own fault causing Jenkins to rebuild all branches. When changing the configuration from script console I removed the existing branch source configuration and recreated it. That seems to be the reason why Jenkins started to build all branches again.
However it would still be very beneficial to set an ignore for newer or older branches. In case one adds an old repository to Jenkins and doesn't want any of the branches to start building automatically.
Also maybe configure not to build automatically when adding a project, but still build automatically when new commits are pushed up and triggered with notifyCommit, but that is maybe not in the scope of this plugin.