It is possible to configure the job such a way that it starts continuously triggering even if no further SCM changes were committed. I described the detailed reproduction steps below. The GIT plugin code modification with a fix for this issue is also attached in a patch file.

=== REPRODUCTION ===
1. Configure a new job (type "Pipeline") with the following conditions:

• Specify 2 repositories: ssh://<server>:<port>/<project>/pipeline_script.git, ssh://<server>:<port>/<project>/source_code.git in the "Pipeline" section of the job configuration GUI.
• Unclick "Lightweight checkout" (that is important, without it the bug will not reproduce).
• Commit a first version of the pipeline script inside the repository ssh://<server>:<port>/<project>/pipeline_script.git (let's call it PipelineBug_01) with such a dummy content as shown below and configure the job to use this pipeline script:

node ('master') {
  stage ('Checkout') {
  }
  stage ('Build') {
    echo 'Hello World!'
  }
}

(Use case: let imagine one is not too familiar with Jenkins and wrongly put both source code and pipeline script repository instead of just a pipeline script repository to the job configuration GUI)

2. After the job saved, run it first time manually.
Ignore the job failure with "ERROR: /var/lib/jenkins/workspace/<job_name>@script/PipelineBug_01 not found" if it happens.

Check: make sure that the build.xml was updated with both repositories (they must be now mentioned in the remoteUrls section).
grep "remoteUrls|\.git<" /var/lib/jenkins/jobs/<job_name>/builds/1/build.xml | grep -v "git ssh" | grep -v '<url>ssh'
<remoteUrls>
<string>ssh://<server>:<port>/<project>/pipeline_script.git</string>
<string>ssh://<server>:<port>/<project>/source_code.git</string>
</remoteUrls>

3. Modify the pipeline script by adding of the source code repository ssh://<server>:<port>/<project>/source_code.git under the "git" section and add polling (the snippet is below). For safety, add polling "H/3 * * * *" in the job configuration GUI as well, otherwise you need one manual job run in order the polling settings from the pipeline script are automatically applied and appeared in the job configuration GUI as well. So, the resulting pipeline script should look so:

properties([
    pipelineTriggers([
        [$class: "SCMTrigger", scmpoll_spec: "H/3 * * * *"],
    ])
])

node ('master') {
  stage ('Checkout') {
      git(
	  poll: true,
          url: 'ssh://<server>:<port>/<project>/source_code.git',
          credentialsId: '<crId>',
          branch: 'master'
        )
  }

  stage ('Build') {
    echo 'Hello World!'
  }
}

Commit this as a 2nd version of the pipeline script to the repository ssh://<server>:<port>/<project>/pipeline_script.git.
Remove the source code repository ssh://<server>:<port>/<project>/source_code.git from the GUI part and save the configuration.
(Use case: the person learned the mistake and wants to configure the job correctly by moving of the source code repository from the GUI section to the pipeline script.
So, now we suppose that the GUI section contains only the repository for the pipeline script itself and the pipeline script contains the source code repository)

4. Run the job manually again. Then wait 2-3 minutes and make sure that the polling log appears, but no changes are detected, so that the job is not triggered any more.
Check: the build.xml for the build #2 should still contain 2 repositories mentioned in the "<remoteUrls>".

5. Make a commit to the source code repository ssh://<server>:<port>/<project>/source_code.git, wait 2-3 minutes and make sure that the new build has been triggered.
NOTE: at this stage build.xml for the new build becomes much different:

• it contains the section <hudson.plugins.git.util.BuildData plugin="git@3.4.1"> two times.
• <revisionStates> section is also complemented with a new <entry> with the pipeline repository mentioned inside.
• <checkouts class="hudson.util.PersistedList"> section is also complemented with a new <org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout> with the pipeline repository mentioned inside.

6. Now do nothing and just observe how the subsequent builds are triggered more and more even if you don't change anything.
The state machine got to such a condition when it never ends. The only one can do is to disable the job now.
IMPORTANT: Check the polling log and make sure that it wrongly detects the previous build revision(it is seen in the message "[poll] Last Built Revision: Revision <last_build_revision>"): instead of the previous build revision of the source code repository it weirdly takes the previous build revision of the pipeline script repository - that is the reason the new builds trigger more and more and this is actually a bug.

By the way.. If you add some new source code repository to the pipeline script you will see that any newly added repository is then added correctly to the build.xml (<remoteUrls> contains only 1 repository), but only that one which was once wrongly added to the GUI section is "merged" with the pipeline script repository, so that the corresponding sections of the build.xml contains <remoteUrls> with 2 repositories mentioned in the list.
=== ===

=== EXPECTED BEHAVIOR ===
After the removal of the source code repository from the GUI the state machine (build.xml, change*.xml and everything else) should be returned to the state as if the job was configured properly since the beginning (as if the pipeline script repo was never mentioned in the pipeline script and the source code repo was never mentioned in the GUI). I am not sure if it should happen upon the Pressing "Save" button immediately or after the next build. It is up to the Jenkins maintainers how to handle it correctly.

=== WORKAROUND ===
Probably, only a manual modification of the build.xml (did not try) or a re-creating of the job from scratch.

=== FIX ===
According to my investigations, the root cause is the function "public @CheckForNull BuildData getBuildData(Run build)" searches the first matching repository and exits using "break" from the search loop. There should be more precise check that the repository URL is really corresponding.
To store the correct URL I used the scmName field of the class BuildData that is always empty (it is possible to add and use another one, of course). The getKey() method evaluates the URL always properly, so I used it to get the indeed URL.

This fix does not pretend to the final one, since I am not a developer of the plugin. I would be happy if someone makes a review and check my investigations. The patch is to be applied to the plugin version 3.4.1. Please, use the file JENKINS-48938.patch.