ForemanSharedNodeCloud.canProvision(Label l) has to use some caching mechanism. It is utilized very often by Jenkins core, AJAX etc. not only if Jenkins master is looking for where it can schedule a run from the queue but for many other reasons.

Typical time spent in this method in our testing environment is 5-30 seconds, so if you get threaddump you can be pretty sure you can see at least one thread ("jenkins.util.Timer") in it every time.

One simple approach can be to obtain all possible labels from the Foreman, cache it and use it for let say couple of seconds/minutes etc. and then drop it and obtain it again. Caching time can be configurable (lower value for development, higher value for production) and it can improve current state much.

In addition there are several path to ForemanSharedNodeCloud.canProvision() from the core where hudson.model.Hudson or hudson.model.Queue is locked before, so Jenkins is effectively blocked by this for couple of seconds which can be a big issue in a production deployment.

You can monitor that behavior simply by using Dumpling plugin like:

import com.github.olivergondza.dumpling.query.*;
Dumpling.runtime.threads.grep {it.thread.stackTrace.toString().contains("foreman")}