To enable more controlled deployment of SeleniumServer to Jenkins nodes, make provision that the service will only deploy servers to nodes with specific labels (or explicitly named nodes).

For example; "Deploy to nodes which match the following label: [SELENIUM]"

This would allow the grid to exist only on machines capable/allocated for running Selenium, eg VNC/Headed, correct browsers installed etc.