If launching selenium locally, on a per-job basis, it's easy enough to use the Xvnc plugin to manage a Xvnc session for the lifetime of the build. But using the selenium grid plugin, we can't use that plugin because the Selenium RC server is launched along with Jenkins, and for Selenium to access an X display, the X server has to be already running at the time Jenkins starts.
It would be helpful if Jenkins could a) start Xvnc similar to the way the Xvnc plugin does it, adding DISPLAY to the environment so that Selenium RC knows what display to use, b) try to determine if Selenium is going to work at all, based on the presence (or absence) of an X display. Currently, Jenkins will kick off several Selenium servers on each node (one per executor), but if there is no X display on that machine, it's a complete waste, because selenium will never be able to run on that machine. Jenkins should be able to know that Selenium is doomed to fail, and simply refuse to start it if it can't.
It would be helpful to have some sort of feedback from the plugin (like in the log file, "WARN: Not launching selenium because no display is available") – so that at least we know that there is a problem, and a hint as to how to fix it.
I haven't gone too far into how Jenkins interacts with Selenium yet, but ideally, if Jenkins is able to launch the selenium RC server on-demand, rather than on startup, it would be able to wrap that session in an Xvnc/Xvfb launch, so that X is only running for the duration of the tests. I'd rather not keep an X server running 24/7 just so we can get once-a-day selenium tests (for example).