At this moment is impossible to keep a hot-swap backup Jenkins instance running because all schedulers will start builds.
The workaround of disabling jobs in order to prevent them from starting on a schedule does work only for the most basic deployments, ones that probably do not need a backup.
- production may have disabled jobs on normal use, disabling all on backup copy would prevent us from enabling back the correct ones if needed.
- disabled jobs cannot be manually triggered by users, it means that if they want to run them on the 2nd server, they need to enable them.
The only safe way to enable this is to add a DRY option that tells Jenkins to:
- not trigger any timer based schedulers (just ignore them)
- not trigger on incoming SCM changes (silent skip)
- allow only manual job triggering
- skip sending emails
As it can be seen some of these features depend on plugins and it would be up to the plugin authors to implement this feature. Still, the core Jenkins needs to be able to expose this "DRY" property.
Ideally this property should be configurable without having to restart Jenkins so we can implement a failover service from main Jenkins master to secondary one. Still, if there are technical reasons which would not make this possible, even a JAVA_OPTS option would be acceptable.
- job timers/schedulers - no workaround other than disabling all the jobs on secondary server which can be a serious issue for those using configuration management and lots of jobs.
- email delivery can be prevented at firewall level, really ugly and with side effects on logs
- gerrit triggers can be configured to be offline which means that will not trigger new builds but they can be used to manually trigger specific changesets. Bad part is that this means config change which would make a sync with primary server much more complex as we would have to patch the config.