Although, I set the FindBugs Jenkins Plugin to "Compute new warnings (based on the last successful build unless another reference build is chosen below)" and fail on 1 found high priority issue, it does not fail. Here is the output:

[findbugs] Executing SpotBugs FindBugsTask from ant task
[findbugs] Running SpotBugs...

[findbugs] Warnings generated: 361
[findbugs] Calculating exit code...
[findbugs] Setting 'bugs found' flag (1)
[findbugs] Exit code set to: 1
[findbugs] Java Result: 1
[findbugs] Output saved to findbugs.xml

BUILD SUCCESSFUL
Total time: 2 minutes 32 seconds

[FINDBUGS] Collecting findbugs analysis files...

[FINDBUGS] Searching for all files in .... that match the pattern **/findbugs.xml
[FINDBUGS] Parsing 1 file in ....
[FINDBUGS] Successfully parsed file ....\findbugs.xml with 360 unique warnings and 1 duplicate.
Skipping warnings blame since Git is the only supported SCM up to now.%n
[FINDBUGS] Computing warning deltas based on reference build #195
[FINDBUGS] Plug-in Result: Success - no threshold has been exceeded