I had the same issue and I was able to fix it by running jenkins like this (in elevated CMD window):
java "-Djna.nosys=true" -jar jenkins.war
then Jenkins was able to install as a service.
Also I added that system property to the jenkins.xml file like this:
<arguments>-Xrs -Xmx1536m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djna.nosys=true -jar "%BASE%\jenkins.war" --httpPort=8080 --webroot="%BASE%\war"</arguments>