Running debug with Maven supports hot swap of code. I would like to see this feature also for Gradle builds. It may be possible to explore the Gradle continuous builds feature for this purpose https://docs.gradle.org/current/userguide/continuous_build.html.
That would possibly mean that the server must be spawned as an external java process, which also gives opportunity to set debug flags and port as parameter or configuration inside the build script instead of GRADLE_OPTS before starting Gradle.