I couldn't get the problem you faced at all.
What does Retry mean?
What's the behavior you expected?
Please report more details.

Normally if I configure "Retry build after failure" it will trigger another build. If I use Flexible publish with the same configuration it does not trigger another build

I got it. You mean Naginator plugin doesn't work with Flexible Publish plgin, right?

I had a look at the implematation of naginator plugin, and saw it does't work with flexible-publish plugin.
Rerun post build step works just as a marker to decide whether rerun is required, and wrapping with flexible-publish makes it hidden and not work as a marker.

This is an issue of naginator plugin, and flexible-publish plugin can do nothing for that.
I changed the target component of this issue.

What naginator plugin can do (this is notes for maintainer(s) of naginator plugin):

• Use Action to mark whether to rerun.
• Add a feature of conditional rerun.
• Doesn't support conditional rerun, and close this issue with Won't Fix.

Code changed in jenkins
User: ikedam
Path:
src/main/java/com/chikli/hudson/plugin/naginator/NaginatorListener.java
src/main/java/com/chikli/hudson/plugin/naginator/NaginatorPublisher.java
src/main/java/com/chikli/hudson/plugin/naginator/NaginatorPublisherScheduleAction.java
src/main/java/com/chikli/hudson/plugin/naginator/NaginatorScheduleAction.java
src/test/java/com/chikli/hudson/plugin/naginator/NaginatorListenerTest.java
src/test/java/com/chikli/hudson/plugin/naginator/NaginatorPublisherTest.java
src/test/java/com/chikli/hudson/plugin/naginator/NaginatorScheduleActionTest.java
http://jenkins-ci.org/commit/naginator-plugin/9621e5d8f41158771440eab2677796cb331f4f03
Log:
[FIXED JENKINS-29715] [FIXED JENKINS-23984] Introduced NaginatorScheduleAction that makes Naginator to reschedule the build. This allows other plugins to trigger naginator and also allows naginator-plugin works even wrapped by other plugins.

Fixed in naginator-1.16.
It will be available in the update center in a day.
Please try that.