It is possible to create custom config files since config-file-provider-2.16.0. Some other plugins like maven-plugin test how they manage their configuration files, but this feature is not tested by any plugin. It would be recommendable to create a test to cover it.