This will never be really finished, but I filed a bunch of issues and PRs around. Marked progressively everything as resolved. We will for sure see some issues coming back, but hopefully less and less. Or at least not the same .
Currently, for the record, I've marked all issues as resolved.