Specifically, CpsFlowExecution holds the loaded scripts, which are potentially large:

https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L239

And the main script:

https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L233

We don't need to load these EVERY time we load a build.  We could lazy-load them as needed (should happen once in a blue moon I hope). 

Reducing the number of times we have to re-persist the build.xml would help even more  though - https://issues.jenkins-ci.org/browse/JENKINS-47174