I would like to be able to get the specific agent(s) that were used during a Pipeline build using the API. (eg JENKINS_URL/job/pipeline/1/api/json?pretty=true)

This information is available for Freestyle jobs using the API, you can access the builtOn field:
JENKINS_URL/job/freestyle/1/api/json?pretty=true

{{ "builtOn" : "ssh-agent",}}

This information is missing for Pipelines.

I started to work on a Groovy script to traverse the Pipeline flow graph, but so far have stumbled to figure out how to get the actual agent information, right now I can just find the labels used for that job.

Regardless if I can get this groovy script working, I think it's much more convenient for everyone if we can provide this data via the API instead of having to use the CLI to run a groovy script.