Is this really not considered "downstream"?
Actually, no. Among the many confusing things about Jenkins’ dependency graph is the fact that merely having an UpstreamCause does not suffice. The formal upstream/downstream build relationship presupposes a static project relationship, and then picks a particular build via fingerprint matches, not causes.
my downstream jobs are called recursively
Not quite sure what you mean, but this statement just makes me even less inclined to trust any magical build number determination Jenkins might make. Track the exact numbers of the builds you started and request them. You can pretty easily pass data from an upstream Pipeline build to a downstream build of any type (using parameters), and you can easily pass data (other than its own build number, which is obvious) from a downstream Pipeline build to an upstream Pipeline build (using .buildVariables); a downstream freestyle project could do so with various plugins perhaps.