Another trade route to consider, though it's largely speculative,
would be from Pelargir or Osgiliath south. This is the sort of trade
which (at least minimully) might be kept open all through the Third
Age, and be especially important at the height of Gondor's influence
in the Third and Fourth Ages. The basic trade routes might be The
Harad Road and by sea (especially when South Gondor was under
Gondor control but even at times of relative peace by the Corsairs).
Perusing Karen Fonstad's Atlas it's at least interesting as an
insight into Tolkien's thinking that (see pps. 38-39 and 52-53)
that Far Harad and south bear a similarity to Africa. Since oliphaunts
were thriving there why not other exotic animals that an imperial
Gondor, not unlike imperial Rome, would be interested in.
Wonder what Sam would think of a giraffe or even a saber-toothed
tiger

).
Southrons and such might be interested in turn in fine armor,
tobacco, and rare wines.
About the
Legate's comments. You might be underestimating the
importance of the dwarves in the Blue Mountains. Wasn't it said they
regularly traversed the Shire? Who were they trading with?
But the areas east of the Misty Mountains do seem to be increasing in
population and importance. The Beornings and the woodsmen were
growing and spreading even in The Hobbit.