Yes there is that factor such as when Merry is tripped over by Elfhelm in the fog en route to minaa tirith and Elfhelm is described as tall and Merry eishes he was a tall rider of Rohan. Not clear at all if Elfhelm is tall for a Rohir or just seems tall because Merry is a Hobbit and the Rohirrimnare taller generally than the men of Bree that hobbits might be more familiar with.
There was some horrible eugenics experiment to breed tall warriors IIRC..Failed mo doubt because tall people lumber around due to the lack of oxygen at that altituee....