Quote:
|
Never again shall there be any such league of Elves and Men; for Men multiply and the Firstborn decrease, and the two kindreds are estranged.
|
(Council of Elrond)
It's not that contact wasn't possible, it's entirely down to the estrangement of Men and Elves. In the late Third Age the only real contact between Men and Elves (outside of isolated and special cases such as the Rangers) appears to have been that between Thranduil's realm and Lake-town.
So no, you haven't discovered a plot hole.