Really good question! Not least because these divisions between Men and Elves show just how far apart they had grown in a mistrustful world, possibly due to the threat of Mordor. Sauron's might could only have grown seeing Men and Elves grow even more separate, bet he was rubbing his hands to see it.
I think if you look at what some of the ordinary Rohirrim and Gondorians we meet in the book say, it's clear they were all suspicious, even superstitious about the Elves who lived in Lorien. The Rohirrim have plenty of old tales, fairy tales even, about Lorien, but they do seem to more quickly accept that it is not dangerous or an enemy once Theoden has been won round/cured. They seem to be more open to new ideas, but they are a new culture, still in the process of growing in the place they've found. Whereas the Gondorians are an old culture, and seem to ahve drawn in on themselves a lot more - understandable being so close to Mordor, but possibly also influenced by Denethor who has grwon increasingly suspicious.
Either way, both cultures are quite insular out of necessity, just because they have to expend so many energies in self-protection. I've mentioned this idea before, but this time I'm going to include Lorien in that criticism, as they too are incredibly insular! It's not just the fault of Men!
