Yes, I think it comes down to a matter of understanding. What do Men do when they don't understand something? They either try to break it down, find out what it's made of and master it, or they create whacked out stories and try to avoid the place as much as possible.
I definitely got a sense that Rivendell was a place that Men (of Gondor) knew and understood it to be:
Quote:
|
'Of these words we could understand little, and we spoke to our father, Denethor, Lord of Minas Tirith, wise in the lore of Gondor. This only would he say, that Imladris was of old the name among the Elves of a far northern dale, where Elrond the halfelven dwelt, greatest of lore-masters.'
|
So, Rivendell was at least a place that Men could understand and knew about.
Where Lorien was a place where nobody had been, they thought of it as a mystery, they couldn't understand, so it must have been under the power of some dark Witch and should be avoided at all costs.