Gondor, Rohan and Lorien

[Raynor]

I've always wondered what the difference with Rivendell is.

Elrond was more active in his fight against evil (or, at least, more known - seeing that one could perceive that Lorien is protected by singing in the trees); he fough directly against Sauron in Eregion and at the last alliance, he sends his sons into battle, he houses the dunedain, and, very importantly, is descending from human himself. The very name and description of the Homely House, as told in A short rest, the Hobbit, quoted in the Many Meetings, FotR:

His house was perfect, whether you liked food, or sleep, or work, or story-telling, or singing, or just sitting and thinking best, or a pleasant mixture of them all

seems to suggest that this was a haven for not only the resistance, but for visitors too.