That's a nicely designed flier! The
Prancing Pony must have had some of those in a stand with a sign over it saying "Eriador Attractions".
From the books, it appears that no one but Elves and Gandalf go to Rivendell on a regular basis. Even the Rangers (other than Aragorn, who, as you say was a special case) aren't mentioned as staying there, and you'd think they would be likely to do so if they could.
The most frequent travelers after them were probably Dwarves, and they likely would sleep in ditches filled with Trolls' bath water before they'd willingly put themselves at the hospitality of Elves.
Gandalf does appear to be the catalyst that brought 'outsiders' to Rivendell in most cases, and as his real identity was known to Elrond, his judgement was obviously trusted.
The Council was a unique event, and Elrond seems to have accepted that it was 'meant' to be, and that all who showed up for it should be admitted.
Beyond that, I think it was pretty much closed to the wanderer. After all, it was originally founded by Elrond as a 'refuge' and a 'stronghold', so you wouldn't think any old ragged wayfarer would be allowed to find it. Otherwise, couldn't the Witch-king possibly have infiltrated it somehow when he was warring with the Dúnedain?