Boats travelling up and down Anduin and Lorien's isolation.

[Inziladun]

Originally Posted by Elmo

Wonder why they never returned. The perils of the journey or the perils of Galadriel and Lothlórien to mortal men?

I would think the journey would have been the reason, with Lórien's relative nearness to Dol Guldur resulting in Sauron's minions being about the area.
Glóin said to Frodo in Rivendell that the journey from Dale would have been "impossible" if not for the Beornings, and those Gondorians would have been much nearer to the peril of southern Mirkwood.

It wouldn't seem that any Man of Gondor had actually entered Lórien, since Aragorn was presented as such a special case.