[Child of the 7th Age]

Fingolfin II -

This may help support your point. See Morgoth's Ring, Athrabeth Finrod ah Andreth, Author's Notes on the Commentary, note 4, for this :

....because the Elves believed that the fëar of dead Men also went to Mandos (without choice in the Matter: their free will in regard to death was taken away.) There they waited until they were surrendered to Eru. The truth of this is not asserted. No living Man was allowed to go to Aman. No fëa of a dead Man ever returned to life in Middle-earth.

This seems to be one of those clarifications framed in terms of "It is said or believed", rather than a clear statement of fact. But this is a device that Tolkien employs time and again (such as Sam eventually going to the West), which many readers (myself included) accept as "fact".

See also the statement in the usually reliable Encyclopedia of Arda:

Death; the gift of Ilúvatar to his Younger Children, allowing them to go beyond the confines of the World. After death, Mortal Men were gathered in the Halls of Mandos, and then departed from the World for a destination unknown even to the Valar. Whereas all other beings in Arda, including the Valar themselves, were bound to the World and its fate, the Gift freed Men from this destiny, allowing them to shape their own lives as they wished.

Hope this helps. Perhaps someone has additional quotes on this?