Thanks Lalwende for the support!
The Fourth Age was the Age of Men and so would be those to come. Elves and Dwarves would slowly pass out of memory and would become legends.
Men were now left to take care for themselves, no need to sit in a tower like Elendil and gaze towards the west dreaming of help from there or of what could have been.
While you say it makes no sense, I disagree. If the Elves thought that all kings of Men would be like Eomer and Aragorn then be sure that the palantir would have stayed, but they knew better. As hinted by Tolkien in a couple of places M-e did not remain a happy place until the Dagor Dagorath...evil Men came to power and in this case it might have been to keep the memory of Valinor hidden from them.
Also, what better explanation should there be....greed? I doubt it...
