I think that's exactly what he intended to get across - note that those left in Middle-earth are the Moriquendi who had not seen the Light of Valinor. However a little of it is that Elves simply withdraw from men's lives, as do Hobbits (and Wizards and Dwarves), which is also a neat writerly device for showing how the magic has declined and the world has become more mundane.
Indeed in Mists of Avalon, an important book for modern Pagans, the only descent we see is the 'descent' to Christianity which takes the power away from the Land, the Britons (or Brythons if you prefer, having noted this term when I was reading about the long lost Cumric language yesterday) and from women. But I wouldn't expect you to be kind about this work as Zimmer Bradley's not all that kind to Christians. Gwynhwyfwr (can't remember the spelling) is a bit of a caricature TBH, but hey, so are 'heathens' in Christian Arts.
Is it all 'redressing the balance' or just having a go back? You decide.
There's certainly a descent into something or other in LotR, possibly just mundanity, a decline from the mystical and magical (which is what makes me sad at the end, no more Elves and wizards and dragons! Boo!), so that's very interesting if we're saying it marks the beginning of more 'earthlike' religions in Middle-earth.