As I understand it, the line of demarcation between the two is in the presentation of the occult. In LOTR, there are the good and the bad, and it's pretty easy to see which side is the 'better.' The use of magic by the good is low key.
In Potter, which I've never read but have seen, I assume that the use of magic is encouraged, and it's not always clear on which side the characters are. The usual occult props (wands, snakes, skulls, ghosts, etc) are seen and not as 'bad.'
So one argument would be that children like the Potter books, the books encourage magic, occultism and witchcraft, those things are banned by the Bible, are therefore assumedly of the Devil, and so you get:
Kids->Potter->Magic->Devil.
Pretty clear to me.
However, the other POV is
here.
And, back in the day, my parents were sure that playing Dungeons and Dragons was leading me into demon-worship.
Teens->The Hobbit->LOTR->D&D->Devil.
My assumption is that, like that Kevin Bacon game, if you try hard enough, you can always work something back to the Devil if that's where you hope to go.