Because of the involvement more or less required to get really into his world and understand it. Others see this sort of committment to one book, world, author, et. al as 'nerdy' simply because of the amount of committment.
Note these definitions of 'nerd':
'a student who studies excessively' (confirms my statement above - Tolkien's world being wide open for in-depth study)
'1. [mainstream slang] Pejorative applied to anyone
with an above-average IQ and few gifts at small talk and ordinary
social rituals. 2. [jargon]
Term of praise applied (in conscious
ironic reference to sense 1) to someone who knows what's really
important and interesting and doesn't care to be distracted by
trivial chatter and silly status games'
If we're considered 'nerds,' I'd have to say that last definition is the reality of it.
Well, some of us...