[The Squatter of Amon Rûdh]

By no means are you too attached. There's no experience quite like the first reading of any novel, and to some extent knowing how things are going to turn out can remove some of the freshness from subsequent visits. I've found, however, that Middle-earth has a lot to offer the habitual tourist: such is the depth and complexity of Tolkien's world that one journey there will barely scrape the surface of what there is to be seen and enjoyed; and there's always something new to be discovered.
I suppose that the experience may be likened to visiting a foreign country: at first what grasps the attention is the newness and strangeness of everything, but on subsequent occasions, perhaps knowing more of the history, or having learned more of the language, new sides of the place reveal themselves that make it seem entirely new again.
The last time that I read The Lord of the Rings, for example, I began to see subtleties of characterisation and plot that I had missed completely on every previous occasion; new information that I couldn't believe I missed before was suddenly startlingly clear to me, and familiar scenes seemed subtly different.

Therefore do not despair. To some extent every reading of the books is like the first; in some ways it's even more enjoyable to return and visit the old people and places, and perhaps to discover them anew.