To us, it's magic. Because they use powers that we can't explain. As Nerwen said, the elves didn't like the use of the westron word, "Magic" because of the wrong connotations used.
In my opinion, and I think someone may have had a similar idea, Elves were just more...in tune with nature and the things around them because they've had longer to get in touch with the things around them. It doesn't necessarily mean they use magic.
Edit:Legate was the person who had the similar idea lol. They've basically summed what I was thinking up and even described it better than I could have.