Mithril it is,
Eönwë!
You've already correctly explained lines 3 and 5; here's the whole of my thoughts behind the verses:
I keep the guard though I have no eyes,
- mithril helmets worn by the Guards of the Citadel in Minas Tirith,
I speak old words but am none of the Wise;
- inscription on west gate of Moria in ithildin (which the Elves made of mithril),
prize of thief not won by theft,
- the mithril shirt Bilbo received for his burgling services,
yet from my hidden home I was reft.
- dug from lodes under Caradhras,
No beast am I, yet by hunters sought;
- Aragorn's words about the shirt when he found it on Frodo,
to some who craved me their bane I brought.
- Dwarves digging to deep for it awakened Durin's Bane (but the Orcs fighting over Frodo's shirt actually fit quite well, too).
See? I told you you were close. Take the floor!