[Inziladun]

I think Legate has the right of it.
Obviously those not in Elendil's line had touched it before. The shards of Narsil had been brought to Rivendell after Isildur's death by a 'nobody' esquire. Also, Elven-smiths had reforged it in Rivendell, and you'd think that might involve laying hands on it (unless Aragorn was paranoid enough to want to hold it while they did their thing).