[the phantom]

I know that everywhere else would fall; Gondor, Rohan even Rivendell

Yes, Rivendell would fall, but remember what Gandalf said to Frodo in LOTR (FOTR, Many Meetings)-

"What about Rivendell and the Elves. Is Rivendell safe?"
'Yes, at present, until all else is conquered.'

So it looks like Gandalf thought that Rivendell would last the longest.