Did the Eye ever leave Barad Dûr?

I think the eye did left Barad dûr after the dagorlad as Sauron's shattered and dispersed spirit and could not take physical form (in a human shape at least) since nearly all his power was put in the one ring as JRR Tolkien says in the LOTR and the silmarillion. there's no written or told affirmation, not even by Gandalf or the white council that the Necromancer (Sauron) was actually an embodied self at the time of his defeat in Dol Guldur. All his remaining power could gave to him when he finally rebuilded Barad dûr was the shape of a terrible eye, perhaps trying to inspire again the fear that the eye produced when he had a body. I don't know....just a thought.