Where did Saruman go?

[Amarie of the Vanyar]

But Sauron ended up in the Void after the destruction of the Ring, his spirit is not wandering in Middle-Earth:

'Go back to the abyss prepared for you! Go back! Fall into the nothingness that awaits you and your Master. Go!'.
Gandalf to the Witch King; The Siege of Gondor; LotR

But in the years he rose like a shadow of Morgoth and a ghost of his malice, and walked behind him on the same ruinous path down into the Void.
Of the Enemies; Valaquenta; The Silmarillion