[The Might]

But alatar, the big problem here is that Saruman does nothing at all to fight against the three. He would have recognized them and I have no doubt he knew he could take them down. He was no coward, he faced Gandalf the Grey and so the Three Hunters would quickly be done with. Why should he not have done it? It would end any possibility for Gondor to have a new king and it would weaken Rohan as they would not receive any help from the three.

Is simply makes no sense at all for Saruman himself to see them and then just go away.

Maybe he looked into the palantir to determine their position (keep in mind palantiri were useful as long as there was enough light where the observation took place - the campfire) and then conjured up some evil ghost of himself and sent it their to annoy them.

It could be that just like Sauron, Saruman was so certain of getting the Ring soon that he saw no need to leave Orthanc at all.