Who was the old man?

[The Might]

Suddenly Gimli looked up, and there just on the edge of the fire-light stood an old bent man, leaning on a staff, and wrapped in a great cloak; his wide-brimmed hat was pulled down over his eyes. Gimli sprang up, too amazed for the moment to cry out, though at once the thought flashed into his mind that Saruman had caught them. Both Aragorn and Legolas, roused by his sudden movement, sat up and stared. The old man did not speak or make, sign.
'Well, father, what can we do for you?' said Aragorn, leaping to his feet. 'Come and be warm, if you are cold!' He strode forward, but the old man was gone. There was no trace of him to be found near at hand, and they did not dare to wander far. The moon had set and the night was very dark.

It was an evil phantom of Saruman that we saw last night. I am sure of it, even under the light of morning.

'Wait a minute!' cried Gimli. 'There is another thing that I should like to know first. Was it you, Gandalf, or Saruman that we saw last night?'
'You certainly did not see me,' answered Gandalf, 'therefore I must guess that you saw Saruman. Evidently we look so much alike that your desire to make an incurable dent in my hat must be excused.'
'Good, good!' said Gimli. 'I am glad that it was not you.'

The question I have is what did the three see - Saruman or only just a phantom?
It does make sense Saruman perhaps so keen to get his hands on the Ring went as far so as to leave Isengard and go take it himself, but only discovered the corpses of the Uruks and afterwards caused the horses to flee hoping to weaken the three Hunters that way.

However, he was planning much at that time and was preparing to attack at the Fords of the Isen, this makes me doubt him leaving Orthanc at that time.

What is your opinion?