[HerenIstarion]

Two passages out of UT may be found of interest here:

Yet in truth Saruman's spying and great secrecy [concerning the Shire - H-I] had not in the beginning any evil purpose, but was no more than a folly born of pride. Small matters, unworthy it would seem to be reported, may yet prove of great moment ere the end. Now truth to tell, observing Gandalf's love of the herb that he called "pipe-weed" (for which, he said, if for nothing else, the Little People should be honoured), Saruman had affected to scoff at it, but in private he made trial of it, and soon began to use it; and for tins reason the Shire remained important to him. Yet he dreaded lest this should be discovered, and his own mockery turned against him, so that he would be laughed at for imitating Gandalf, and scorned for doing so by stealth. This then was the reason for his great secrecy in all his dealings with the Shire even from the first before any shadow of doubt had fallen upon it, and it was little guarded, free for those who wished to enter. For this reason also Saruman ceased to go thither in person; for it came to his knowledge that he had not been all unobserved by the keen-eyed Halflings, and some, seeing the figure as it were of an old man clad in grey or russet stealing through the woods or passing through the dusk, had mistaken him for Gandalf.

Also:

Now because of his dislike and fear, in the later days Saruman avoided Gandalf, and they seldom met, except at the assemblies of the White Council. It was at the great Council held in 2851 that the "Halflings' leaf" was first spoken of, and the matter was noted with amusement at the time, though it was afterwards remembered in a different light. The Council met in Rivendell, and Gandalf sat apart, silent, but smoking prodigiously (a thing he had never done before on such an occasion), while Saruman spoke against him, and urged that contrary to Gandalf's advice Dol Guldur should not yet be molested. Both the silence and the smoke seemed greatly to annoy Saruman, and before the Council dispersed be said to Gandalf: "When weighty matters are in debate, Mithrandir, I wonder a little that you should play with your toys of fire and smoke, while others are in earnest speech."
But Gandalf laughed, and replied: "You would not wonder if you used this herb yourself. Yon might find that smoke blown out cleared your mind of shadows within. Anyway, it gives patience, to listen to error without anger. But it is not one of my toys. It is an art of the Little People away in the West: merry and worthy folk, though not of much account, perhaps, in your high policies."
Saruman was little appeased by this answer (for he hated mockery, however gentle), and he said then coldly: "You jest, Lord Mithrandir, as is your way. I know well enough that you have become a curious explorer of the small: weeds, wild things and childish folk. Your time is your own to spend, if you have nothing worthier to do; and your friends you may make as you please. But to me the days are too dark for wanderers' tales, and I have no time for the simples of peasants."
Gandalf did not laugh again; and he did not answer, but looking keenly at Saruman he drew on his pipe and sent out a great ring of smoke with many smaller rings that followed it. Then he put up his hand, as if to grasp them, and they vanished. With that he got up and left Saruman without another word; but Saruman stood for some time silent, and his face was dark with doubt and displeasure.

I believe the sentence which started the thread (Your love of the halflings leaf has clearly slowed your mind) is the effort on screenplayers part to reflect all these subtle playes 'behind the stage'