And you have to exclude PJ's foolishness in ROTK, with Gandalf assaulting Denethor.
In Letters:
"Why they [the istari] should take such a form is bound up with the 'mythology' of the 'angelic' Powers of the world of this fable. At this point in the fabulous history the purpose was precisely to limit and hinder their exhibition of 'power' on the physical plane, and so that they should do what they were primarilt sent for: train, advise, instruct, arouse the hearts and minds of those threatened by Sauron to a resistance with their own strenghts; and not just to do the job for them."
(And I rather doubt Gandalf made dinner materialize at Rivendell
).
The staffs do seem to be a critical focusing element, as Gandalf was so keen to keep his at Edoras. Since he came back enhanced in power you must assume Gandalf the Grey was significantly limited, and presumably then lesser in power to the head of his order.