[Inziladun]

Though impatient with those who displayed foolishness (understandably, considering the deadly seriousness of his business), I don't see Gandalf ever physically punishing any creature out of frustration or anger. He didn't hurt Pippin when he gave away the presence of the Nine Walkers in Moria, or when he nearly brought ruin on them all with his use of the Palantír of Orthanc.
Saying he thought of "roasting (Butturbur) over a slow fire" is hardly the same as doing it.
As for Gollum, Gandalf only threatened him with fire, and did not actually harm him.
In Bilbo's case, Gandalf had to do something to get Bilbo's attention, before the Ring gained too much of a hold for him to be able to let it go.
Gandalf always knew where to draw the line, and so avoided becoming like Sauron or Saruman, who, I'm sure, had no compunction as regards to venting their displeasure on their slaves.