LotR -- Book 3 - Chapter 09 - Flotsam and Jetsam

[the phantom]

But the Ents do resist Saruman when they arrive at Isengard, in fact they chase him:

I would not even begin to classify that as resisting Saruman- certainly not in the context of the quote in question-

There are not many in Middle-earth that I should say were safe, if they were left alone to talk with him

The ents were not there to "talk". They were breaking and throwing things and trying to chase Saruman down. Plus there were many of them, far from "alone".

This event does not relate to the quote at all.