Hooom, let us not be hasty. Radagast dwelt in Rhosgobel near Mirkwood. The center of the coming war was going to be Gondor but times were hard in the north too. The inhabitants of Lake Town and Mirkwood had to deal with orcs and it wasn't safe to wander in the wilderness by oneself.
There may have been many reasons, why Radagast didn't go. The Ents' decision to go to war came quite suddenly. He couldn't have travelled such a long distance in time to charge with the Ents. One reason could indeed have been that he didn't know, what Saruman was busying himself with. Or maybe he was scared and didn't want to stick his nose into Saruman's business. And after all, what he could have done? A humble child of nature against the head of the White Council. That doesn't seem very convincing to me.
