[The Squatter of Amon Rûdh]

This might shed some light on the situation:

Indeed, of all the Istari, one only remained faithful, and he was the last-comer. For Radagast, the fourth, became enamoured of the many beasts and birds that dwealt in Middle-earth, and forsook Elves and Men, and spent his days among the wild creatures. Thus he got his name (which is in the tongue of Numenor of old, and signifies, it is said, 'tender of beasts').

Unfinished Tales - The Istari

There you have it: Radagast neglected his duty in order to become "Radagast the Bird-tamer", and only Gandalf remained faithful to the original mission of the Istari.