This has been the subject of debate, with some trying to reconcile what Gandalf says in Shadow of the Past in the manner you suggest, and others simply suggesting that this was an inconsistency in JRRT's story telling. You may want to look at this thread (and there are others):
http://forum.barrowdowns.com/showthread.php?t=19710
Feel free to continue the discussion and voice your views in either thread.