[Child of the 7th Age]

The Unfinished Tales does go into this at length in a chapter that is entitled The Quest of Erebor. Gandalf says he has been "attracted by Bilbo long before , as a child, and a young hobbit." According to Gandalf, he had shown eagerness, bright eyes, and a love of tales and adventures as a youth. However, Bilbo had changed considerably, becoming a much more conventional figure, a fact that dismayed Gandalf when he first met him. Still, it came down to gut feeling. Perhaps we might call it providential intuition (with the benefit of hindsight). Or, as Gandalf says, "I knew in my heart that Bilbo must go with him, or else the whole quest would be a failure--or, as I should say now, the far more important events by the wa would not com to pass."

As to why a hobbit in particular, JRRT does not explicitly discuss that here. However, if one truly wants a burglar rather than a hero, hobbits have those qualities in abundance--ability to move silently with good wits and arts of deception. In both the hobbit and the Ring quest, it is these skills, rather than brute force or primitive heroism, that are needed.

With the benefit of hindsight (since this was written after LotR), the wizard also alludes to the fact that he was troubled in general by the fact that Saruman was hindering all his plans and that Sauron had risen again.

Still, in interpreting all this, we have to remember it was a later 'tidying up' of the story line. Perhaps, if we start at the beginning, rather than the end, we'd have to say it was plain good luck or blind providence that chose Bilbo!