The Old Forest

[The Saucepan Man]

It's because that's where Old Man Willow, the source of evil power in the Old Forest, has his roots.

My Complete Tolkien Companion (not definitive admittedly, but generally pretty reliable) states:

The chief peril of the Old Forest lay in the malevolent influence exerted over its trees by the Great Willow which, from earliest times, was able to dominate the Withywindle valley, and eventually came to control almost the entire forest.

The Old Willow's influence over the trees of the forest ensured that all who wandered in it came to him.

It does make me wonder, though, how Farnmr Maggot was able to visit Tom Bombadil. Perhaps he was protected in some way by Tom.