[Boromir88]

I think "during the Dark Years" may be indicative that they were fleeing Sauron.~Kuru

I'd agree. They didn't like the Numenorean invaders, but in general didn't like invaders of any kind:

'They were a secretive people, suspicious of other kinds of Men by whom they had been harried and persecuted as long as they could remember, and they wandered west seeking a land where they could be hidden and have peace.'
...
It is said here that the native people of Enedwaith, fleeing from the devastations of the Numenoreans along the course of the Gwathlo, did not cross the Isen nor take refuge in the great promontory between Isen and Lefnui that formed the north arm of the Bay of Belfalas, because of the 'Pukel-men,' who were a secret and fell people, tireless and silent hunters, using poisoned darts. They said that they had always been there, and had formerly lived also in the White Mountains. In ages past they had paid no heed to the Great Dark One (Morgoth), nor did they later ally themselves with Sauron; for they hated all invaders from the East. From the East, they said, had come the tall Men who drove them from the White Mountains, and they were wicked at heart.~Unfinished Tales: The Druedain