I don't think it's completely fair to forget that Sauron did send three assaults against Lorien out of Dol Guldur, and there was also fighting on the part of the Mirkwood Elves on the same day as the second assault. It's not like they were sitting at home knitting socks.
You can blame them for not leaving their land to fight with the Rohirrim and the Gondorians, but that's really not quite fair. The Dwarves and the Men of Dale didn't go either; they had to defend their own homes. And consider Theoden: he didn't go to the aid of Gondor until Saruman was taken care of and they didn't have another army ready to march on them from the other side.
As for the Elves at Rivendell and the Grey Havens, I can't say I really blame them, either. They don't really want another fight. And after thousands of years of fighting off and on, first Morgoth then Sauron, can you really say that you'd feel differently? Most of them were pretty much ready to leave M-E; they were just waiting for the right time to do it. Why get embroiled in another war? Especially with Men, who really don't seem to think to highly of them - mythical, sorcerous...