Keen observation.
Envying 'at times' I could accept. It shouldn't be likened to the elves' constant day-to-day struggle with the ever-changing world.
The envy of the Valar would be of a different kind - they know 'the end' that men experience (what happens to their souls after death); it seems to me that an afterlife outside of the world in which some sort of 'fellowship' or contact with Eru is implied, and the Valar would undoubtedly miss this (which they experienced before the Music, before entering Ea) and wish to get away from their position as supervisors of the children.