To me, in the first half of the statement he is expressing regret that he hasn't got to know some people as much as he would have liked. In the second half, he is saying how there are some in the crowd, but by no means all, who he thinks might have deserved to be more highly esteemed by him.
It's a compliment all round. A very English and veiled one at that.
It's also one of my favourite sayings and I have long been waiting for a chance to use it, though I think it's possibly too well-known to use it in my work; maybe if I had won the lottery and was not afraid of getting the sack...
