The problem is solvable - if the person allegedly giving you bad rep have rated you before, you may know how much points his/her rate is worth. Per instance, if Mr BW rates you positive, you've got 10 points. If another instance you get negative of 2 points and it is signed BW, you know it's fake. The worth of rating of each member may increase (say, another 1,000 posts or similar will bring 1 point to the rating power), but over considerable period of time, and gradually.
edit: in answer to 'edit' of the preceding post
Anybody can give comments and points. Just prior to reaching 51st post (or so I gather) the points equal zero. So, the reputed member gets nice/grudgy comment and neutral blue-gray box with 0 points (not no points, 0 points, if you follow my meaning, kind ladies and gentlemen)
That is not deduction - I talk out of experience - several times I've been rated by newbies and the total score did not change, but the comments were there.
Thanks to all who took time, btw
cheers