I've asked k00b about this and I don't think you get punished (as in your trust score going down), but zapping non-top content slows how quickly your trust score grows.
Based on conversations with k00b, I'd be very surprised if your trust score weren't almost as high as his and Siggy's.