I think much like in HN, downvoting requires a certain karma threshold, here downvoting should be expensive, but powerful.
We can make it more expensive. If you have low "reputation" or no reputation, flagging posts has no impact. The impact it has is proportional to your reputation.
Another avenue like the one you describe: we can perhaps allow users to pay more to make the impact more severe.
Imagine stacker.news becoming influential. Powerful interests pay a lot to take down content they don't like. I am not sure the power of downvoting should be proportional to payment amount, but the cost of a distinct downvote should be sufficient to make a downvoter pause and think.
People do easily become temporarily irrational when their assumptions are challenged. It might be wise to allow people to 'reverse' their flag by upvoting.