That only happens if you're subscribed to the post, assuming it's not your own. more, I don't have the "someone replies to someone who replied to me" option turned on either.