I think he means that people who cared about peace used to associate strongly with the left, not that everyone on the left was peaceful.