Legalising burglary would be a consensus change, most people would agree that theft is malum in se. Its more like legalising insider trading, or tax evasion, or pissing on a public square. Worse in fact, it's denying the right of anyone to make pissing on the public square just because there is no immediate and direct victim. Some people believe that there is no such thing as a victimless crime, and it's impossible to say that they are wrong. You just have to fight them until that opinion is irrelevant.