For me it's a bit of both. I have some pretty nasty issues with food, so I have to be pretty vigilant. I can gain weight on a very healthy diet due to how much I always want to eat.

For most people though, I'd 100% agree. There's been plenty of people who've lost significant weight just by no longer drinking their calories