If you know someone's address, presumably it is not empty

Dust attackers only target empty addresses that have already spent their coins.

If it is empty, presumably it will never be reused.

That's the point: Dust attackers purposely refill it so it will be reused by a clumsy victim.