It does not work that way. Yes it is true that it accumulates in your wallet but they are composed of SATS from different mints. You can send SATS from a specific token mints but if you want to send all of them in just one token mint you have to swap it. Swapping will incur a minimal fee. If your balance is composed of SATS from different mints you cannot send all them in a wallet of a specific token unless you trust it and want to swap it directly in your wallet.