I see it like people who pay more money for rare coins (limited edition or just very old).

I don't think this suddenly makes my EUR coins no longer fungible just because a few people would pay more.

The same holds with sats: You can still buy the same with any sat at any place which accepts bitcoin.

The worst case would be that a counterparty which cares about ordinals gets lucky and receives a rare satoshi which you weren't aware of.