Even those who don't sell directly will likely want to spend their Bitcoin on goods and services. Some of the recipients of that spending will exchange for fiat, so even indirectly there will be lots of downward pressure on the exchange rate.