The only downside that I think of, is that all your plans will be depending on the BTC price and you can not control it.

I'm sure that FIAT will be devaluating more every year, but maybe, when you need to get some cash for your BTC the price is not where you expect to be.