Based on the places I've been to and to most people I've talked to (those that don't use/have bitcoin), I would say the lack of merchants that accept it or situations where using it is a "no-brainer".

For them bitcoin is just another investment/gamble.