I think it's common for many people to start with something else and then learn Tai Chi later in life - for many reasons. I heard that the Yang kids were first taught external martial arts - very physical punching and kicking - something like 'Chang Chuan' which they did until they had built a solid foundation.