The problem (at least w.r.t. scaling) is not the size of the historical chain, it's the size of the state, which must be maintained in order to prevent double spending.