That's irrelevant imo. It could generate a word after a scentence it wrote before that contains brilliant information. It already does so to e.g. write sourcecode that nobody has written before.

By your same argument you could also argue humans were incapable of new thoughts. Human brains also only remix thoughts that it heard before and extrapolate from there. That's not that different