I found the answer to my question: https://en.bitcoin.it/wiki/Weight_units

I never realized that the witness data was stored on nodes that don't prune. Pruning nodes will only ever end up with a maximum of 1MB per block, but I imagine that's not what most nodes do.