It's very difficult to brute force 6 words, but not beyond the realms of possibility. Here's an example of someone brute forcing the last 4 words of a 12 word seed phrase in a day:

Note: It's much easier to bruteforce the last word as this is just a checksum.

But the BEST reason not to do this is because it's just inferior to using a passphrase. Both are 2-of-2 schemes, but a passphrase is A) Easier to memorise, B) Gives you plausible deniability with the seed-only wallet and C) Is an industry standard supported by every hardware wallet.