I wonder how they're verifying inference. Ah open source is "coming soon" aka they probably aren't verifying it. Also, isn't using the internet as a bus going to choke tok/s?

Distributed computing like this makes more sense for training/tuning afaict - because the results are easier to verify than generate.