pull down to refresh
You seem to be under the impression that distillation was a bad thing?
This is not the case. This is not a "hopium" narrative. Quite the opposite: it's a pro-china narrative that makes the US situation look bleak because of how much better this strategy is
The narrative I hear out there - maybe this isn't what you personally mean - and maybe I am interpreting this wrong, is:
- It's all distillation (which is true for some but not all Chinese-made models, after all Kimi K2 was from scratch, and iirc GLM 5.0 was too)
- Therefore the source of all LLMs is the US labs
- Therefore if the US innovates on the scratch-to-base layer, the Chinese labs are helpless
All that I'm saying is that if the new tech is just some proprietary Google software then there is no moat, because software can be reverse engineered in a few days now, thanks to LLMs. See for example this repo that reverse engineered the unpublished Kimi K3 training pipeline
If there is some Nvidia or Goog hardware needed as well, then the moat will last for a few months at most; the Chinese hardware manufacturers can go from lab prototype to production at scale much faster than anyone else.
Therefore, I reject any claim of "China cannot do this". Not because I am pro-China, but because I am extremely skeptical of location/geopolitical based superiority of intellect and attitude. I think I know this to be a lie, as I've had the honor of working with people in almost every culture thinkable and the one thing we all excel at is overcoming barriers, if only we want to overcome it.
Oh. I skipped the step that the whole "China only has distillation" is an American hopium narrative. Similar to "China doesn't have the GPUs required" (beaten by Deepseek) or "China doesn't have the traces" (beaten by Kimi) or "China doesn't have the optimizations" (beaten by Deepseek 2nd time). Every time someone claims that China is inferior and cannot do something, then gives a reason for the why, it turns out to be an arrogant statement that doesn't age well.
Have you used Kimi K3? Is it really inferior for you? It was for me at least equal to Fable 5 in both depth and usability for the times that I have used it. Maybe I'm missing a point here and there are use cases where it is poor, but all these things Chinese labs couldn't do are narrated by reducing the humanity of the researchers there to "China", then making up reasons why Chinese tech is inferior, then claiming that this is why USA wins. But is that really true? What's left of the 10 year headstart? Seems like it is 2-3 months now.
Also, whenever OpenAI and Anthropic researchers proudly claim that their LLMs are doing most of the work for the next iteration, they in all their joy are forgetting that if Kimi K3 == Fable 5, then the next iteration of Kimi has no reason to be less well trained as the one that Fable 5 trained for Anthropic, or GPT 6 for OpenAI. Worse for everyone wanting to dominate the world by setting a research pace, Kimi K3 is open, so basically that training capability is now a commodity.
Thus, what I am warning about is that the nationalism of political leaders that need votes and the reality of what everyone's capabilities truly are, may either be already not be true, or otherwise turn out really easy to overcome.