pull down to refresh

15 sats \ 3 replies \ @SimpleStacker 15 May \ on: Neural networks do math by rotating shapes. AI

It's cool, but very straightforward when you think about it. An embedding, the fundamental unit of analysis for a LLM, is simply a point in a N-dimensional space. Every operation in a LLM's architecture is an operation on a collection of points in N-dimensional space, so of course it's all shapes.

write
compose
82 sats \ 0 replies \ @zuspotirko OP 15 May

I'm going to be honest with you: this goes over my head. I'm in awe and understand none of that.

reply
20 sats \ 1 reply \ @south_korea_ln 15 May

How do they decide the dimensions of this N-dimensional shape to use for the 3d plots in the figures?

reply
20 sats \ 0 replies \ @optimism 15 May

Exactly what you said up there, I think.

The 3D plot takes point+distance over an aggegated (3D instead of 2048D) space and shows the idea of (but not really representative due to the pre-aggregation) the relations as coordinates.

reply