pull down to refresh
97 sats \ 4 replies \ @carter OP 14 Jul \ parent \ on: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs AI
Hmm you just made me think about the Principle of Explosion tldr; it says if you accept one false fact its kinda a poison pill for your logical system and it will let you prove literally anything. This behavior seems similar. Once we teach it to be underhanded in one respect all the controls that protected us before can also be bypassed. This may be because "truthfulness" is a direction in the latent space and you can't separate the contexts
Ah! So then by introducing inconsistency into the model through refinement, the end result becomes systemically misaligned. Which is kind of like operant conditioning in behavioral psychology?
Scary analogy to Jason-Bourne-style conditioning, actually.
reply
I have seen other things saying that spot training was like "labodimizing" the model and that you sacrifice competency in a specialized task for a loss in general performance. So it may be that those tasks representations where somehow correlated with each other so when you mess with one you hurt the other. You could optimize it to keep everything the same but then you need to train more and specify all the constraints so its not practical
reply
reply
I've said "we are the poor bastards who are forced to live through the learning process" before :)
reply