This is a little outside my wheelhouse. I know it's an issue that experimental economists (major overlap with behavioral) always have to deal with. Oftentimes, they're trying get subjects to jump through some pretty complicated hoops, so they need the UX to be as intuitive as possible.

I'm not aware of one canonical treatment of the subject, though. It seems like individual experimentalists largely navigate it on their own, while borrowing heavily from each other.

It is mechanism design, but mechanism design is generally more theoretical and would tell you about the incentives created by the system you're studying and what it's equilibrium conditions are. I'm sure there's a bunch of work on applied mechanism design, though.