Apologies for posting a dupe of another post (#1032946) from @hn, but I thought the article would get more engagement if not posted by a bot. Apologies also for not zap forwarding to the bot, since I wasn't sure if that's what the bot owner would've wanted. If you own @hn bot and would like to be forwarded all the zaps from this post, let me know.
But I thought this was a pretty interesting experiment (randomized control trial) in which experienced developers were asked to complete issues in open source projects, and were randomly allowed or disallowed to use AI. They found that the devs who used AI were actually slower.
I have a few hypotheses about why this might be:
- The devs already had a lot of experience with the repos they were assigned tasks in, and thus the marginal gain from using AI is small
- Devs who were allowed to use AI may have felt obligated to complete their task using AI, even if it wasn't the optimal thing to do
- In my experience, AI is better at prototyping ideas from scratch, or helping you understand a project that you have little experience with, than it is at making surgical changes to a large code base to resolve a highly specific issue
Thoughts?