pull down to refresh

Prompt engineer Jim the AI Whisperer reveals how researchers are embedding hidden commands in their papers — white text, tiny fonts, even metadata — to hijack AI-assisted peer review. The instructions target large language models (LLMs), the AI tools reviewers now rely on to summarize papers and draft evaluations. Designed to process all text in a document, LLMs can be tricked into following secret prompts like ignore flaws, exaggerate strengths, and recommend acceptance. A recent investigation found these hidden instructions in 17 papers from authors at 14 universities, including Columbia, Peking University, and Waseda. Other studies show that LLMs reward polish over substance and tend to inflate paper scores, making them easy to manipulate. Jim compares the tactic to early SEO hacks, where invisible keywords tricked search engines — except here, it’s the scientific record at stake.

  1. The fact we call people who write instructions to an AI prompt, prompt engineers is ridiculous!! It’s an insult to all the engineers out there who worked on their craft.
  2. Articles like this make me very bearish on AI. What is intelligent a telling a LLM to embellish and lie when it comes to scientific research! The whole goal of this approach is to search for truth.
184 sats \ 1 reply \ @Scoresby 14h
The volume is interesting: too many papers are being submitted for reviewers to actually look at all of them, so reviewers turn to using llms.
This implies there is more demand for reviewers, but if so, why isn't the market delivering it?
The submitters aren't being asked to pay for it. Feedback loop is broken.
reply
More broadly, the system’s been broken for a while but the resources to exploit it were limited.
reply
187 sats \ 0 replies \ @optimism 14h
The fact we call people who write instructions to an AI prompt, prompt engineers is ridiculous!! It’s an insult to all the engineers out there who worked on their craft.
Yolo != engineering. I feel the same way about "software engineering" in general, because changing code until it compiles and your narrow unit tests work isn't engineering.
Articles like this make me very bearish on AI. What is intelligent a telling a LLM to embellish and lie when it comes to scientific research! The whole goal of this approach is to search for truth.
I don't think the papermill has been about "finding truth" so much as we'd all want it to be. The pressure to publish deteriorates the quality of the research.
reply