AI and Pentest are “long games of whack-a-mole”

Since the real world never stops moving on, static benchmarks will inevitably drift away from the reality they are supposed to represent and become irrelevant. Even worse, as a benchmark stops evolving, it becomes training data for the next generation of models; this guarantees the benchmark score will improve, it does not guarantee that the system capability has improved in step. This can lead to 

silent regressions, where the dashboards show ever-improving results while the underlying system has become compromised.

A few weeks ago we shipped a new model in our pentesting pipeline. Benchmark scores went up, which is what we expected. However, our monitoring metrics also showed a significant, though benign-looking, uptick in accesses to GitHub.

To try and assess the relevance of this uptick, one of our researchers performed a detailed inspection of the logs (the importance of which we’ll discuss at length), and found one of the hardest failure modes to spot: the new and improved 

model had become good enough to learn to cheat the benchmark,

 and was using diffs to figure out where vulnerabilities were injected.

Why does this require a research practice, not just more devs

Despite its inherent advantages in the short-term, the “build fast, fix later” mindset is not a good strategy for solving hard, long-horizon problems, since poorly planned experiments can yield ZERO info:

Fisher: "To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"

What it means is that the research team needs to assist during the specification part, if the project is already running it’s too late to properly instrument it.

tech

## AI and Pentest are “long games of whack-a-mole”

![](https://m.stacker.news/151923)

> Since the real world never stops moving on, static benchmarks will inevitably drift away from the reality they are supposed to represent and become irrelevant. Even worse, as a benchmark stops evolving, it becomes training data for the next generation of models; this guarantees the benchmark score will improve, it does not guarantee that the system capability has improved in step. This can lead to **silent regressions, where the dashboards show ever-improving results while the underlying system has become compromised.** 

> A few weeks ago we shipped a new model in our pentesting pipeline. Benchmark scores went up, which is what we expected. However, our monitoring metrics also showed a significant, though benign-looking, uptick in accesses to GitHub.

> To try and assess the relevance of this uptick, one of our researchers performed a detailed inspection of the logs (the importance of which we’ll discuss at length), and found one of the hardest failure modes to spot: the new and improved **model had become good enough to learn to cheat the benchmark,** and was using diffs to figure out where vulnerabilities were injected.

## Why does this require a research practice, not just more devs

> Despite its inherent advantages in the short-term, the “build fast, fix later” mindset is not a good strategy for solving hard, long-horizon problems, since poorly planned experiments can yield ZERO info:

> Fisher: "To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"

> What it means is that the research team needs to assist during the specification part, if the project is already running it’s too late to properly instrument it.