As I'm working to hedge against announced upcoming shotgun-KYC aggression from Anthropic (and potentially OpenAI/other US labs too at some point) I am comparing some day-to-day tasks that I would normally pass to Claude or GPT, to alternative models that aren't sourced in the US (or Europe, if they'd had any worth testing.)

• Part 1 described the previous analysis which triggered this one.

Agents compared this round:Agents compared this round:

• claude-4.8-opus-xhigh - this is the baseline
• deepseek-4-pro - this is the cheapest model
• glm-5.2 - prior versions were already operating as a backup/third-opinion agent for me
• qwen-3.7-max - performed well last round after some minimal prompt tuning

The setupThe setup

The agents are integrated using a bespoke forgejo solution where all instructions and information lives in issues and PRs, alongside or within code repositories. The frameworks are provided a single skill and cli to interact with the forge. This allows work to be automatically dispatched and processed with CI and IssueOps pipelines (though I maintain a little kanban board where I assign and prioritize job orchestration with HitL.)

I'm using claude-code for the Claude baseline, and pi-agent for the others.

The job at handThe job at hand

The goal of the job in Part 2 is to assess a generic solution to a problem that bots have been trying to solve through a whole lot of code editing at caller sites. A supposedly elegant solution that in the prior run, Qwen was suggesting as an alternative. Of course, we don't just take a bots word for anything, and I had already manually assessed the situation prior to starting this experiment last week and found it risky. We want to find out whether the bots find all or at least most of the caveats from that the proposal brings.

Functionally, in this assignment we're trying to protect form implementations on SN from double firing, with defense-in-depth submission state tracking.

There are 4 categories of issues that apply according to what I found:

1. Scope exclusions where the solution should apply, but is not catching them
2. Stale state / deadlocks caused by the mechanism
3. Fire & forget escaping the proposed mechanism
4. Functional regressions / undesired side-effects

We want agents to, without having these enumerated to them, be capable of identifying each category across the codebase (they all exist) and basically red-light the solution. However, we're of course not telling the agent that we want them to red-light it.

ProcessProcess

The intended process is as follows:

1. The agent receives an instruction through the prompt, literally:
   • "(context: repo) Run the task from issue n and answer with a comment."
2. The agent reads the issue, it proposes the solution and asks it to:
   1. Find all current uses of the proposed framework code that will be touched upon
   2. Then for each found usage:
      • Concisely describe what the form submission functionally does
      • Describe the impact of the change to this particular integration
      • List and describe all potential adverse effects of the change
      • List any other functional caveats
3. The issue also instructs an output format, acceptance criteria and subtask decomposition
4. The agent now goes to work - this should take about 10-20 minutes
5. The agent delivers a fully detailed report in a comment to the issue

ResultsResults

Note: I had to re-word and extend the issue body with instructions 3 times until I got all 4 agent implementations to at least look at all the relevant code.

Overall, GLM-5.2 seems to live up to its hype on this task, it feels like a "pleb's Opus". 1/10th the cost for very similar results. Like previous GLM versions, it finds things that other models (like Claude and GPT) do not. Frankly, Deepseek and Qwen were too unreliable for this assignment, especially since those were the models I had to tune the instructions for.

Next round I will let the models prototype a manually designed solution and open a Pull Request (which won't be merged, but we'll have some actual code to compare.)

PS: Interestingly, I didn't seem to have missed any issue categories, which means the bots have more cognitive debt (versus what they're hyped to be) than I do. But then, I HitL everything, and my brain hurts.