pull down to refresh
Opus 5.5, Claude Code on the cluster. So it can submit test jobs using SLURM, check how it affects my old example folders, and go through all the regression tests I've set up previously when opening a PR on github. Did not change its default effort setting.
I'm digging Opus 5.5 for hunting too right now, it finally beats GLM 5.2. I'm running multiple instances concurrently (each in its own sandbox container inside an isolated VPS) when I have a huge queue, which has been the case all week. I'm now letting it do a first pass over the code that it in a previous step mapped end-to-end. Average assessment on a 50kB diff is currently 90 minutes, which is still 3-4x as fast as I could do it manually. I dig most of all that it gives me well structured, documented code it wrote for the hunting, and that that is fairly readable, without me having to tune it too much in the prompt.
Guess I was lucky then in my choice of Opus 5.5. I mostly picked it as I have enjoyed using it recently, and it was not eating up my token reserve too quickly (I alternate between a private and an institutional account 20$, so I rarely have to wait for a new 5-hour slot to free up).
I've been writing md files in the last 2 months or so as I was interacting with this code for development and production jobs, taking note of everything it learns about my code base along the way. It's one of the reasons I did not start this bug hunt earlier as I hadn't mapped out the code base yet.
I rarely have to wait for a new 5-hour slot to free up
Interesting. I cannot run a single job on the $20 with Opus 5.5. It always dies before the job is finished. So I upgraded it because the extra usage cost was crazy. I was using Kimi K3 and GLM 5.2 a lot, mostly because I find claude's slop very annoying to read, but it got better with 5.5 (it still doesn't do what I tell it though.) Now testing K3.1, I hope it will run with more precision, and that I need less "critic" runs because that was getting expensive too.
Haha, either your frameworks are much more demanding than mine, or they've identified me as someone whose Claude interactions can help them solve one of the problems they've listed in their next "we've solved math/physics/science" #AISlop dumps. That's probably giving my research too much importance, but I'm happy with a bit of delusion in my life ;)
I'm only half-joking, I read some mathematicians sharing experiences getting unreasonable amounts of tokens for free, out of the blue.
I'm using it for large scale security work so it needs to do things precisely. For example, I monitor about 3500 libraries across npm/crates/pypi through an automated (non-LLM) framework.
Assess this 2 MB diff for a low level dependency version bump line by line, then trace its usage end-to-end through a spaghetti architecture of 800 other 3rd party dependencies into the bespoke code of this repo, and while you're at it, cluster in functional chunks, explain what the change does, scan for injection of malware, decode every base64/hex/eval() and so on and report what it does. Prepare scopes with exact source, file, line ranges to hand off to hunters.
Then step 2, which is what I'd normally give to a framework of mixed kimi/glm agents/subagents: take this code stub, with this and that integration path, identify all areas susceptible to bugs according to <list> throughout the whole path, fire subagents to build harnesses, fuzz the fuck out of it, for each candidate hand it off to a critic. If the critic agrees, write a bug report for the candidate, submit to <private bug server>, report to the issue with all drafted candidate reports, and list all candidates that were not confirmed by the critic.
Then step 3, I read everything, which is why I need it to not be retarded slop.
That's one use-case. Others are either more reactive (CVE got published, assess which code uses it), release reviews (for graphene or lnd or the android apps I run), explain every line in PR xyz... and so on. Everything runs in private tho, no github except to fetch package sources.
That settles it. It was the first option: I'm not a VIP (very important physicist) in Anthropic's eyes, your frameworks are just much more demanding.
Have you tried this /ultrareview thing?
Oh. Yes, it's pretty nice. But that's not integrated with my frameworks and I have my own that isn't locked in to anthropic that works with any bot - I also don't run claude code manually ever unless the auth token expired. I just queue up jobs and assign it to the right bot/harness/skill combination and it does what I tell it to do - all asynchronous so that stuff moves even when I sleep.
I'll give those a try then.
Was referring to /ultrareview. Were you?
which model and which harness are you using?