The bad news, as always, is that I don't know what the best fix for this is.

Fine-grained access tokens allow you to limit repositories to an explicit set:

But that's not really enough because it lacks per-repo settings. Every 3rd party tool you give an access token to is potentially malicious, even when it's open-source and you run it locally only, like gh. Remember supply chain attacks; can you really trust anyone else's software?

The most secure way of using 3rd party apps on GitHub that I know of (other than "don't") is:

1. Create a specific account for each linked application and lock it down using permissions RBAC in each repo's settings, and then
2. Tightly managing access tokens for that account; short expiry.

Letting tools work under your account, especially since GH does awful things like pgp signing merge commits with their key on your behalf which then shows as "verified", is rather risky anyway.