Here's an older paper that tries to explain selfish mining dynamics with coin tosses. I'm not sure that it made it that much easier for me to think about selfish mining and the dynamics there, but perhaps it is helpful to others.

One element that I hadn't considered before was this: it matters a lot how connected to the rest of the network the potentially selfish miner is. If a miner can see most of the network and has high confidence that they will know very quickly when another miner finds a block, they can more safely withhold their own block in the attempt to mine selfishly on it. However if they don't have any better connectivity to the network than others selfish mining becomes more risky.

The other interesting part of this paper is how they talk about the difficulty adjustment. Difficulty adjusts every 2016 blocks, but this doesn't count any orphan blocks that are found. So, a miner who is able to cause its competitor's blocks to get orphaned more frequently gains an additional benefit of not increasing the difficulty as fast as it otherwise might. And this is some of the reason a selfish miner gains an advantage.

...at least I think that's what it is saying.

They propose a pretend fix for this dynamic that is interesting to think about:

let’s imagine that they could. Let’s even imagine that miners are incentivized to do so by modifying the rule that governs the official blockchain. In the event of a tie between two blockchains, we should select the one that contains the most proof of work, taking into account orphaned blocks as well, provided that they have an ancestor in the considered blockchain. This would be a kind of reinforcement of Satoshi Nakamoto’s rule. In case of another tie between two blockchains with the same characteristics, a node would select, as it does today, the one that was transmitted to it first.

We now consider another mining problem. At what relative hash power q does it no longer make sense for a miner to be honest? For a miner, being honest means always mining on the last block of the official blockchain and always making any blocks discovered public. Not being honest means the opposite: keeping discovered blocks secret or not mining on the last block of the official blockchain.

The threshold beyond which a miner no longer has an interest in staying honest on Bitcoin is at most equal to 32.94% (and even less of course if the miner can rely on a non-zero connectivity parameter).

It seems that there has been some renewed interest in this concept, or at least it reminds me of inertial mining which I'm still trying to wrap my head around (#1494186).