I took a lot of inspiration from joinmarket, which is the only coinjoin software I've ever used (other than testing my own). But I think I found a better way to banish sybils aka trolls, though I haven't yet implemented it.

Suppose in round 1, 100 people register for a coinjoin. If a troll never enters this round, no harm is done. If a troll sends a registration message for this round, there are three ways he can do it incorrectly: not register a valid change output, not register 1 or 2 valid inputs, or register 1 or 2 valid inputs but without proving ownership (you're supposed to sign a recent blockhash with the inputs' private keys). All of those are publicly detectable by every participant, so everyone simply ignores messages that fail any of these steps, and thus every honest participant continues to round 2 with the same set of other honest people. In other words, trollish "Round 1" messages are discarded by all honest parties, so they have no effect, it is as if the trolls never sent a message and thus never entered Round 1.

Suppose the trolls did round 1 correctly and are now in round 2. There are three ways a troll can do round 2 incorrectly: not register a valid "equal amount" output, register "too many" equal amount ouputs, or not provide a valid ring signature proving they were in round 1. All honest parties discard messages that lack valid ring signatures, so that part has no effect, it is as if they never sent the message. To detect a troll who uses either of the other two trollish behaviors (i.e. they register 0 or more than 1 "equal amount" output), all parties sum up the number of equal amount outputs and, if it is not equal to the number of people who were in round 1, they know a troll is among them. Therefore, every honest participant should send a new message to the group that unmasks their ring signature, thus revealing which of their inputs map to which of their outputs. If anyone does not do this, or if their now-mapped signature demonstrates that they submitted multiple outputs, the honest participants have identified a troll and the troll's inputs (they are whichever inputs were in a "round 1" message whose "ringsig pubkey" has not been unmasked as "belonging to" one of the outputs), so they kick that troll's inputs out of the group and restart with the remaining honest people. Continue this procedure until you enter round 3 or you are the only coinjoiner left, which just means there were no other honest coinjoiners in your group, so try again in the next round.

Suppose the trolls did rounds 1 and 2 correctly and are now in round 3. There are two ways a troll can do round 3 incorrectly: not provide valid "btc sigs" for their inputs or not provide a valid ring signature proving they were in round 1. All honest parties discard messages that lack valid ring signatures, so that part has no effect, it is as if they never sent the message. And if any troll did not send valid btc sigs, kick his inputs out of the group and restart with the remaining honest people. Continue this procedure until round 3 is done or you are the only coinjoiner left, which just means there were no other honest coinjoiners in your group, so try again in the next round.

If any troll goes through rounds 1, 2, and 3 properly, then they were not a troll, they did the coinjoin all the way through, so huzzah! But by the above methods you can kick a "real troll" out of any round (1, 2, or 3) and then redo the coinjoin with the remaining honest participants. This seems better than using fidelity bonds which make it so that many bitcoin holders cannot participate in joinmarket. Please let me know if you see any flaws in this protocol. I hasten to add, I have not implemented this "kickout protocol" yet, so Emessbee is currently flawed in exactly the way you identified: sybils can flood any attempt with fake messages to disrupt it and stop it from happening. But if Emessbee works in the happy path (it does) and if the kickout protocol can "enforce" the happy path, then I think we're in good shape.

I think it's implied that's out of band via the bbs, which could enforce membership