Handling of concurrent transactions (two transactions spending the same UTXO, eg "double spending") is a local policy, meaning each node is free to use whatever rule they want as long as it doesn't break any of the protocol's rules.
Now there is, of course, the default policy in Bitcoin Core, which often ends out being the default policy of most of the network. Historically it was a "first-seen" rule: nodes would consider that the first version of the transaction they see is the "right" one, and hence consider any subsequent attempt at spending the same coins as invalid.
Then came Opt-In RBF, which is the current default in Core. Through this mechanisms, transactions can signal (using nSequence) that it's ok for them to be replaced by a concurrent transaction, under some conditions. One of the conditions is that the replacing transaction must pay more fees than the original one, and enough so to prevent spam/DoS attacks. So, to put it shortly, the version of the transaction that pays the most fees takes priority.
However, it is still possible to double spend a transaction that doesn't signal RBF. That's because some nodes implement a distinct policy, which is to allow any transaction that pays more fees to replace the original one. This policy particularly makes sense for miners, as it allows them to extract more revenue. Some nodes in the relay network also implement this policy, and even if a small portion of the network does, it might be enough for my transaction to reach a miner, although it will probably take more time to propagate.
That's one of the arguments in favor of Full-RBF, which was merged into Bitcoin Core a few months ago but is off by default. Full-RBF essentially looks a lot like Opt-In RBF, but without the requirement for transactions to actively signal they're eligible for replacement.
So to more directly answer your question: it depends! There will be two (or more) concurrent transactions, out of which only one can ultimately be mined into a block. But they're strictly speaking all valid. You can nudge other nodes, and notably miners, to consider the transaction you'd like to be mined as the "real one" by setting higher fees, but that's ultimately up to them. You can't force them. Once your transaction is out there in the wild and you want to replace it, all you can do is hope miners learn about the replacement transaction and decide to mine it over your initial transaction. Which is the case 99% of the time, because miners want money.
TL;DR: we used to use a "first seen" priority rule but it doesn't align with miners incentives and can yield different results across the networks. The current widespread rule is that whichever transaction pays the most fees is the one that takes priority. But anyone can implement whatever stupid rule they want on their own node. If their rule is really stupid, they'll just disagree with everyone else more often, but converge onto the most-work chain anyway.
Got a bit verbose here, but I hope it helps answer your questions!
Thank you for this excellent answer! This cleared it up for me.
reply