Bitcoin Core’s mempool data structure is limited to 300 MB by default. If a node’s mempool is full, it will start evicting the transactions with the lowest feerates. Transactions get into the mempools of nodes much the same way as they get into blocks: if they offer a higher feerate, they are added to the mempool, potentially displacing lower feerate transactions, whether they are big or small transactions.

OP_RETURN transactions do not have any advantage in the competition for blockspace, and OP_RETURN data appears in the output script. The output script is not subject to segwit’s witness data discount, and therefore each byte of OP_RETURN data incurs 1 kvB (4 wu) of blockspace.