There's no need to keep the whole UTXO set in memory. Most of it is rarely read from disk during normal operation. The dbcache (i.e. UTXO set cache) is 450MB by default and you usually only want to increase that for IBD performance.
10% of transactions spent UTXOs created in the same block ( and a big chunk of UTXOs is spent after a few blocks anyway