There is a lot more to wallet security and trying to boil it all down to reproducible builds is actively harmful to the ecosystem.
I disagree that Wallet Security is harmful to the ecosystem.
But to get further, there is the tragedy of the commons. Security audits on the code would help with that. Security audits are labor expensive. Who will pay for these security audits to vet the open source code on these wallets?