I'm very interested in why some orgs (your friend's company) can do better than the theoretical optimum, and other orgs (the govt) are generally recognized as not being able to?
This is too deep for me to tackle let alone right after I've met it. Everything obvious is probably something you've already considered.
In this specific case at least, their methodology isn't so much 'do something different and better with existing data' as it is 'other people collect data that is easy to collect but meaningful conclusions cannot be drawn from it.' Their meta-alpha is ignoring everything that's normally used to segment people and instead focusing on the intent of subjects and the fruitfulness of their intent, and only then drawing conclusions.
I don't know anything about this field but their methodology sounds like some level of innovation. So the question could be rephrased as why isn't the government good at innovation?
Governments were great at space stuff, but that's when governments were the only place space people could do space stuff.
As a free, exceptionally talented person, why would you go work for a government when the expectation is less pay and less impact? Governments can short employees on one or the other and probably stand a chance at competing for talent, but both? It's like choosing to move to the soviet union near its end.
(meta: what will happen to this reply when your post disappears?)
Replies will be preserved. The text of the comment will just disappear.