Could it be that the organizing principles of the societies that are very poor tend to make most people very poor while only a few get fabulously wealthy? When a society degrades property rights to nothing then only people with wealth can protect themselves from the ravening state (probably because they are cronies of the state). When there is no protection for property rights of the people, everyone has no human rights and will remain poor and oppressed.

I agree, the textbook answer ignores a lot of issues.

It does explain why working class natives are more likely to oppose immigration while business leaders are more likely to support it.

Another thing that happens a lot is that economics students only learn this part: trade increases economic output without remembering the part where trade creates winners and losers.

And lastly, you're right... the textbook theory doesn't address at all the actions that developed nations can take to keep developing nations poor, and it doesn't address at all the cultural frictions that come from mass migration.