The FSF article (2) explains a bit why it is hard: firmware for many components are closed, also if you go through the open-BOM phones that are generously listed on Wikipedia under (3). Unless these are replaceable components ¹ you're still manufacturer/model locked, and what I understood from a friend that has a fairphone, you're also kind of vendor locked; you order replacement components through fairphone.

So a lot of the stuff calling itself open source now is what Stallman would probably go nuts over in a talk as there's a lot of open-source washing going on. Librephone ought to be different, so that's something to keep an eye on. But they've just started building an inventory of everything closed that needs to be replaced, it's in prelim phase.