Check this out: https://sailfishos.org/

It's Linux-based, but it has non-free components. For $50 you get the version with a closed-source but seemingly good Android emulator. I've seen some review videos and it seems promising. I would consider trying it out. I know I'll get some flak for supporting paid software, but I would love to financially support any company that offers a viable alternative to the Google/Apple duopoly. The more competitors, the lower the barriers to additional players. KaiOS had some promise a few years back but once they took funding from Google you could see where things were going.

I guess that doesn't exactly answer the OP question about how to create a new platform, but I think that what worked for Sailfish is that they allow you to maintain compatibility with Android. Google has very powerful network effects, it would be very difficult to take them head on.

I don't think there's any serious need to integrate bitcoin, but if this platform is developed by bitcoiners I think giving away some sats along with the device would be positive for bitcoin adoption. Having the OS use, by default, a nostr relay and LN node run by the company would create some network effects around the phone, albeit most likely weak ones.