You're right. Phoenix is pretty open about the trust issues in the FAQ:
Phoenix is trust-minimized, but not trustless. Wise people know that there is no such thing as trustless and that it's all a matter of trade-offs.
The following operations require trust:
channel opening (until the funding tx is confirmed) swaps (you pay upfront, and then our node does the swap) You can configure Phoenix to use your own Electrum server to watch the Blockchain and monitor your channels. This significantly reduces your dependency on third parties to secure your wallet.
For a more detailed overview of the trade-offs, see our blog post.