That would work but it's not secure unless everyone in the group is trustworthy, because any group member can publish the decryption key or share it with bad people
I think you should absolutely have “private” groups. Just because that is how it works in telegram.
You’re not making sure the posts are super duper secret. You’re just making it so that one person can’t mass download every single message from every channel.
When I see a list of people subscribed to my channel, my expectation is that more or less those are the people reading my things.