In some cases, yes, it is illegal to compete directly with a state agency. The Post Office is a fairly trivial example: it's illegal for UPS or FedEx to carry letters. Air traffic control is another example.

In other cases, the issue is that the state effectively floods the sector and leaves no room for a private alternative to be profitable. Research and adjudication are both examples of this: there is some private sector activity in those areas but the vast majority of it is funded through the state.