Yeah, it seems caddy is http only. Trying a ufw rule and IP table entry next to forward all traffic on the port. Will definitely write a guide after figuring it out.