Your client and server cannot share a port. Your friends can connect as long as you are not playing, but as soon as you are playing then you and your server are fighting over the port. Change the server to a alternate set of ports and you shouldn’t have anymore connection issues. Obviously you will need to forward the alternate ports to the servers IP and open up the firewall. If you continue to have issues, post your difficulties in that thread I linked above.