I was having this issue last week. What worked for me was restarting my server and Xbox, and unplugging my network cable.
This thread addresses it in more detail: