Cant change difficulty setting on xbox one

I keep trying to change the difficulty setting on my server but it tells me I need admin access. I made myself admin, I can change everything else but the difficulty.