There’s a suspected bug that ties into the building system causing followerers to decay. So submitting a bug report here in the forums would be the correct route.
If it is that.
But there can be official server related issues that can affect it. Like a stealth change for that specific server or like tia had explained, hackers.
Here’s a guide Bug Reporting - Here's advice on how to do it