Does this happen instantaneously or after a certain time?
If it’s the latter, maybe you need to check whether the Abandon mechanic is enabled in your game’s settings.