Returning home is an intended feature. If you set your thrall to stop following they will scout for a brief period of time before returning home. Home is wherever you last set them to guard. Also happens if you die.