Once thralls stop following (for example when you die, or order them to stop following), they will eventually teleport to the last place they were set to guard.
You have some time to go back and order them to follow again if you are quick.