From my experience, thralls don’t seem to return to their place when you are waiting at just that place. So maybe move away a little bit, e. g. farming some stone with another thrall in tow.