Even if one thrall returns to you, it shouldn’t be an issue if you’re fighting alongside your thralls.