Can not say from experience but I read on forum turn off ownership and u can use your buildings and thralls although that might be a last resort