Yes, it’s completely illogical that this perk only persists while following. A warrior cannot drastically weaken just because he does not follow his master. It was necessary to keep this perk on a permanent basis if it is leveled.