Kind suggestions here from a CE lover 
Change the Authority Attribute to allow 1 thrall follower for every 5 Authority points.
Thus, one would get 1 extra follower at 5, another at 10, another at 15, and another 20. At level 20 in Authority, one would have 4 extra followers. Don’t make the extra follower a choice, simply make it so you get one at the given point levels.
Get rid of the the ‘well trained’ perk (see the pet specific attribute specializations below).
As one gains authority, one gains ‘Leadership’ and is able to command more followers at one time.
Build in some of the bonuses for every 5 attribute levels, and make the selections every 5 levels favor either humans or animals/monster pets. Animal ‘pets’ take damage, do damage, and behave differently than human 'thralls as they wear no armor and do not use weapons. Every 5 levels one chooses between 2 (or more?) perks. (this could be done for all attributes…add choices every 5 rather than at 10 and 20).
Perhaps the level perks for animals should be distinctly different than those of the human followers, too.
Perhaps add a Pet armor slot similar to a saddle slot (this would be a new crafting line and stuff to make). Perhaps a pet specialization feat etc etc.
The way animal followers aggro things is faulty - a pet like a panther ends up tanking because it seems that bleed and burst damage get aggo. Perhaps this is by design, as an enemy would want to take out the thing that is hurting it the most. However, a low armor pet will get burned down quickly even with high health because their armor is often poor. Pet Armor would help with this problem, if the aggro dynamic is not somehow adjusted.