Nothing scales off your agility. Authority is the only stat that affects followers, and follower damage %+ buffs.