I think I can help here seeing as I was literally looking into the potential for making a mod related to horses earlier today
You are correct!
All the horse perks indeed are either Vitality, Strength or Grit.
Here are all the horse perks currently in the game (Might is Strength and Stamina is Grit)
On top of these they can have the 3 perks marked as “Universal”:
Jack of all Trades: +1 to Vit, Str, Grit, Agi
Jack of all Trades: +2 to Vit, Str, Grit, Agi
Universal Warrior: +3 to Vit, Str, Grit, Agi
As for “Agility increases the speed of horses”… I have to admit, for a very long time I had the same impression, especially when my horse leveled while I was riding it and gained some new Agility points, it seemed to have sped up.
However even then I thought that might be some weird placebo effect or simply resulting from the “Your follower gained a level” giant message breaking the flow of things up and causing you to suddenly pay attention at the actual speed…
Why? Because whenever I played with someone else and we traveled together on horses, no matter how super-stat max level horse I had, their level 0 horse kept up with the same saddle literally on point, which made me skeptical of any speed difference between them.
After poking around in the devkit today around horses and saddles, I still did not find any evidence as of yet that stats would have any effect on the speed or maneuverability of mounts.
The speed values are defined in the saddle table in 3 values Walk speed / Run speed / Sprint speed. The first 2 being identical for every saddle in the game and the sprint speed slightly varying based on the type of saddle.
Alongside this there’s a 4th parameter defining their acceleration template, (slow / travel / fast) that refers to a separate table where each of those has acceleration speeds defined in stages.
Easier if I show you I guess
Here’s the Saddle-table record for the Aquilonian Scout Saddle for example:
And here’s the acceleration profile for Travel saddles that the last value is referring to: