We do indeed. (And I find Warden comes up plenty.) For some samples of what perks I get, you can see this thread (again). But the thing you are missing is the impact that one bad perk has (again - I’m talking feeling, not numbers) - it can change your entire perception of the follower you’ve just spent around 15 hours plus as your constant companion while you worked to build them up (and that’s assuming you’re levelling them pretty quickly). It doesn’t matter that the other two perks might be decent, that negative has a direct impact.
There is also considerable difference in different approaches to the game, and thralls in general: you have 79 thralls in varying stages of levelling, and you presumably work on levelling and equipping them bit by bit. I am coming to the end of a long playthrough and have 12 fighting thralls (plus a couple of dancers), all of them level 20, all of them having been with me on numerous different adventures against a range of enemies. Each of them matters to me. Each has a name, their own weapons and armor, their own history of the things we fought together. I’ve done playthroughs where I’ve built up hordes of faceless base defenders - I didn’t enjoy the game anywhere near as much - they meant less to me, and the game meant less to me. Perhaps that helps to better illustrate why this matters to me?