Once upon a time, there was no thrall cap and players regularly had hundreds of followers.
Some of these were bad, particularly PvP who would have all of their hundreds of followers crammed into a very small space, but others would have them spread out and, while high in number, were not concentrated in a single location.
Regardless, because of the bad actors, Funcom announced they would be introducing the follower cap. Everyone was scared of losing their best thralls and rushed around to delete all of their excess decorative followers.
The effect on server performance at that time was… negligible.
Server performance back when everyone had armies of thralls was vastly superior to server performance now, despite several optimization patches between then and now.
I knew that Funcom was interested in Living Settlements for a long time, but I always assumed they would work on making the server performance a lot better before trying to implement something like this. I even said as much:
In order to fully staff your base with crafters (i.e. have one of every profession and specialization), you need 1 alchemist + 3 armorers + 1 barkeeper + 3 blacksmiths + 3 carpenters + 1 cook + 21 priests + 1 smelter + 1 sorcerer + 1 tanner + 2 taskmasters = 38
Now, realistically, you don’t need all of those, edgesmiths are quite useless, and a lot of priest variants are quite hard to get and not particularly useful either. You can sort out a good dozen or so that you don’t need, but if we’re being realistic here, most players also have an outpost in addition to their main base. Outposts do not need a full set of crafters, but you might want a few to work the stations there. You are still looking at around half of a solo player’s cap is going to be occupied by crafters.
If clan members live together, they will be able to fill out their stations better and have enough slots left over to recruit combat followers, entertainers for their Transportory Stones, and set up a purge base. But clan members each having their own separate base… they are in for a tough time.
The largest problem in all of this is the game was not ready for living settlements. Funcom wanted a cool feature and jumped ahead on implementing it before making the game capable of handling it.
However, implement it they did, and the current follower cap is way too low, for both solo players and clans, to play this game in the way it was intended. Sure, you can live in a shack with only two followers if you enjoy doing that, but that is a personal playstyle. The game is not supposed to be tiny shacks with two followers. Every single update is pushing towards larger, grander bases, then making the living settlements dead because the follower cap is far too low to have any followers actually live there.