You can only interact with the fighters, archers, and dancers. Also the sellsword once you’ve unlocked and purchases him. Everyone else is just passing through.