Correct, you go into the follower menu when you bring up clan/inventory/map/bazaar/etc and the follower list will populate in a few and allow you to delete them.