True, there should definitely be something like that to prevent switching them on the fly.

Maybe you’d have to do it at a very specific location, ritual spot, or a workbench, and as you say - only when you dismiss a current one and only once in a while.