The game should be able to dynamically turn them off or on prior to connecting to a world that doesn’t use a particular mod (or no mods).
…not sure why the game has to load or unload a mod while still at the main menu. Nothing should be ‘touched’ at that point of the game.
But at the very least we should be able to turn off mods that are loaded (aka Enabled) without needing to exit the game.