Not sure if this will help and you may already be aware of this. I’ve noticed my private server, if my game has the same mods installed as the server, and they are set in the same order as the server mod order, the game loads into the server without restarting.
If the mods are out of date, or the load order is different, the game will restart to use a temporary mod list, which it then downloads (even if you have the mods in a different order).
If the server is using a large number of mods and/or mods with large downloads, that may be compounding the issue of the load times.
I get frequent failed game starts using the temporary mod list than by ensuring I am using the same mods and mod order.
Hope this helps.