Yeah, chances are it has to do with the amount of mods as some can be rather taxing depending on how many objects are being used on the servers. Things like exiles extreme can really hinder server performance with some of the objects and the effects they have. However, there are others who know far more about this subject such as @Multigun and others than I do.
You have several very memory heavy mods, several of which aren’t interested in optimizations and run resource heavy in general. Nothing Funcom can do about that. This is what you chose.
Wierd that it slowly decreases. I would think it would happen pretty quickly. Unless the mods are slowly adding a cache and not getting cleared. Figure if the amount of mods was the main reason, it would start as soon as you load in.