I can’t say that lots of buildings and stored loot doesn’t exacerbate the problem but I can say that even the fresh start servers that have almost nothing built or stored are also unplayable as far as combat goes once the number of people online gets above 20’ish.
Though it was a bit better today after the patch.