Unfortunately. The more people/ buildings/thralls, the worse it gets.